Medical image real-time fusion method and device, equipment and storage medium

Abstract

The invention discloses a medical image real-time fusion method and device, equipment and a storage medium. The medical image real-time fusion system comprises an image acquisition terminal, an image processing terminal and an image display terminal. The method comprises the following steps: acquiring a first angiography of a target patient before an operation and a real-time first perspective image in the operation through the image acquisition terminal; performing frame-by-frame real-time fusion on the first angiography and the first perspective image through the image processing terminal under the condition that a medical operation is detected to be started, and generating a real-time medical fusion video; and displaying the medical fusion video image fused in real time through the image display terminal. The blood vessel image and the perspective image are fused in real time, so that a doctor observes the dynamic change of the blood vessel of a patient in real time, the complexity of surgical operation is remarkably reduced, and the surgical experience and the operation precision of the doctor are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image fusion, and in particular to a medical image real-time fusion method, device, equipment and storage medium. BACKGROUND

[0002] With the aging of the population and changes in lifestyle, the incidence of cardiovascular disease is increasing year by year, so that the demand for accurate medical image analysis of vascular interventional surgery in the medical field is increasing.

[0003] In the traditional medical image display, the X-ray image and the DSA contrast data are displayed in split screen, which requires the doctor to frequently switch the line of sight for observation, resulting in distraction of attention during the operation, which may prolong the operation time when dealing with complex lesions. In addition, since the DSA contrast is non-real-time image data, it cannot reflect the dynamic anatomical structure changes when the catheter moves, resulting in an increase in the misjudgment rate of the doctor for the spatial position of the catheter.

[0004] Therefore, the traditional medical image display has been unable to meet the actual needs of intraoperative medical image assistance at the present stage. SUMMARY

[0005] The present application provides a medical image real-time fusion method, device, equipment and storage medium, which fuses the blood vessel image and the fluoroscopy image in real time, so that the doctor can observe the dynamic changes of the patient's blood vessels in real time, significantly reducing the complexity of the operation and improving the doctor's operation experience and precision.

[0006] According to an aspect of the present application, a medical image real-time fusion method is provided, which is applied to a medical image real-time fusion system, the system comprising an image acquisition terminal, an image processing terminal and an image display terminal; the method comprising:

[0007] acquiring, by the image acquisition terminal, a first angiogram of a target patient before operation and a first fluoroscopy image in real time during operation;

[0008] In the case where the medical operation is detected, the first angiogram and the first fluoroscopy image are fused in real time by the image processing terminal, and a real-time fusion medical fusion video is generated;

[0009] The real-time fusion medical fusion video image is displayed by the image display terminal.

[0010] According to another aspect of the present application, a medical image real-time fusion device is provided. The device is applied to a medical image real-time fusion system, comprising:

[0011] an image acquisition module configured to acquire, by the image acquisition terminal, a first angiogram of a target patient before surgery and a first fluoroscopy image in real time during surgery;

[0012] an image processing module configured to, in a case where it is detected that the medical surgery starts, perform real-time fusion of the first angiogram and the first fluoroscopy image by the image processing terminal, and generate a medical fusion video of real-time fusion;

[0013] an image display module configured to display the medical fusion video of real-time fusion by the image display terminal.

[0014] According to another aspect of the present application, an electronic device is provided, which comprises:

[0015] at least one processor; and

[0016] a memory in communication connection with the at least one processor; wherein,

[0017] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the medical image real-time fusion method according to any one of the embodiments of the present application.

[0018] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to perform the medical image real-time fusion method according to any one of the embodiments of the present application when executed.

[0019] The technical scheme of the embodiments of the present application acquires, by the image acquisition terminal, a first angiogram of a target patient before surgery and a first fluoroscopy image in real time during surgery. The image processing terminal performs real-time fusion of the first angiogram and the first fluoroscopy image when the X-ray fluoroscopy image data is collected by the acquisition card, and generates a medical fusion video of real-time fusion. The image display terminal displays the medical fusion video of real-time fusion. Through deep fusion of deep learning and computer vision technology, millisecond-level accurate registration and dynamic superposition of angiogram images and X-ray fluoroscopy images are realized, and a blood vessel dynamic navigation atlas with augmented reality effect is finally generated, which breaks through the pain points of traditional interventional surgery, such as dependence on repeated angiography and strong subjectivity of visual judgment, so that doctors can observe the anatomical details and hemodynamic changes of the patient's blood vessels in real time through the super-low delay fusion image without additional injection of contrast agent, which significantly reduces the cognitive load and physical complexity of the operation.

[0020] It is to be understood that the embodiments described herein are merely exemplary of the application and that a person skilled in the art can devise other embodiments without departing from the scope of the present application. It is also to be understood that not all of the benefits described herein need necessarily be realized in any particular embodiment of the application. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0022] Figure 1 is a flow chart of a medical image real-time fusion method according to an embodiment of the present application;

[0023] Figure 2 is a structural diagram of a medical image real-time fusion device according to an embodiment of the present application;

[0024] Figure 3 is a structural diagram of an electronic device for implementing the medical image real-time fusion method according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the technical personnel in the art better understand the present application, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should be within the scope of the present application.

[0026] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment one

[0028] Figure 1A flowchart of a medical image real-time fusion method provided for the first embodiment of the present application. The present embodiment can be applied to the case of observing the dynamic changes of blood vessels in real time. The method can be executed by a medical image real-time fusion device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in Figure 1 The method comprises the following steps.

[0029] S101. Acquire, by the image acquisition terminal, a first angiogram of a target patient before surgery and a first fluoroscopy image in real time during surgery.

[0030] The image acquisition terminal can be a capture card. It should be noted that medical image devices such as DSA (Digital Subtraction Angiography) devices and C-arm X-ray machines do not have image data export ports and cannot obtain medical image data, thus cannot perform medical image fusion. Therefore, the present application connects the capture card with the medical image device to acquire medical image data (such as the first angiogram and the first fluoroscopy image) captured by the medical image device through the capture card, thereby providing a data basis for subsequent medical image fusion.

[0031] The first angiogram can refer to a digital subtraction angiogram of a target patient obtained by injecting contrast agent before surgery. The first fluoroscopy image can refer to an X-ray fluoroscopy image obtained in real time during surgery of the target patient.

[0032] Specifically, the first angiogram of the target patient captured by the DSA device is acquired through the capture card before surgery, and the first fluoroscopy image of the target user captured by the X-ray machine is acquired through the capture card during surgery.

[0033] S102. In the case where the medical procedure is detected to start, the first angiogram and the first fluoroscopy image are fused in real time by the image processing terminal, and a medical fusion video fused in real time is generated.

[0034] The medical fusion video can refer to a medical fusion video fused in real time from the first angiogram and the first fluoroscopy image. Through the medical fusion video, the doctor can view the dynamic changes of the blood vessels of the target patient in real time, which facilitates the doctor to perform precise vascular intervention operation.

[0035] Specifically, the first angiogram and the first fluoroscopy image are fused in real time by the image processing terminal to obtain a medical fusion video.

[0036] Exemplarily, the first angiogram and the first fluoroscopy image are fused in real time to generate a medical fusion video fused in real time, which comprises the following steps.

[0037] determine a latest first image frame in the first perspective image according to a video frame timestamp;

[0038] determine a blood vessel mask image corresponding to the target patient according to the first angiogram;

[0039] perform morphing fusion processing on the blood vessel mask image and the first image frame based on an image morphing matrix between the first image frame and the first angiogram, to obtain a medical fusion image corresponding to the target patient.

[0040] It should be noted that, since the first angiogram and the first perspective image need to be fused in real time, the real-time frame of the first perspective image, i.e., the latest first image frame, needs to be determined, and the first image frame and the first angiogram need to be fused to obtain a fusion image.

[0041] The blood vessel mask image can be a mask image in which only blood vessels are retained after denoising of the first angiogram. The image morphing matrix can be a transformation matrix between the first image frame and the first angiogram, including a stretching matrix, a torsion matrix, and a rotation matrix, etc.

[0042] Specifically, the first image frame and the blood vessel mask image are determined, the first image frame and the blood vessel mask image are subjected to morphing fusion processing with reference to the image morphing matrix, to obtain a medical fusion image corresponding to the latest first perspective image. The medical fusion image can also be sent to an image display terminal in real time for real-time display.

[0043] For example, the determination of the blood vessel mask image corresponding to the target patient according to the first angiogram includes mask generation according to the first angiogram and a pre-trained target blood vessel segmentation model, and obtaining the blood vessel mask image based on the output of the target blood vessel segmentation model.

[0044] The target blood vessel segmentation model is obtained according to sample angiogram data and blood vessel labels.

[0045] Specifically, the first angiogram is input to the target blood vessel segmentation model for mask generation, and the blood vessel mask image is obtained according to the output of the target blood vessel segmentation model.

[0046] It should be noted that the target blood vessel segmentation model is mainly used to segment blood vessel structures in angiogram images, and a suitable segmentation model can complete the segmentation task. Here, the SAM-VMNet segmentation model is taken as an example. The segmentation process of this model is divided into two main stages, and feature fusion and optimization are achieved through a parallel encoder structure:

[0047] (1) Coarse segmentation and hint generation

[0048] 1. VM-UNet coarse segmentation: The input image is first coarsely segmented by a lightweight VM-UNet to generate an initial blood vessel mask. The VM-UNet is based on a visual state space model (VSS) and efficiently captures long-range dependencies through a linear complexity selective state space module (SS2D).

[0049] 2. Prompt point selection: 10 spatial points are uniformly selected from the coarse segmentation mask as input prompts for MedSAM, guiding it to focus on key vessel regions.

[0050] (2) Dual-encoder feature fusion

[0051] 1. MedSAM feature extraction: The MedSAM encoder based on Vision Transformer (ViT) processes the original image and prompt points to generate high-dimensional feature vectors, emphasizing global context information.

[0052] 2. VM-UNet feature extraction: The parallel running VM-UNet encoder extracts local detail features through VSS blocks, and its asymmetric encoder-decoder structure optimizes computational efficiency.

[0053] 3. Feature fusion: The two-way feature vectors are fused through concatenation and attention weighting, and input into the VM-UNet decoder to generate the final segmentation result.

[0054] Exemplarily, the mask generation according to the first angiogram and the pre-trained target blood vessel segmentation model determines a blood vessel mask image, including: inputting the first angiogram image into the target blood vessel segmentation model to make the target blood vessel segmentation model generate a mask, and obtaining an initial mask image; performing background elimination processing on the initial mask image to obtain a pure blood vessel mask image.

[0055] It should be noted that in the present application, the first angiogram can be input into the target blood vessel segmentation model to obtain a blood vessel mask image. However, due to the existence of background images (such as bones and tissue muscles) in the first angiogram, it will have a certain impact on the accuracy of the blood vessel mask image. Therefore, in another embodiment of the present application, the initial mask image output by the target blood vessel segmentation model can be processed by a suitable algorithm to eliminate the background and obtain a pure blood vessel mask image, so as to improve the accuracy of the blood vessel mask image and further improve the accuracy of the medical fusion video.

[0056] Exemplarily, the first angiogram image is input into the target blood vessel segmentation model to make the target blood vessel segmentation model generate a mask to obtain an initial mask image, including:

[0057] based on a preset image resolution, the first angiogram is image segmentation processing, obtain a plurality of angiogram sub-image; for each of the angiogram sub-image, respectively input to the target blood vessel segmentation model for mask generation, obtain a blood vessel sub-mask image; according to each of the blood vessel sub-mask image is mask restoration splicing processing, obtain the initial mask image.

[0058] Wherein, the preset image resolution can be referred to as the resolution of the first angiogram image segmentation.

[0059] Specifically, the first angiogram is, according to the preset image resolution is image segmentation processing, obtain a plurality of angiogram sub-image. Then a plurality of angiogram sub-image in turn into the target blood vessel segmentation model for reasoning, obtain each angiogram sub-image corresponding blood vessel segmentation mask image. Further splicing blood vessel segmentation mask image restore original size, get the final binary initial mask image, and then get the blood vessel contour map. By cutting the first angiogram, and then the angiogram sub-image is mask generation, so as to further improve the accuracy of the blood vessel mask image.

[0060] On the basis of each of the above embodiments, the training process of the target blood vessel segmentation model comprises:

[0061] Obtaining angiogram sample data and the blood vessel structure annotation result corresponding to the angiogram sample data; the angiogram sample data is input into the preset blood vessel segmentation model to generate a predicted blood vessel mask, the difference between the predicted blood vessel mask and the blood vessel structure annotation result is determined, and the training error is propagated to the preset blood vessel segmentation model, the network parameters in the preset blood vessel segmentation model are adjusted; the above process is iteratively executed until the preset convergence condition is met, the preset blood vessel segmentation model training is determined to be completed, and the target blood vessel segmentation model is obtained.

[0062] Wherein, the obtained DSA data set is blood vessel structure annotation, obtaining angiogram sample data and blood vessel structure annotation result.

[0063] Specifically, the training error can be determined based on the training function according to the difference between the predicted blood vessel mask generated by the preset blood vessel segmentation model and the blood vessel structure annotation result, and the training error can be back propagated to the preset blood vessel segmentation model to adjust the network parameters in the preset blood vessel segmentation model. The above process is iteratively executed until a preset convergence condition is met, such as when the number of iterations reaches a preset number or the training error converges, the training of the preset blood vessel segmentation model is determined to be completed, and the preset blood vessel segmentation model at this time can be used as the target blood vessel segmentation model. By using the angiography sample data and the blood vessel structure annotation result for model training, the accuracy of the target blood vessel segmentation model for mask generation can be ensured, and the accuracy of the medical fusion image can be ensured.

[0064] Exemplarily, the determination process of the image deformation matrix between the first video frame and the first angiogram includes: determining first position information of a preset feature point in the first video frame; determining second position information of the preset feature point in the first angiogram; and matching the first position information with the second position information to determine the image deformation matrix between the first video frame and the first angiogram.

[0065] The preset feature point is obtained by neural network implicit learning, which can be a specific bone point or a specific tissue point of the target patient. The preset feature point is retained in the first video frame and the first angiogram. It should be noted that the determination process of the preset feature point includes obtaining all feature points in the first video frame and the first angiogram by neural network implicit learning, pairing each feature point in the first video frame and the first angiogram to obtain each feature point pair therebetween. Each feature point pair is checked and examined to eliminate mis-matched feature point pairs, thereby obtaining the preset feature point.

[0066] The first position information of the preset feature point in the first video frame and the second position information of the preset feature point in the first angiogram are determined. The first position information is matched with the second position information, so that the image deformation matrix can be obtained to deform and fuse the blood vessel mask image and the first video frame.

[0067] S103, through the image display terminal, the real-time fused medical fusion video image is displayed.

[0068] Specifically, the video track technology or the data channel technology is used for transmission to the image display terminal (such as a doctor client page), which can be viewed by doctors. Exemplarily, the client page supports viewing of real-time perspective images obtained by the acquisition card, viewing of real-time fusion images, and controlling of blood vessel transparency, display of blood vessels, display of outlines, and replacement of blood vessel color to meet different needs of doctors during surgery.

[0069] The technical scheme of the embodiment of the present application acquires the first angiogram before operation and the first real-time perspective image during operation of a target patient through the image acquisition terminal. The first angiogram and the first perspective image are fused in real time when the X-ray perspective image data is collected by the image acquisition card through the image processing terminal, and a real-time fused medical fusion video is generated. The real-time fused medical fusion video image is displayed through the image display terminal. Through the deep fusion of deep learning and computer vision technology, millisecond-level accurate registration and dynamic superposition of angiogram images and X-ray perspective images are realized, and a vascular dynamic navigation atlas with augmented reality effect is finally generated, which breaks through the pain points of traditional interventional surgery, such as the need for repeated angiography and strong subjectivity of visual judgment. In this way, the doctor can observe the anatomical details and hemodynamic changes of the patient's blood vessels in real time through the super-low-delay fusion image without additional injection of contrast medium, which significantly reduces the cognitive load and physical complexity of the operation.

[0070] Embodiment two

[0071] Figure 2 A structural schematic diagram of a medical image real-time fusion device provided by the second embodiment of the present application is shown in FIG. 2. Figure 2 As shown in the figure, the device comprises:

[0072] An image acquisition module 201 is configured to acquire the first angiogram before operation and the first real-time perspective image during operation of a target patient through the image acquisition terminal.

[0073] An image processing module 202 is configured to fuse the first angiogram and the first perspective image in real time and generate a real-time fused medical fusion video when the medical operation starts through the image processing terminal.

[0074] An image display module 203 is configured to display the real-time fused medical fusion video image through the image display terminal.

[0075] Optionally, the image processing module 202 comprises:

[0076] An image frame acquisition submodule is configured to determine the latest first image frame in the first perspective image according to the video frame timestamp.

[0077] A mask image determination submodule is configured to determine the blood vessel mask image corresponding to the target patient according to the first angiogram.

[0078] A fusion image determination submodule is configured to perform morphological fusion processing on the blood vessel mask image and the first image frame based on the image deformation matrix between the first image frame and the first angiogram to obtain the medical fusion image corresponding to the target patient.

[0079] Optionally, the mask image determination sub-module is specifically configured to:

[0080] perform mask generation according to the first angiogram and a pre-trained target blood vessel segmentation model, and obtain a blood vessel mask image based on an output of the target blood vessel segmentation model, wherein the target blood vessel segmentation model is obtained according to sample angiogram data and blood vessel labels.

[0081] Optionally, the mask image determination sub-module further includes:

[0082] The angiogram image processing unit is configured to input the first angiogram image into the target blood vessel segmentation model, so that the target blood vessel segmentation model performs mask generation to obtain an initial mask image.

[0083] The mask image determination unit is configured to perform background elimination processing on the initial mask image to obtain a pure blood vessel mask image.

[0084] Optionally, the angiogram image processing unit is specifically configured to:

[0085] perform image segmentation processing on the first angiogram based on a preset image resolution to obtain a plurality of angiogram patch images.

[0086] For each angiogram patch image, input into the target blood vessel segmentation model for mask generation to obtain a blood vessel patch mask image.

[0087] perform mask restoration splicing processing according to each blood vessel patch mask image to obtain an initial mask image. Optionally, the mask image determination sub-module further includes a model training module, wherein the model training module is configured to:

[0088] obtain angiogram sample data and a blood vessel structure annotation result corresponding to the angiogram sample data;

[0089] input the angiogram sample data into a preset blood vessel segmentation model to generate a predicted blood vessel mask, determine a training error based on a difference between the predicted blood vessel mask and the blood vessel structure annotation result, and propagate the training error to the preset blood vessel segmentation model to adjust network parameters in the preset blood vessel segmentation model.

[0090] iteratively perform the above process until a preset convergence condition is met, determine that the preset blood vessel segmentation model training is completed, and obtain a target blood vessel segmentation model.

[0091] Optionally, the image processing module 202 further includes a deformation matrix determination sub-module configured to:

[0092] Determine the first position information of the preset feature point in the first image frame;

[0093] Determine the second position information of the preset feature point in the first angiogram;

[0094] Match the first position information with the second position information to determine the image deformation matrix between the first image frame and the first angiogram.

[0095] The technical scheme of the embodiment of the application, through the image acquisition terminal, acquires the first angiogram of the target patient before operation and the first perspective image in real time during operation. Through the image processing terminal, when the X-ray perspective image data is collected by the acquisition card, the first angiogram is fused with the first perspective image in real time, and a real-time fused medical fusion video is generated. Through the image display terminal, the real-time fused medical fusion video image is displayed, through the deep fusion of deep learning and computer vision technology, the millisecond-level accurate registration and dynamic superposition of angiogram image and X-ray perspective image are realized, and finally the vascular dynamic navigation atlas with augmented reality effect is generated, which breakthroughly solves the pain points such as strong subjectivity of visual judgment and repeated angiography in traditional interventional surgery, so that the doctor can observe the anatomical details and hemodynamic changes of the patient's blood vessels in real time through the super-low delay fusion image without additional injection of contrast agent, and the cognitive load and physical complexity of the operation are significantly reduced.

[0096] The medical image real-time fusion device provided in the embodiment of the application can execute the medical image real-time fusion method provided in any embodiment of the application, and has the corresponding function modules and beneficial effects of the execution method.

[0097] Embodiment three

[0098] Figure 3 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the applications described and / or claimed in this document.

[0099] As Figure 3As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0100] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0101] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the medical image real-time fusion method.

[0102] In some embodiments, the medical image real-time fusion method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the medical image real-time fusion method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the medical image real-time fusion method by any other appropriate means, such as by means of firmware.

[0103] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0104] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0105] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0106] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0107] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0108] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0109] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0110] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above, but only by the scope of the appended claims.

Claims

1. A real-time medical image fusion method, characterized in that: Applied to a real-time medical image fusion system, the system includes an image acquisition terminal, an image processing terminal, and an image display terminal; the method includes: Acquire the target patient's first preoperative angiography and real-time first intraoperative fluoroscopic image through the image acquisition terminal; When the start of a medical operation is detected, the image processing terminal fuses the first angiography image with the first fluoroscopic image in real time to generate a real-time fused medical fusion video; The real-time fused medical fusion video image is displayed through the image display terminal.

2. The method according to claim 1, characterized in that The step of fusing the first angiography image with the first fluoroscopic image in real time and generating a real-time fused medical fusion video image includes: determining, according to the video frame timestamp, a latest first image frame in the first perspective image; determining a blood vessel mask image corresponding to the target patient based on the first angiography; Based on the image deformation matrix between the first image frame and the first angiography, the blood vessel mask image and the first image frame are deformed and fused to obtain a medical fusion image corresponding to the target patient.

3. The method according to claim 2, characterized in that Determining the vascular mask image corresponding to the target patient based on the first angiography includes: A mask is generated according to the first angiography and a pre-trained target vessel segmentation model, and a vessel mask image is obtained based on an output of the target vessel segmentation model, wherein the target vessel segmentation model is trained based on sample angiography data and vessel labels.

4. The method according to claim 3, characterized in that The step of generating a mask based on the first angiography and a pre-trained target blood vessel segmentation model to determine a blood vessel mask image includes: inputting the first angiography image into the target blood vessel segmentation model so that the target blood vessel segmentation model performs mask generation to obtain an initial mask image; The initial mask image is subjected to background removal processing to obtain a pure blood vessel mask image.

5. The method according to claim 4, characterized in that Inputting the first angiography image into the target blood vessel segmentation model so as to generate a mask for the target blood vessel segmentation model to obtain an initial mask image includes: Based on a preset image resolution, performing image segmentation processing on the first angiography to obtain a plurality of angiography block images; Inputting each of the angiography block images into the target blood vessel segmentation model to generate a mask, thereby obtaining a blood vessel block mask image; Mask restoration and splicing processing is performed on each of the blood vessel block mask images to obtain an initial mask image.

6. The method according to claim 3, characterized in that The training process of the target blood vessel segmentation model includes: Acquiring angiography sample data and a vascular structure annotation result corresponding to the angiography sample data; Inputting the angiography sample data into a preset blood vessel segmentation model to generate a predicted blood vessel mask, determining a training error based on a difference between the predicted blood vessel mask and the blood vessel structure annotation result, and backpropagating the training error into the preset blood vessel segmentation model to adjust network parameters in the preset blood vessel segmentation model; The above process is iteratively executed until a preset convergence condition is met, and it is determined that the training of the preset blood vessel segmentation model is completed, and a target blood vessel segmentation model is obtained.

7. The method according to claim 2, characterized in that The process of determining the image deformation matrix between the first image frame and the first angiography includes: Determining first position information of a preset feature point in the first image frame; determining second position information of a preset feature point in the first angiography; The first position information and the second position information are matched to determine an image deformation matrix between the first image frame and the first angiography.

8. A real-time medical image fusion device, characterized in that: Applied to a real-time medical image fusion system, the device comprises: An image acquisition module, configured to acquire a first angiography before surgery and a first real-time fluoroscopic image of a target patient during surgery through the image acquisition terminal; an image processing module, configured to, through the image processing terminal, fuse the first angiographic image with the first fluoroscopic image in real time upon detecting the start of a medical procedure, and generate a real-time fused medical fusion video; The image display module is used to display the real-time fused medical fusion video image through the image display terminal.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the real-time medical image fusion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the real-time medical image fusion method according to any one of claims 1 to 7 when executed.

Images