Image fusion processing method and device, ultrasonic equipment and image fusion processing system

Abstract

The invention relates to an image fusion processing method and device, ultrasonic equipment and an image fusion processing system. The method comprises the following steps: acquiring a to-be-displayed video signal; the video signal is generated based on a signal acquired by medical imaging equipment; acquiring a medical image and an imaging parameter from the video signal, or acquiring the imaging parameter from medical imaging equipment; based on the imaging parameters, performing image fusion on the medical image and an ultrasonic image of the ultrasonic equipment to obtain a fused image; and displaying the fused image. The ultrasonic equipment can be used in combination with any type of medical imaging equipment, so that dependence on specific medical imaging equipment is eliminated, fusion display of ultrasonic images and medical images can be achieved only by using the ultrasonic equipment for a scene where the medical imaging equipment is deployed, the limitation problem of image fusion is solved, and the image fusion efficiency is improved. The application scene of image fusion is increased, and the compatibility between the ultrasonic equipment and any medical imaging equipment is improved, so that the universality of image fusion processing is improved.

Description

Technical Field

[0001] The present application relates to the field of image fusion technology, and in particular to an image fusion processing method, device, ultrasound equipment and image fusion processing system. Background Art

[0002] With the development of medical technology, surgical solutions have emerged that use multiple different imaging devices to perform auxiliary operations simultaneously. For example, when performing esophageal-related surgeries, esophageal ultrasound equipment and endoscopic subarachnography (DSA) equipment can be used. Among them, the esophageal ultrasound equipment can provide real-time 3D images of the patient's soft tissue in the surgical area during surgery, and the DSA equipment can provide real-time 2D images of the patient's bones, blood vessels, and surgical instruments and consumables in the surgical area during surgery.

[0003] In order to improve the viewing effect of doctors on images from different imaging devices, the real-time 3D image of the ultrasound device and the real-time 2D image of the DSA device can be fused and displayed.

[0004] However, traditional image fusion processing solutions are only applicable to specific imaging devices and have great limitations. Summary of the Invention

[0005] Based on this, it is necessary to provide an image fusion processing method, device, ultrasound equipment, image fusion processing system, computer-readable storage medium and computer program product that can be applied to any medical imaging equipment to improve the scope of application of image fusion processing and thus improve the universality of image fusion processing.

[0006] In a first aspect, the present application provides an image fusion processing method, applied to an ultrasound device, comprising:

[0007] Acquire a video signal to be displayed; the video signal is generated based on a signal acquired by a medical imaging device;

[0008] Acquiring medical images and imaging parameters from video signals, or acquiring imaging parameters from medical imaging equipment;

[0009] Based on the imaging parameters, the medical image and the ultrasound image of the ultrasound device are fused to obtain a fused image.

[0010] In one embodiment, obtaining a video signal to be displayed includes:

[0011] Acquire a video signal to be displayed output by a medical imaging device sent by a video distributor; connect the medical imaging device to the video distributor; or,

[0012] The video signal to be displayed output by the signal conversion device sent by the video distributor is obtained; the medical imaging device is connected to the video distributor through the signal conversion device.

[0013] In one embodiment, obtaining a medical image and imaging parameters from a video signal includes:

[0014] identifying an image region and an information region from a frame image corresponding to a video signal;

[0015] acquiring a medical image from an image region of the frame image;

[0016] Imaging parameters are obtained from the information area of ​​the frame image.

[0017] In one embodiment, identifying the image region and the information region from a frame image corresponding to a video signal includes:

[0018] Obtaining preset configuration information, and identifying an image area and an information area from a frame image corresponding to a video signal according to the preset configuration information; or,

[0019] The frame image corresponding to the video signal is input into a preset marking model to obtain the image area and information area in the frame image.

[0020] In one embodiment, performing image fusion on a medical image and an ultrasound image of an ultrasound device based on imaging parameters to obtain a fused image includes:

[0021] Identify the position information of the probe of the ultrasound equipment from the medical image;

[0022] Based on imaging parameters and posture information, the medical image and the ultrasound image are fused to obtain a fused image.

[0023] In a second aspect, the present application further provides an image fusion processing device, comprising:

[0024] A first acquisition module is configured to acquire a video signal to be displayed; the video signal is generated based on a signal acquired by a medical imaging device;

[0025] A second acquisition module is used to acquire medical images and imaging parameters from video signals, or to acquire imaging parameters from medical imaging equipment;

[0026] The fusion module is used to fuse the medical image and the ultrasound image of the ultrasound device based on imaging parameters to obtain a fused image.

[0027] In a third aspect, the present application further provides an ultrasound device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the image fusion processing method in the first aspect when executing the computer program.

[0028] In a fourth aspect, the present application further provides an image fusion processing system, the system comprising: a medical imaging device, an ultrasound device, a video distributor, and a display, wherein the video distributor is communicatively connected to the medical imaging device, the ultrasound device, and the display respectively;

[0029] A medical imaging device configured to send a video signal containing a medical image to a video distributor, or to send the medical image to a signal conversion device, which in turn sends the video signal containing the medical image to the video distributor via the signal conversion device; or

[0030] The medical imaging device is further configured to transmit imaging parameters to the ultrasound device;

[0031] a video distributor for sending video signals to ultrasound equipment;

[0032] An ultrasound device, configured to obtain a medical image and imaging parameters from a video signal, or to obtain imaging parameters from a medical imaging device; perform image fusion on the medical image and an ultrasound image of the ultrasound device based on the imaging parameters to obtain a fused image, and send a fused video signal corresponding to the fused image to a video distributor;

[0033] a video distributor, further configured to send the converged video signal to a display;

[0034] The display is used to display the fused image according to the fused video signal.

[0035] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image fusion processing method in the first aspect.

[0036] In a sixth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the image fusion processing method in the first aspect.

[0037] The above-mentioned image fusion processing method, device, ultrasound equipment, image fusion processing system, storage medium and computer program product, the ultrasound equipment can obtain a video signal to be displayed based on the signal collected by the medical imaging equipment, and obtain the medical image and imaging parameters from the video signal, or directly obtain the imaging parameters from the medical imaging equipment; then, based on the imaging parameters, the medical image and the ultrasound image of the ultrasound equipment are image fused to obtain a fused image, so as to display the fused image. That is to say, by adopting the image fusion processing method proposed in the present application, any type of medical imaging device can be connected to the ultrasound device in the embodiment of the present application, so that the ultrasound device can obtain the video signal to be displayed generated by the signal collected by any medical imaging device, and extract the medical image and imaging parameters from the video signal, or the ultrasound device directly obtains the imaging parameters from the medical imaging device, and finally realizes the fusion processing and fusion display of the medical image and the ultrasound image; in other words, the ultrasound device proposed in the embodiment of the present application can be used in combination with any type of medical imaging device, thereby getting rid of the dependence on specific medical imaging devices. For scenarios where medical imaging devices have been deployed, it is only necessary to use the ultrasound device proposed in the embodiment of the present application to realize the fusion display of ultrasound images and medical images. This can not only solve the limitation problem of image fusion processing, greatly increase the application scenarios of image fusion processing, and improve the compatibility between ultrasound equipment and any medical imaging equipment, but also reduce the deployment cost and deployment difficulty of image fusion processing, and improve the universality of image fusion processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A diagram showing an application environment of an image fusion processing method in one embodiment;

[0040] Figure 2 1 is a flow chart of an image fusion processing method according to an embodiment;

[0041] Figure 3 is a flowchart of an image fusion processing method in another embodiment;

[0042] Figure 4 is a structural block diagram of an image fusion processing device in one embodiment;

[0043] Figure 5Schematic diagram of the structure of an image fusion processing system in one embodiment;

[0044] Figure 6 FIG. 1 is a diagram showing the internal structure of an ultrasonic device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0046] With the development of medical technology, surgical solutions have emerged that use multiple different imaging devices to perform auxiliary operations simultaneously. For example, when performing esophageal-related surgeries, esophageal ultrasound equipment and endoscopic subarachnography (DSA) equipment can be used. Among them, the esophageal ultrasound equipment can provide real-time 3D images of the patient's soft tissue in the surgical area during surgery, and the DSA equipment can provide real-time 2D images of the patient's bones, blood vessels, and surgical instruments and consumables in the surgical area during surgery.

[0047] In order to improve the doctor's viewing effect of images from different imaging devices, the real-time 3D image of the ultrasound device and the real-time 2D image of the DSA device can be fused and displayed. The fusion display of the two can effectively reduce the operation time and improve the surgical accuracy.

[0048] In related technologies, when performing image fusion processing and display, for example, when fusing ultrasound images with DSA images for display, it is necessary to simultaneously make technical changes to the DSA device and the ultrasound device in order to implement a fusion display solution for the images of the two based on a specific DSA device and a specific ultrasound device; wherein, the specific DSA device and the specific ultrasound device refer to the need to meet a specific communication interaction protocol between the two in order to achieve communication and data transmission. In one implementation, esophageal ultrasound can transmit real-time 3D ultrasound images to a fusion server via a dedicated 3D data transmission cable, and the DSA device transmits real-time 2D images to the fusion server via a proprietary protocol, so that the real-time 3D ultrasound image and the real-time 2D image are fused by the fusion server, and then the fused image is sent to the display for fusion image display. In another implementation, the esophageal ultrasound host is connected to the DSA device, and the DSA device sends the real-time 2D image to the esophageal ultrasound host via a proprietary protocol, and the esophageal ultrasound host performs image fusion processing on the real-time 3D ultrasound image and the real-time 2D image, and displays the fused image.

[0049] Therefore, for traditional image fusion processing solutions, image fusion and display can only be achieved for specific DSA devices and specific ultrasound devices after establishing proprietary connections and setting specific protocols. The limitations of image fusion processing are relatively large and cannot be universally applied to any ultrasound device and DSA device, or even the application of image fusion processing between any ultrasound device and any medical imaging device.

[0050] Based on this, the present invention proposes an image fusion processing method that can obtain the data required for image fusion and achieve real-time image fusion and display without establishing a proprietary connection or setting a specific protocol. The method proposed in the present invention is compatible with all medical imaging devices, including DSA devices, and has a wider range of applications.

[0051] The image fusion processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The ultrasound device 102 is connected to the medical imaging device 104. The ultrasound device 102 can obtain a video signal generated based on the signal collected by the medical imaging device 104, and obtain the medical image collected by the medical imaging device 104 from the video signal, thereby performing image fusion processing on the medical image and the ultrasound image of the ultrasound device 102 to obtain a fused image, and display the fused image. The medical imaging device 104 can directly output the video signal to be displayed based on the collected signal, or the medical imaging device can also output the medical image generated based on the collected signal to the signal conversion device, so that the signal conversion device can generate a video signal to be displayed based on the medical image and send it to the ultrasound device; in the second implementation method, the ultrasound device 102 can be indirectly connected to the medical imaging device 104 through the signal conversion device. In addition, the video signal to be displayed at least includes a medical image generated based on the signal collected by the medical imaging device 104.

[0052] Exemplarily, when performing image fusion processing, other required parameters or information can be obtained through data transmission via a wired / wireless connection between the ultrasound device 102 and the medical imaging device 104, such as the medical imaging device 104 can send the required other parameters or information to the ultrasound device 102, or can be obtained through a video signal, wherein the video signal includes not only the medical image but also other parameters or information required for image fusion.

[0053] In an exemplary embodiment, Figure 2 As shown, an image fusion processing method is provided, which is applied to Figure 1 The ultrasonic device in the embodiment is used as an example to illustrate the method, which includes the following steps 202 to 208.

[0054] Step 202: Acquire a video signal to be displayed.

[0055] The video signal is generated based on signals collected by a medical imaging device. The medical imaging device can be any type of imaging device in the medical field, including but not limited to digital subtraction angiography (DSA) equipment, computed tomography (CT) equipment, and magnetic resonance imaging (MR) equipment. These medical imaging devices are typically connected to a display to display the medical images scanned by the medical imaging device. For example, the display can be a main display within the surgical operating room, used to display medical images in real time. The display can also be a display in another operating room (e.g., an operating room outside the scanning room), used for operators to view medical images and determine whether to rescan or perform other control operations based on the medical images.

[0056] That is to say, in an optional implementation, for the medical imaging device, it can itself have the function of outputting a video signal. On this basis, the ultrasound device can obtain the video signal output by the medical imaging device for subsequent image fusion processing. For example, a video transmission line can be used to connect the medical imaging device and the ultrasound device, so that the video signal to be displayed containing the medical image collected by the medical imaging device is sent to the ultrasound device through the video transmission line. The medical image is generated after image reconstruction based on the signal collected by the medical imaging device. Among them, the video signal to be displayed includes at least a real-time medical image scanned by the medical imaging device. For example, the video signal can also include other relevant information, such as scanning parameter information, basic patient information, image-related information, etc., which is not specifically limited in the embodiment of the present application.

[0057] In another optional implementation, the medical imaging device can also be connected to the ultrasound device through a signal conversion device. The medical imaging device can have an image output function, such as sending a medical image generated after image reconstruction based on the collected signal to the signal conversion device, and generating a video signal to be displayed containing the medical image through the signal conversion device, and then sending the video signal to the ultrasound device; that is, the ultrasound device can obtain the video signal to be displayed output by the signal conversion device for subsequent image fusion processing.

[0058] Step 204: Acquire a medical image and imaging parameters from the video signal, or acquire imaging parameters from a medical imaging device.

[0059] The video signal at least includes a medical image. Of course, the video signal may also include imaging parameters. For example, the imaging parameters may include, but are not limited to, at least one of scanning parameters and image parameters. Scanning parameters may include, but are not limited to, scanning angle, scanning duration, and scanning bed height. Image parameters may include, but are not limited to, image type, image size, image pixels, and image resolution. In an optional implementation, in actual applications, imaging parameters related to image fusion may be extracted from the video signal. Of course, all or part of the imaging parameters may also be extracted for other functional applications or displays.

[0060] For example, when an ultrasound device acquires a video signal to be displayed, it can obtain medical images and imaging parameters from the video signal. The video signal may include multiple frame images, i.e., the video signal may consist of a sequence of frame images. Obtaining the medical image from the video signal may involve obtaining multiple medical images from the multiple frame images of the video signal. It should be noted that the imaging parameters corresponding to each medical image may be the same or different.

[0061] Exemplarily, for each frame image in a video signal, it may include an image area, or it may include an image area and an information area. For example, when the frame image only includes an image area, the imaging parameters may be displayed in the image area, i.e., superimposed on each frame image; when the frame image includes an image area and an information area, the imaging parameters may also be displayed outside the image area, such as in the information area. That is, the display area on the display corresponding to the video signal may include an image area and an information area, wherein the image area is used to display the medical image, and the information area is used to display the imaging parameters corresponding to the medical image. As an optional implementation, when the imaging parameters include variable parameters and invariant parameters, the invariant parameters may also be displayed in the information area, and the variable parameters may be displayed in the image area. That is, for each frame of medical image, the variable parameters corresponding to the frame of medical image may be displayed on the medical image, for example, the variable parameters may be superimposed on the target position of each frame of medical image. In the embodiments of the present application, there is no specific limitation on the display method between the medical image and the imaging parameters. Different content recognition algorithms may be used for different display methods to obtain the medical image and imaging parameters from the video signal. In the embodiments of the present application, there is no specific limitation on the content recognition algorithm.

[0062] In another optional implementation, other types of data transmission can also be achieved between the medical imaging device and the ultrasound device. For example, the medical imaging device can also send imaging parameters to the ultrasound device independently of the video signal, so that the ultrasound device can directly obtain the imaging parameters from the medical imaging device for subsequent image fusion processing.

[0063] Step 206 : Based on the imaging parameters, perform image fusion on the medical image and the ultrasound image of the ultrasound device to obtain a fused image.

[0064] The ultrasound image may be a real-time 3D image, and the medical image may be a real-time 2D image. Based on the imaging parameters of the medical image, the 2D medical image and the 3D ultrasound image may be fused to obtain a fused image.

[0065] For example, when performing image fusion processing, the position information of the ultrasound probe of the ultrasound device can also be identified from the medical image, and the ultrasound image and the medical image can be registered and fused in combination with the imaging parameters of the medical imaging device to obtain a fused image. As an optional implementation method, a preset image fusion algorithm can be used to perform registration and fusion processing on the ultrasound image and the medical image based on the position information of the ultrasound probe and the imaging parameters of the medical imaging device to obtain a fused image; wherein the preset image fusion algorithm can be a matrix transformation algorithm or a neural network model, which is not specifically limited in the embodiments of the present application.

[0066] Exemplarily, after obtaining the fused image, the fused image can be displayed, wherein the fused image can be displayed on the display of the ultrasound device, or the fused image can be sent to other displays via a wired connection to display the fused image, such as: the image signal or video signal corresponding to the real-time fused image can be sent to the display via a wired data transmission line to display the fused image on the display.

[0067] In the above-mentioned image fusion processing method, the ultrasound device can obtain a video signal to be displayed based on the signal collected by the medical imaging device, and obtain the medical image and imaging parameters from the video signal, or directly obtain the imaging parameters from the medical imaging device; then, based on the imaging parameters, the medical image and the ultrasound image of the ultrasound device are fused to obtain a fused image so as to display the fused image. That is to say, by adopting the image fusion processing method proposed in the present application, any type of medical imaging device can be connected to the ultrasound device in the embodiment of the present application, so that the ultrasound device can obtain the video signal to be displayed generated by the signal collected by any medical imaging device, and extract the medical image and imaging parameters from the video signal, or the ultrasound device directly obtains the imaging parameters from the medical imaging device, and finally realizes the fusion processing and fusion display of the medical image and the ultrasound image; in other words, the ultrasound device proposed in the embodiment of the present application can be used in combination with any type of medical imaging device, thereby getting rid of the dependence on specific medical imaging devices. For scenarios where medical imaging devices have been deployed, it is only necessary to use the ultrasound device proposed in the embodiment of the present application to realize the fusion display of ultrasound images and medical images. This can not only solve the limitation problem of image fusion processing, greatly increase the application scenarios of image fusion processing, and improve the compatibility between ultrasound equipment and any medical imaging equipment, but also reduce the deployment cost and deployment difficulty of image fusion processing, and improve the universality of image fusion processing.

[0068] In an exemplary embodiment, the image fusion scenario may also include a display. For example, the medical imaging device may be connected to the display and output a video signal containing a medical image to be displayed or a generated medical image to be displayed to the display for display. As an optional implementation, a video distributor, also referred to as a video signal distributor, video splitter, etc., may be configured between the medical imaging device and the display. The video distributor may also be connected to the ultrasound device. When the video distributor receives the video signal to be displayed, the video signal may be copied and sent to the ultrasound device so that the ultrasound device can obtain the video signal to be displayed generated based on the signal acquired by the medical imaging device.

[0069] In the case where the medical imaging device has the function of outputting a video signal, the medical imaging device can generate a medical image based on the signal collected by the medical imaging device, and generate a video signal to be displayed containing the medical image, so as to output the video signal to the video distributor. In the case where the medical imaging device does not have the function of outputting a video signal, the medical imaging device can also output the medical image generated based on the signal collected by the medical imaging device to a signal conversion device, and convert the medical image into a video signal to be displayed containing the medical image through the signal conversion device, thereby sending the video signal to be displayed to the video distributor. Based on this, obtaining the video signal to be displayed in the above-mentioned step 202 may include: obtaining the video signal to be displayed output by the medical imaging device sent by the video distributor, wherein the medical imaging device is connected to the video distributor.

[0070] In other words, by providing a video distributor between the medical imaging device and the display, the video distributor can transmit the video signal output by the medical imaging device to the ultrasound device, thereby enabling the ultrasound device to, upon receiving the video signal output by the medical imaging device, fuse and display the ultrasound image with the medical image based on the video signal. For example, the video distributor can also transmit the video signal output by the medical imaging device to the display, thereby displaying the medical image on the display.

[0071] Alternatively, obtaining the video signal to be displayed in the above step 202 may also include: obtaining the video signal to be displayed output by a signal conversion device sent by a video distributor; and connecting the medical imaging device to the video distributor via the signal conversion device.

[0072] That is, by providing a video distributor between the medical imaging device and the display, and also providing a signal conversion device between the medical imaging device and the video distributor, the signal conversion device first converts the medical image output by the medical imaging device into a video signal to be displayed, and then sends the video signal to be displayed to the video distributor, so that the video distributor can copy the video signal to be displayed and send it to the ultrasound device. For example, the video distributor can also transmit the video signal output by the signal conversion device to the display, so that the medical image can be displayed on the display.

[0073] Exemplarily, the above-mentioned fused image can also be sent to a display for display through the video distributor. Based on this, displaying the fused image in the above-mentioned step 208 can include: sending the fused video signal corresponding to the fused image to the display through the video distributor to display the fused image through the display.

[0074] That is to say, the display can display both medical images and fused images, or can display both medical images and fused images simultaneously to provide users with intraoperative reference.

[0075] In this embodiment, the video signal to be displayed output by the medical imaging device is sent to the ultrasound device through a video distributor, or the video signal to be displayed output by the signal conversion device is sent to the ultrasound device through a video distributor, wherein the signal conversion device can convert the medical image output by the medical imaging device into a video signal; then, after the ultrasound device performs image fusion on the medical image and the ultrasound image to obtain a fused image, the fused video signal corresponding to the fused image is sent to the display through the video distributor, and finally the fused image is displayed on the display. The image fusion processing scheme proposed in the embodiment of the present application is simple to deploy, easy to operate, and has low deployment cost and low deployment difficulty, which can improve the universality of image fusion processing.

[0076] In an exemplary embodiment, Figure 3 As shown, the above step 204 of "obtaining medical images and imaging parameters from video signals" may include steps 302 to 306. Among them:

[0077] Step 302: Identify an image area and an information area from a frame image corresponding to a video signal.

[0078] For example, when displaying a medical image captured by a medical imaging device based on the video signal, the corresponding display interface may include an image area and an information area, wherein the image area is used to display the medical image, and the information area is used to display imaging parameters and other related information, such as patient information. In this case, when identifying the medical image and imaging parameters from the video signal, the image area and information area may be first identified from each frame image corresponding to the video signal. It should be noted that the frame image here may include an interface image corresponding to the display interface.

[0079] As an optional implementation method, taking a single frame image as an example, when identifying the image area and the information area from the frame image, the preset configuration information can be obtained first, and based on the preset configuration information, the image area and the information area can be identified from the frame image corresponding to the video signal; wherein, the preset configuration information can include the size and position of the image area, as well as the size and position of the information area, so that the image area and the information area can be accurately segmented from the frame image based on the preset configuration information. It should be noted that the preset configuration information corresponding to different medical imaging devices can be the same or different. When obtaining the preset configuration information corresponding to the medical imaging device, it can be obtained from the medical imaging device or from a preset database, etc., and the embodiments of the present application do not make specific limitations on this. wherein, the preset database can be a proprietary database of a hospital or a proprietary database of a manufacturer, etc.

[0080] As another optional implementation, when identifying image regions and information regions from a frame image, the frame image corresponding to the video signal can also be input into a preset labeling model for region identification and labeling to obtain the image region and information region in the frame image. That is, in this example, a region identification model, i.e., a preset labeling model, can be pre-trained to identify and label image regions and information regions in an image. In this case, each frame image in the video signal can be input into the preset labeling model for region labeling to obtain the image region and information region in each frame image.

[0081] Step 304: Acquire a medical image from the image region of the frame image.

[0082] For example, the image area in the frame image can be cut out / cropped to obtain a medical image corresponding to the image area, or the regional image obtained after cutting out / cropping can be post-processed to obtain a medical image within the image area, wherein the post-processing may include but is not limited to border removal processing, etc., and the embodiments of the present application do not specifically limit this.

[0083] Step 306: Obtain imaging parameters from the information area of ​​the frame image.

[0084] For example, when an information region in a frame image is identified, imaging parameters can be identified from the information region of the frame image. For example, an optical character recognition (OCR) algorithm can be used to identify parameters in the information region of the frame image, thereby obtaining imaging parameters. For example, when the information region contains multiple parameters, all parameters can be identified first. Then, based on the type of imaging parameters, imaging parameters related to image registration and fusion can be obtained from all identified parameters for use in subsequent image fusion processing.

[0085] In this embodiment, the image region and information region are first identified from the frame image corresponding to the video signal. Then, the medical image is obtained from the image region of the frame image, and the imaging parameters are obtained from the information region of the frame image. The method proposed in the embodiment of the present application can realize the parsing and processing of the video signal and obtain the data required for image fusion, namely the medical image and imaging parameters, from the video signal, thereby realizing image fusion processing and improving the efficiency and accuracy of image fusion processing.

[0086] In an exemplary embodiment, a complete embodiment of image fusion processing is provided, taking the medical imaging device as a DSA device as an example, including the following steps, wherein:

[0087] Step 1: The ultrasound device obtains the DSA video signal through a video signal distributor.

[0088] Step 2: Use the OCR algorithm to identify the text information in the acquired DSA video signal and parse out the required parameter information, such as imaging parameters such as rack angle and bed height.

[0089] Step 3: Using an image recognition algorithm, identify the DSA real-time fluoroscopic image area in the DSA video signal and obtain a DSA fluoroscopic image.

[0090] Step 4: Identify the position and posture of the ultrasound probe in the DSA fluoroscopic image, that is, the posture information of the ultrasound probe, through a preset algorithm.

[0091] Step 5: Based on the identified position and posture of the ultrasound probe, the ultrasound image and the DSA fluoroscopic image are fused to obtain a fused image.

[0092] Step 6: Generate a fused image signal corresponding to the fused image, and send it to a display through a video signal distributor to display the fused image.

[0093] The image fusion processing method proposed in this embodiment does not require establishing a dedicated connection to obtain the required data. Instead, it directly obtains the video signal output by the DSA to the display, extracts the necessary information from the video signal, and thus achieves real-time image fusion. The advantage of this method is that all DSA devices have the function of outputting video signals. Therefore, the solution proposed in this embodiment of the application is adaptable to all DSA devices and has a wider range of applicability.

[0094] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0095] Based on the same inventive concept, the embodiments of the present application also provide an image fusion processing device for implementing the aforementioned image fusion processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following embodiments of the image fusion processing device can be found in the above-mentioned limitations on the image fusion processing method and will not be repeated here.

[0096] In an exemplary embodiment, Figure 4 As shown, an image fusion processing device is provided, comprising: a first acquisition module 402, a second acquisition module 404 and a fusion module 406, wherein:

[0097] The first acquisition module 402 is used to acquire a video signal to be displayed; the video signal is generated based on a signal collected by a medical imaging device.

[0098] The second acquisition module 404 is configured to acquire medical images and imaging parameters from video signals, or acquire imaging parameters from medical imaging equipment.

[0099] The fusion module 406 is configured to perform image fusion on the medical image and the ultrasound image of the ultrasound device based on the imaging parameters to obtain a fused image.

[0100] In one embodiment, the first acquisition module 402 is specifically used to acquire the video signal to be displayed output by the medical imaging device sent by the video distributor; the medical imaging device is connected to the video distributor; or, to acquire the video signal to be displayed output by the signal conversion device sent by the video distributor; the medical imaging device is connected to the video distributor through the signal conversion device.

[0101] In one embodiment, the second acquisition module 404 includes:

[0102] A first recognition unit is used to recognize an image area and an information area from a frame image corresponding to a video signal;

[0103] A first acquiring unit is configured to acquire a medical image from an image region of a frame image;

[0104] The second acquiring unit is configured to acquire imaging parameters from the information area of ​​the frame image.

[0105] In one embodiment, the recognition unit is specifically used to obtain preset configuration information, and identify the image area and information area from the frame image corresponding to the video signal according to the preset configuration information; or, input the frame image corresponding to the video signal into a preset marking model to obtain the image area and information area in the frame image.

[0106] In one embodiment, the fusion module 406 includes:

[0107] A second recognition unit is used to recognize the position information of the probe of the ultrasound device from the medical image;

[0108] The fusion unit is used to fuse the medical image and the ultrasound image based on imaging parameters and posture information to obtain a fused image.

[0109] Each module in the above-mentioned image fusion processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0110] In an exemplary embodiment, an image fusion processing system is provided, which includes: a medical imaging device 51, an ultrasound device 52, a video distributor 53 and a display 54, and the video distributor 53 is communicatively connected to the medical imaging device 51, the ultrasound device 52 and the display 54 respectively.

[0111] The medical imaging device 51 is configured to transmit a video signal containing a medical image to a video distributor 53, or transmit the medical image to the video distributor 53, or transmit the medical image to a signal conversion device (not shown), which transmits the video signal containing the medical image to the video distributor 53 via the signal conversion device. For example, the video distributor 53 may be integrated with a signal conversion device, which converts the medical image transmitted from the medical imaging device 51 into a video signal to be displayed. Alternatively, the video distributor 53 and the signal conversion device may be independently provided, with the signal conversion device connecting the medical imaging device 51 and the video distributor 53, and the signal conversion device converts the medical image transmitted from the medical imaging device 51 into a video signal to be displayed, which is then transmitted to the video distributor 53.

[0112] The medical imaging device 51 is further configured to transmit imaging parameters to the ultrasound device 52. For example, the medical imaging device 51 may be in communication with the ultrasound device 52 (not shown in the figure) so that the ultrasound device 52 can directly obtain the imaging parameters from the medical imaging device 51, thereby improving the efficiency of obtaining the imaging parameters.

[0113] The video distributor 53 is used to send the video signal to the ultrasound device 52 .

[0114] The ultrasound device 52 is used to obtain medical images and imaging parameters from the video signal, or to obtain imaging parameters from the medical imaging device 53; based on the imaging parameters, the medical image and the ultrasound image of the ultrasound device are fused to obtain a fused image, and the fused video signal corresponding to the fused image is sent to the video distributor 53.

[0115] The video distributor 53 is further configured to send the fused video signal to the display 54 .

[0116] The display 54 is used to display the fused image according to the fused video signal.

[0117] By using the image fusion processing system proposed in the embodiment of the present application, any type of medical imaging device can be connected to the ultrasound device in the embodiment of the present application, so that the ultrasound device can obtain the video signal to be displayed based on the signal collected by the medical imaging device, and extract the medical image and imaging parameters from the video signal, or directly obtain the imaging parameters from the medical imaging device, and finally realize the fusion processing and display of the medical image and the ultrasound image; in other words, the ultrasound device proposed in the embodiment of the present application can be used in combination with any type of medical imaging device, thereby getting rid of the dependence on specific medical imaging devices. For scenarios where medical imaging devices have been deployed, it is only necessary to use the ultrasound device proposed in the embodiment of the present application to realize the fusion display of ultrasound images and medical images. This can not only solve the limitations of image fusion processing, greatly increase the application scenarios of image fusion processing, and improve the compatibility between ultrasound equipment and any medical imaging equipment, but also reduce the deployment cost and difficulty of image fusion processing, and improve the universality of image fusion processing.

[0118] In addition, it should be noted that the image fusion processing method for ultrasound equipment proposed in the embodiment of the present application can also be applied to other medical imaging equipment, or other image fusion equipment. For example, an ultrasound video signal carrying an ultrasound image can be sent to an image fusion device through a video distributor, and a medical video signal carrying a medical image can be sent to an image fusion device through a video distributor. Then, the ultrasound image is obtained from the ultrasound video signal, and the medical image and imaging parameters are obtained from the medical video signal through the image fusion device. Based on the imaging parameters, the ultrasound image and the medical image are subjected to image fusion processing to obtain a fused image. Finally, a fused video signal corresponding to the fused image is generated, and the fused video signal is sent to a display to display the fused image.

[0119] Alternatively, the video signal of medical imaging device A is sent to medical imaging device B, medical imaging device B obtains medical image A from the video signal of medical imaging device A, and performs image fusion processing on medical image A and medical image B to obtain a fused image, and finally sends the fused video signal corresponding to the fused image to the display to display the fused image.

[0120] The image fusion processing method proposed in the embodiment of the present application can be applied to image fusion processing and display between any type of medical imaging equipment, has high universality and a wider range of applications.

[0121] In an exemplary embodiment, an ultrasound device is provided, the internal structure of which can be shown as follows: Figure 6As shown. The ultrasound device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the ultrasound device is used to provide computing and control capabilities. The memory of the ultrasound device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the ultrasound device is used to exchange information between the processor and external devices. The communication interface of the ultrasound device is used to communicate with external terminals via wired or wireless communication, and the wireless communication can be achieved via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements an image fusion processing method. The display unit of the ultrasound device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the ultrasound device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the ultrasound device housing, or an external keyboard, touchpad or mouse.

[0122] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the ultrasonic equipment to which the scheme of the present application is applied. The specific ultrasonic equipment may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] In an exemplary embodiment, an ultrasound device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0124] Acquire a video signal to be displayed; the video signal is generated based on a signal acquired by a medical imaging device;

[0125] Acquiring medical images and imaging parameters from video signals, or acquiring imaging parameters from medical imaging equipment;

[0126] Based on the imaging parameters, the medical image and the ultrasound image of the ultrasound device are fused to obtain a fused image.

[0127] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0128] Acquire a video signal to be displayed output by a medical imaging device sent by a video distributor; connect the medical imaging device to the video distributor; or,

[0129] The video signal to be displayed output by the signal conversion device sent by the video distributor is obtained; the medical imaging device is connected to the video distributor through the signal conversion device.

[0130] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0131] identifying an image region and an information region from a frame image corresponding to a video signal;

[0132] acquiring a medical image from an image region of the frame image;

[0133] Imaging parameters are obtained from the information area of ​​the frame image.

[0134] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0135] Obtaining preset configuration information, and identifying an image area and an information area from a frame image corresponding to a video signal according to the preset configuration information; or,

[0136] The frame image corresponding to the video signal is input into a preset marking model to obtain the image area and information area in the frame image.

[0137] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0138] Identify the position information of the probe of the ultrasound equipment from the medical image;

[0139] Based on imaging parameters and posture information, the medical image and the ultrasound image are fused to obtain a fused image.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0141] Acquire a video signal to be displayed; the video signal is generated based on a signal acquired by a medical imaging device;

[0142] Acquiring medical images and imaging parameters from video signals, or acquiring imaging parameters from medical imaging equipment;

[0143] Based on the imaging parameters, the medical image and the ultrasound image of the ultrasound device are fused to obtain a fused image.

[0144] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0145] Acquire a video signal to be displayed output by a medical imaging device sent by a video distributor; connect the medical imaging device to the video distributor; or,

[0146] The video signal to be displayed output by the signal conversion device sent by the video distributor is obtained; the medical imaging device is connected to the video distributor through the signal conversion device.

[0147] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0148] identifying an image region and an information region from a frame image corresponding to a video signal;

[0149] acquiring a medical image from an image region of the frame image;

[0150] Imaging parameters are obtained from the information area of ​​the frame image.

[0151] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0152] Obtaining preset configuration information, and identifying an image area and an information area from a frame image corresponding to a video signal according to the preset configuration information; or,

[0153] The frame image corresponding to the video signal is input into a preset marking model to obtain the image area and information area in the frame image.

[0154] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0155] Identify the position information of the probe of the ultrasound equipment from the medical image;

[0156] Based on imaging parameters and posture information, the medical image and the ultrasound image are fused to obtain a fused image.

[0157] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the image fusion processing method in any of the above embodiments are implemented.

[0158] Acquire a video signal to be displayed; the video signal is generated based on a signal acquired by a medical imaging device;

[0159] Acquiring medical images and imaging parameters from video signals, or acquiring imaging parameters from medical imaging equipment;

[0160] Based on the imaging parameters, the medical image and the ultrasound image of the ultrasound device are fused to obtain a fused image.

[0161] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0162] Acquire a video signal to be displayed output by a medical imaging device sent by a video distributor; connect the medical imaging device to the video distributor; or,

[0163] The video signal to be displayed output by the signal conversion device sent by the video distributor is obtained; the medical imaging device is connected to the video distributor through the signal conversion device.

[0164] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0165] identifying an image region and an information region from a frame image corresponding to a video signal;

[0166] acquiring a medical image from an image region of the frame image;

[0167] Imaging parameters are obtained from the information area of ​​the frame image.

[0168] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0169] Obtaining preset configuration information, and identifying an image area and an information area from a frame image corresponding to a video signal according to the preset configuration information; or,

[0170] The frame image corresponding to the video signal is input into a preset marking model to obtain the image area and information area in the frame image.

[0171] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0172] Identify the position information of the probe of the ultrasound equipment from the medical image;

[0173] Based on imaging parameters and posture information, the medical image and the ultrasound image are fused to obtain a fused image.

[0174] It should be noted that the data involved in this application (including but not limited to data used for analysis, storage, display, etc.) are all information and data fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0175] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0176] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image fusion processing method, characterized in that: Applied to ultrasound equipment, the method comprises: Acquire a video signal to be displayed; the video signal is generated based on a signal acquired by a medical imaging device; Acquiring a medical image and imaging parameters from the video signal, or acquiring imaging parameters from the medical imaging device; Based on the imaging parameters, the medical image and the ultrasound image of the ultrasound device are fused to obtain a fused image.

2. The method according to claim 1, characterized in that The obtaining of the video signal to be displayed includes: Acquiring a video signal to be displayed output by the medical imaging device sent by a video distributor; connecting the medical imaging device to the video distributor; or, The video signal to be displayed output by the signal conversion device sent by the video distributor is obtained; the medical imaging device is connected to the video distributor through the signal conversion device.

3. The method according to claim 1, characterized in that The acquiring of medical images and imaging parameters from the video signal includes: identifying an image region and an information region from a frame image corresponding to the video signal; acquiring the medical image from the image region of the frame image; The imaging parameters are obtained from the information area of ​​the frame image.

4. The method according to claim 3, characterized in that The identifying of the image area and the information area from the frame image corresponding to the video signal includes: Obtaining preset configuration information, and identifying an image area and an information area from a frame image corresponding to the video signal according to the preset configuration information; or, The frame image corresponding to the video signal is input into a preset marking model to obtain an image area and an information area in the frame image.

5. The method according to claim 1, wherein The step of fusing the medical image and the ultrasound image of the ultrasound device based on the imaging parameters to obtain a fused image includes: Identifying the position information of the probe of the ultrasound device from the medical image; Based on the imaging parameters and the posture information, the medical image and the ultrasound image are fused to obtain a fused image.

6. An image fusion processing device, characterized in that: Applied to ultrasonic equipment, the device comprises: A first acquisition module is configured to acquire a video signal to be displayed; the video signal is generated based on a signal acquired by a medical imaging device; A second acquisition module is configured to acquire a medical image and imaging parameters from the video signal, or to acquire imaging parameters from the medical imaging device; A fusion module is used to perform image fusion on the medical image and the ultrasound image of the ultrasound device based on the imaging parameters to obtain a fused image.

7. An ultrasound device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. An image fusion processing system, characterized in that: The system comprises: a medical imaging device, an ultrasound device, a video distributor and a display, wherein the video distributor is communicatively connected to the medical imaging device, the ultrasound device and the display respectively; The medical imaging device is configured to send a video signal containing a medical image to the video distributor, or send the medical image to a signal conversion device, which then sends the video signal containing the medical image to the video distributor via the signal conversion device; or The medical imaging device is further configured to send imaging parameters to the ultrasound device; The video distributor is used to send the video signal to the ultrasound device; The ultrasound device is configured to obtain the medical image and the imaging parameters from the video signal, or to obtain the imaging parameters from the medical imaging device; perform image fusion on the medical image and the ultrasound image of the ultrasound device based on the imaging parameters to obtain a fused image, and send a fused video signal corresponding to the fused image to the video distributor; The video distributor is further configured to send the fused video signal to the display; The display is used to display the fused image according to the fused video signal.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images