A multi-modal image fusion system and image fusion method

Abstract

The present invention discloses a modality image fusion system and an image fusion method. The system includes: a video controller, a medical imaging device, a binocular camera, a human body surface reference plane positioning module, a handheld device positioning module, a display, a touch screen, and a keyboard and mouse. The video controller obtains the image fusion reference plane in advance according to the placement positions of each first positioning ball in the human body surface reference plane positioning module, obtains the corresponding medical images to be fused according to the acquired image selection instruction, and fuses each medical image to be fused on the image fusion reference plane based on the relative positions of each first positioning ball on the image fusion reference plane to obtain a multimodality fusion image. The present invention realizes the association of each single-modality image by fusing each medical image to be fused of a single-modality image, so that medical staff can diagnose diseases according to the multimodality fusion image, improves the efficiency of defining the scope of surgery, and shortens the time spent in the diagnosis process.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and more specifically, to a multimodal image fusion system and an image fusion method. Background Art

[0002] Medical imaging not only expands the scope of human body examination and improves the level of diagnosis, but also can treat certain diseases. In the traditional solution, each medical imaging device in the operating room displays image information separately on its own monitor. Therefore, during the operation, the doctor needs to look around, or even run to a certain imaging device to view the image information of the device, for example, to view the patient's ECG information on the electrocardiograph; to view the endoscope information inside the body on the endoscope; to view the image information on the C-arm; to view the detailed information on the microscope, etc., which greatly wastes the operation time and increases the risk of infection for the patient.

[0003] To solve this problem, some companies have developed multimodal imaging chain technology, which connects all single-modal imaging sources in the operating room together to form a control link, and finally displays them on a medical display. By converting single-modal display to multimodal display, the time doctors spend viewing imaging information can be greatly reduced.

[0004] Since the existing multimodal display shows multiple single-modal images on the same display screen, each single-modal image is independent of each other and has no correlation. Therefore, medical staff need to diagnose the disease based on experience and comprehensive analysis of each single-modal image. As a result, the efficiency of defining the scope of surgery is low and the diagnosis process takes a long time. Summary of the invention

[0005] In view of this, the present invention discloses a modality image fusion system and an image fusion method to achieve the association of various single-modality images, so that medical personnel can diagnose diseases based on multi-modality fusion images, greatly improve the efficiency of surgical scope definition, and shorten the time spent on the diagnosis process.

[0006] A multimodal image fusion system, comprising: a video controller, a medical imaging device, a binocular camera, a human body surface reference plane positioning module, a handheld device positioning module, a display, a touch screen, and a keyboard and mouse;

[0007] The human body surface reference plane positioning module includes a plurality of positioning balls, each of which is recorded as a first positioning ball, each of which is used to be placed at a preset position, and each of which has a unique corresponding ball code, and the preset position is: on the human body surface of the surgical part, near the entrance set for the surgery;

[0008] The handheld device positioning module includes at least one positioning ball, which is referred to as a second positioning ball. The second positioning ball is used to be placed on a handle for ultrasonic image acquisition, and each of the second positioning balls has a unique corresponding ball code.

[0009] The binocular camera is used to collect a ball position image containing position information of the positioning ball;

[0010] The touch screen is used to input a mode selection instruction for a surgical mode;

[0011] The keyboard and mouse are used to input image selection instructions for the medical images to be fused;

[0012] The display is used to display a multimodal fusion image;

[0013] The video controller is respectively connected to the medical imaging device, the binocular camera, the display, the touch screen and the keyboard and mouse;

[0014] The video controller is used to obtain the image selection instruction, select each medical image to be fused corresponding to the image selection instruction from each stored medical image, obtain the image fusion reference plane obtained by preprocessing, and fuse each medical image to be fused on the image fusion reference plane according to the relative positions of each of the first positioning balls on the image fusion reference plane to obtain the multimodal fusion image;

[0015] The stored medical images are: medical images acquired by medical imaging devices corresponding to the pre-selected surgical mode;

[0016] The image fusion reference plane is obtained by processing the ball position image containing all the first positioning balls at the preset positions collected by the binocular camera.

[0017] Optionally, the medical imaging equipment includes: a C-arm, an ultrasound device, an endoscope, a microscope and a PACS (image archiving and communication system).

[0018] A multimodal image fusion method is applied to the video controller described above, and the fusion method comprises:

[0019] Obtaining an image selection instruction for the medical image to be fused;

[0020] Selecting each medical image to be fused corresponding to the image selection instruction from each stored medical image, wherein each stored medical image is: a medical image acquired by each medical imaging device corresponding to the pre-selected surgical mode;

[0021] Acquire the image fusion reference plane obtained by preprocessing, wherein the image fusion reference plane is obtained by processing the ball position image containing all the first positioning balls at the preset position collected by the binocular camera, and the preset position is: the position on the human body surface of the surgical part, near the entrance set for the surgical operation;

[0022] According to the relative positions of the first positioning balls on the image fusion reference plane, the medical images to be fused are fused on the image fusion reference plane to obtain a multimodal fusion image.

[0023] Optionally, the process of acquiring the image fusion reference plane is:

[0024] Acquire the actual position information of each first positioning ball in the human body surface reference plane positioning module placed at a preset position, wherein the preset position is: a position on the human body surface of the surgical part, near the entrance set for the surgical operation, and the actual position information includes: the ball number and volume size of each first positioning ball, and the shortest distance between each first positioning ball;

[0025] Obtaining a ball position image collected by a binocular camera containing position information of all first positioned balls;

[0026] According to the actual position information of each of the first positioning balls, the position of each of the first positioning balls in the ball position image is adjusted so that the position of each of the first positioning balls in the ball position image is consistent with the actual position;

[0027] The adjusted ball position image is determined as the image fusion reference plane.

[0028] Optionally, also include:

[0029] The multimodal fusion image is output to a display for display.

[0030] Optionally, fusing the medical images to be fused on the image fusion reference plane according to the relative positions of the first positioning balls on the image fusion reference plane to obtain the multimodal fusion image specifically includes:

[0031] Get 3D human anatomy images;

[0032] According to the relative positions of each first positioning sphere on the image fusion reference plane, the 3D human anatomical image and each medical image to be fused are fused on the image fusion reference plane to obtain the multimodal fusion image, wherein the 3D human anatomical image is the bottom layer image in the multimodal fusion image.

[0033] Optionally, also include:

[0034] The 3D human anatomy image, the multimodal fusion image and each of the medical images to be fused are output to a split screen of a display for display.

[0035] Optionally, before obtaining the image selection instruction for the medical image to be fused, the method further includes:

[0036] Acquire a mode selection instruction for a surgical mode, and determine each medical imaging device corresponding to the surgical mode according to the mode selection instruction;

[0037] The medical images collected by each of the medical imaging devices are acquired and stored.

[0038] From the above technical scheme, it can be known that the present invention discloses a modal image fusion system and an image fusion method. The fusion system includes: a video controller, a medical imaging device, a binocular camera, a human body surface reference plane positioning module, a handheld device positioning module, a display, a touch screen and a keyboard and mouse. Before the operation, each first positioning ball in the human body surface reference plane positioning module is placed on the human body surface of the surgical part. At a position near the entrance set by the operation, the binocular camera collects a ball position image containing the position information of all the first positioning balls. The video controller processes the ball position image to obtain an image fusion reference plane. The video controller obtains the corresponding medical images to be fused according to the acquired image selection instruction. According to the relative positions of each first positioning ball on the image fusion reference plane, each medical image to be fused is fused on the image fusion reference plane to obtain a multimodal fusion image. The present invention realizes the association of each single-modal image by fusing each medical image to be fused of the single-modal image, so that medical staff can diagnose diseases according to the multimodal fusion image, greatly improving the efficiency of defining the scope of surgery and shortening the time spent in the diagnosis process. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without paying creative work.

[0040] Figure 1 A schematic diagram of the structure of a multimodal image fusion system disclosed in an embodiment of the present invention;

[0041] Figure 2 A flow chart of a multimodal image fusion method disclosed in an embodiment of the present invention;

[0042] Figure 3 The present invention is a flowchart of a method for obtaining an image fusion reference plane disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The embodiment of the present invention discloses a modality image fusion system and an image fusion method. The system includes: a video controller, a medical imaging device, a binocular camera, a human body surface reference plane positioning module, a handheld device positioning module, a display, a touch screen, and a keyboard and mouse. Before surgery, each first positioning ball in the human body surface reference plane positioning module is placed on the human body surface of the surgical part. At a position near the entrance set for surgery, the binocular camera collects a ball position image containing the position information of all first positioning balls. The video controller processes the ball position image to obtain an image fusion reference plane. The video controller obtains the corresponding medical images to be fused according to the acquired image selection instruction. According to the relative positions of each first positioning ball on the image fusion reference plane, each medical image to be fused is fused on the image fusion reference plane to obtain a multimodal fusion image. The present invention realizes associating each single-modality image by fusing each medical image to be fused of a single-modality image, so that medical staff can diagnose diseases according to the multimodal fusion image, greatly improving the efficiency of defining the scope of surgery and shortening the time spent in the diagnosis process.

[0045] See also Figure 1 , a schematic diagram of the structure of a multimodal image fusion system disclosed in an embodiment of the present invention, the system includes: a video controller 11, a medical imaging device 12, a binocular camera 13, a human body surface reference plane positioning module 14, a handheld device positioning module 15, a display 16, a touch screen 17 and a keyboard and mouse 18.

[0046] in:

[0047] The medical imaging device 12 is connected to the video controller 11 via HDMI (High Definition Multimedia Interface). The medical imaging device 12 includes but is not limited to a C-arm (DSA device), an ultrasound device, an endoscope device, a microscope device and a PACS (Picture Archiving and Communication Systems). The full name of DSA in English is Digital subtraction angiography, which is explained in Chinese as Digital subtraction angiography.

[0048] The human body surface reference plane positioning module 14 includes multiple positioning balls. For the convenience of description, each positioning ball contained in the human body surface reference plane positioning module 14 is recorded as a first positioning ball. Each of the first positioning balls is used to be placed in a preset position, and the preset position is: on the human body surface of the surgical part, near the entrance set for the surgery.

[0049] For example, the human body surface reference plane positioning module 14 includes three positioning balls, namely positioning ball 1, positioning ball 2 and positioning ball 3. In actual application, positioning ball 1, positioning ball 2 and positioning ball 3 are placed on the human body surface of the surgical part near the entrance of the surgical setting, and the distribution of positioning ball 1, positioning ball 2 and positioning ball 3 is a non-isosceles triangle.

[0050] It should be noted that each positioning ball in the human body surface reference plane positioning module 14 has a unique corresponding ball number.

[0051] The handheld device positioning module 15 includes at least one positioning ball. For the convenience of description, each positioning ball included in the handheld device positioning module 15 is recorded as a second positioning ball, and the second positioning ball is used to be placed on the handle of ultrasound image acquisition.

[0052] The data of the second positioning ball included in the handheld device positioning module 15 is determined according to actual needs, for example, including: positioning ball 4 and positioning ball 5 (see Figure 1 ).

[0053] It should be noted that each second positioning ball in the handheld device positioning module 15 has a unique corresponding ball number.

[0054] The binocular camera 13 is connected to the video controller 11 through a network port. The binocular camera 13 is installed at the edge of the bed, which does not need to be changed during the entire operation. It can collect the positions of each positioning ball in the human body surface reference plane positioning module 14 and the handheld device positioning module 15, and will not be blocked by medical staff during multimodal image fusion.

[0055] The binocular camera 13 is used to collect the ball position image containing the position information of the positioning ball.

[0056] In practical applications, the binocular camera 13 can collect a ball position image containing the position information of all first positioning balls, or collect a ball position image containing the position information of all first positioning balls and all second positioning balls.

[0057] The display 16 is connected to the video controller 11 via HDMI, and is used to display the multimodal fusion image. To facilitate medical staff to view the multimodal image fusion image, the display 16 can be a display with a larger screen, such as a 55-inch LCD display.

[0058] The display 16 in this embodiment can be displayed as multiple split screens, such as 4 split screens or 6 split screens.

[0059] Tiled display (ie, non-overlapping mode) and image fusion mode. The display mode of the display screen 16 can be selected by clicking a corresponding button on the touch screen 17.

[0060] The touch screen 17 is connected to the video controller 11 via a serial port, and the touch screen 17 is used to control the video controller 11 and display a human-machine interface. Specifically, the touch screen 17 is used to input a mode selection instruction for a surgical mode, wherein different surgical modes correspond to the same or different medical imaging devices 12.

[0061] The keyboard and mouse 18 are connected to the video controller 11 via a USB (Universal Serial Bus) and are used to input image selection instructions for the medical images to be fused.

[0062] The video controller 11 is used to obtain an image selection instruction for the medical images to be fused, select the medical images to be fused corresponding to the image selection instruction from the stored medical images, obtain the image fusion reference plane obtained by preprocessing, and fuse the medical images to be fused on the image fusion reference plane according to the relative positions of the first positioning balls on the image fusion reference plane to obtain a multimodal fusion image.

[0063] The medical images stored in the video controller are: medical images acquired by various medical imaging devices corresponding to the pre-selected surgical mode.

[0064] The image fusion reference plane is obtained by processing the ball position image containing all the first positioning balls at the preset positions collected by the binocular camera.

[0065] In summary, the multimodal image fusion system disclosed in the present invention includes: a video controller 11, a medical imaging device 12, a binocular camera 13, a human body surface reference plane positioning module 14, a handheld device positioning module 15, a display 16, a touch screen 17 and a keyboard and mouse 18. Before the operation, each first positioning ball in the human body surface reference plane positioning module 14 is placed on the human body surface of the surgical part. At a position near the entrance set for the operation, the binocular camera 13 collects a ball position image containing the position information of all first positioning balls. The video controller 11 processes the ball position image to obtain an image fusion reference plane. The video controller 11 obtains the corresponding medical images to be fused according to the acquired image selection instruction. According to the relative positions of each first positioning ball on the image fusion reference plane, each medical image to be fused is fused on the image fusion reference plane to obtain a multimodal fusion image. The present invention realizes the association of each single-modal image by fusing each medical image to be fused of a single-modal image, so that medical staff can diagnose diseases according to the multimodal fusion image, greatly improving the efficiency of defining the scope of surgery and shortening the time spent in the diagnosis process.

[0066] See also Figure 2 , a flow chart of a multimodal image fusion method disclosed in an embodiment of the present invention, the method is applied to Figure 1 In the video controller of the illustrated embodiment, the method comprises:

[0067] Step S101, obtaining an image selection instruction for a medical image to be fused;

[0068] It should be noted that after receiving the image selection instruction, the video controller will automatically enter the image fusion mode.

[0069] In this embodiment, the doctor selects the medical images to be fused by using the mouse and keyboard before the operation, for example, the fusion of a C-arm image and a B-ultrasound image.

[0070] Of course, during the operation, the doctor can also select the medical images to be fused by using the mouse and keyboard. The time point for selecting the medical images to be fused depends on actual needs, and the present invention does not limit this.

[0071] Step S102, selecting each medical image to be fused corresponding to the image selection instruction from each stored medical image;

[0072] The medical images stored in the video controller are: medical images acquired by various medical imaging devices corresponding to the pre-selected surgical mode.

[0073] In practical applications, the medical images to be fused may include: C-arm images (anteroposterior images of the human body, lateral images of the human body), G-arm images (anteroposterior images of the human body, lateral images of the human body), B-ultrasound images during surgery, and endoscopic images during surgery.

[0074] Step S103, obtaining the image fusion reference plane obtained by preprocessing;

[0075] For details, see Figure 3 , a flow chart of a method for obtaining an image fusion reference plane disclosed in an embodiment of the present invention, the method is applied to a video controller, and the method includes:

[0076] Step S201, obtaining actual position information of each first positioning ball in a human body surface reference plane positioning module placed at a preset position;

[0077] The preset position is: on the surface of the human body of the surgical part, near the entrance set for the surgery.

[0078] The actual position information includes: the ball number and volume of each first positioning ball, and the shortest distance between each first positioning ball.

[0079] It should be noted that, in this embodiment, the volumes of the first positioning balls are equal.

[0080] The reason for obtaining the volume of the first positioning balls is that the sizes of the first positioning balls presented on the image captured by the binocular camera are different due to the distance difference between the first positioning balls and the binocular camera. The present invention compares the pixel grayscale area size of each first positioning ball calculated according to the image with the characteristic parameters of the actual first positioning ball to obtain the distance-related information between the binocular camera and each first positioning ball, and uses this as a correction value when the binocular camera calculates the distance, thereby improving the distance calculation accuracy.

[0081] In this embodiment, after each of the first positioning balls is placed at a preset position, the medical staff can manually measure the shortest distance between each of the first positioning balls and input the measured shortest distance into the video controller via the keyboard and mouse.

[0082] Among them, each first positioning ball has a unique corresponding ball number.

[0083] It should be noted that the human body surface reference plane positioning module includes a plurality of first positioning balls, for example, three first positioning balls, namely, positioning ball 1, positioning ball 2, and positioning ball 3. In practical applications, positioning ball 1, positioning ball 2, and positioning ball 3 are placed on the human body surface of the surgical part near the entrance of the surgical setting, and the distribution of positioning ball 1, positioning ball 2, and positioning ball 3 is a non-isosceles triangle.

[0084] Step S202, obtaining a ball position image including position information of all first positioned balls collected by a binocular camera;

[0085] Step S203, adjusting the positions of the first positioning balls in the ball position image according to the actual position information of the first positioning balls, so that the positions of the first positioning balls in the ball position image are consistent with the actual positions;

[0086] Step S204: determine the adjusted ball position image as the image fusion reference plane.

[0087] Step S104: according to the relative positions of the first positioning balls on the image fusion reference plane, the medical images to be fused are fused on the image fusion reference plane to obtain a multimodal fused image.

[0088] It should be noted that in actual applications, only the C-arm image has the positioning ball, and other medical images to be fused do not have the image of the positioning ball, but only the image information of each device. Although other medical images to be fused other than the C-arm image do not have the image of the positioning ball, each medical image to be fused has a reference plane derived based on positioning ball 1, positioning ball 2 and positioning ball 3.

[0089] Fusion process: After the image fusion reference plane is determined, each medical image to be fused is fused based on this image fusion reference plane.

[0090] The following example illustrates the fusion process of each medical image to be fused on the image fusion reference plane:

[0091] Assume that the medical image to be fused is a C-arm image. During the operation, the operator places positioning balls 1, 2 and 3 on the patient's body surface. The C-arm obtains a C-arm image through the radiographic imaging and image reconstruction technology in its own technology. The C-arm image contains the image information of positioning balls 1, 2 and 3.

[0092] The C-arm image with the positioning ball image information is input to the video controller.

[0093] The video controller will analyze and compare the image of the positioning ball, find out the positions of positioning ball 1, positioning ball 2 and positioning ball 3, and measure the sizes of positioning ball 1, positioning ball 2 and positioning ball 3 by grayscale value. Match the known positioning ball feature information input in advance with the distance between each positioning ball in the image. Adjust the image position and display arm force in real time to make the positioning ball position the same as the known positioning ball feature information input in advance.

[0094] When the two are the same, the reference information of the C-arm image is obtained. So far, it is determined that the image source of the C-arm and the image source reference plane of the binocular camera are successfully matched.

[0095] Assuming that the medical image to be fused is a B-ultrasound image, in this embodiment, a positioning ball (ball numbered 4) is bound to the B-ultrasound handle, the distance between the positioning ball 4 and the point where the probe contacts the human body surface is manually measured, and the distance value is input into the video controller.

[0096] The binocular camera collects the distance position images of the positioning ball 4 and the positioning balls 1, 2, 3 in real time, and sends them to the video controller. The video controller calibrates the relative relationship between the B-ultrasound image and other images according to the distance position images.

[0097] Assume that the medical image to be fused is an endoscope image. Similar to a B-ultrasound image, a positioning ball (ball numbered 5) is bound to the handle of the endoscope, and the distance between ball 5 and the end of the endoscope entering the human body is manually measured. During the operation, the video controller calibrates the depth of the endoscope entering the human body based on the position of positioning ball 5 and positioning balls 1, 2, and 3.

[0098] It should be noted that when the medical image to be fused changes during the operation, the multimodal fusion image obtained by fusion also changes accordingly.

[0099] It should be noted that each medical image to be fused can be seen in the multimodal fusion image. For example, if the multimodal fusion image is obtained by fusing B-ultrasound and C-arm images, both B-ultrasound and C-arm images can be seen in the multimodal fusion image position, thereby making the image information more specific and allowing surgery to be carried out. In actual applications, each medical image to be fused before fusion can be displayed in other split screens of the display, so that the surgeon can make a more intuitive comparison based on the image before fusion and the image after fusion.

[0100] In summary, the present invention discloses a multimodal image fusion method. Before surgery, each first positioning ball in the human body surface reference plane positioning module is placed on the human body surface of the surgical part. At a position near the entrance set for surgery, the binocular camera collects a ball position image containing the position information of all first positioning balls. The video controller obtains the image fusion reference plane by processing the ball position image. The video controller obtains the corresponding medical images to be fused according to the acquired image selection instruction. According to the relative positions of each first positioning ball on the image fusion reference plane, each medical image to be fused is fused on the image fusion reference plane to obtain a multimodal fusion image. The present invention achieves the association of each single-modal image by fusing each medical image to be fused of a single-modal image, so that medical staff can diagnose diseases according to the multimodal fusion image, greatly improving the efficiency of defining the scope of surgery and shortening the time spent in the diagnosis process.

[0101] To further optimize the above embodiment, the following steps may be further included after step S104:

[0102] The multimodal fusion image is output to a display for display.

[0103] Specifically, the multimodal fusion image may be output to a split screen of a display for display.

[0104] In actual applications, in order to facilitate doctors to locate the surgical site, a 3D human anatomical image can also be displayed on the display. The 3D human anatomical image is pre-stored in the background database, and the 3D human anatomical image and the multimodal fusion image are displayed on different split screens of the display.

[0105] It should be noted that the various medical images to be fused before fusion may also be displayed on the display.

[0106] For example, a display is often set to display six split screens in multi-mode:

[0107] The upper left corner shows the multimodal fusion image;

[0108] The upper middle is fusion source 1 (e.g., C-arm image);

[0109] The lower left is fusion source 2 (e.g., B-ultrasound image);

[0110] The upper right is a 3D anatomical image that changes in real time;

[0111] The middle below is the PACS medical record data;

[0112] The lower right side shows the ECG monitor data.

[0113] To further optimize the above embodiment, the multimodal image fusion method may further include, before executing step S101:

[0114] Acquire a mode selection instruction for a surgical mode, and determine each medical imaging device corresponding to the surgical mode according to the mode selection instruction;

[0115] The medical images collected by each of the medical imaging devices are acquired and stored.

[0116] In this embodiment, the doctor can select a surgical mode on the touch screen before the operation, and the surgical mode includes but is not limited to general surgery, trauma orthopedics, minimally invasive intervention, neurosurgery, etc. Since different surgical modes may require different medical images, different surgical modes may correspond to medical imaging devices.

[0117] Optionally, medical images corresponding to different surgical modes may be displayed at different positions on the display. For example, medical images corresponding to general surgery are displayed in the lower right corner of the display.

[0118] After determining the various medical imaging devices corresponding to the surgical mode selected by the doctor, the video controller will acquire the medical images collected by the various medical imaging devices in real time and store them.

[0119] In order to facilitate medical staff to check the position of the multimodal fusion image in the 3D human anatomy map when performing surgery on patients, the 3D human anatomy map can be displayed on a split screen of the display, or the 3D human anatomy map can be fused with each medical image to be fused.

[0120] Therefore, to further optimize the above embodiment, step S104 may specifically include:

[0121] Get 3D human anatomy images;

[0122] According to the relative positions of each first positioning sphere on the image fusion reference plane, the 3D human anatomical image and each medical image to be fused are fused on the image fusion reference plane to obtain the multimodal fusion image, wherein the 3D human anatomical image is the bottom layer image in the multimodal fusion image.

[0123] To further optimize the above embodiment, the multimodal image fusion method may further include:

[0124] The 3D human anatomy image, the multimodal fusion image and each of the medical images to be fused are output to a split screen of a display for display.

[0125] In summary, the present invention discloses a multimodal image fusion method. Before surgery, each first positioning ball in the human body surface reference plane positioning module is placed on the human body surface of the surgical part. At a position near the entrance set by the surgery, the binocular camera collects the ball position image containing the position information of all the first positioning balls. The video controller obtains the image fusion reference plane by processing the ball position image. The video controller obtains the corresponding medical images to be fused according to the acquired image selection instruction, and obtains the 3D human anatomy map. According to the relative position of each first positioning ball on the image fusion reference plane, the 3D human anatomy map and each medical image to be fused are fused on the image fusion reference plane to obtain a multimodal fusion image, and the 3D human anatomy map, the multimodal fusion image and each medical image to be fused are output to a split screen of the display for display. The present invention realizes the association of each single-modal image by fusing each medical image to be fused of the single-modal image and the 3D human anatomy map, so that medical staff can diagnose diseases according to the multimodal fusion image, greatly improving the efficiency of defining the scope of surgery and shortening the time spent in the diagnosis process.

[0126] In addition, by comparing with the 3D human anatomy diagram, the fusion of the selected medical images to be fused can facilitate medical staff to view the relative information of the patient's internal organs in real time, thereby greatly reducing the burden on doctors. For puncture surgeries that require image guidance, no matter whether the doctor is familiar with the anatomical structure, the surgery can be performed clearly with the technical support provided by the present invention. Moreover, the present invention also effectively reduces the long operation time caused by repeated positioning by the doctor, which leads to excessive bleeding in the patient.

[0127] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0128] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0129] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal image fusion system, characterized in that: include: Video controller, medical imaging equipment, binocular camera, human surface reference plane positioning module, handheld device positioning module, display, touch screen and keyboard and mouse; The human body surface reference plane positioning module includes a plurality of positioning balls, each of which is recorded as a first positioning ball, each of which is used to be placed at a preset position, and each of which has a unique corresponding ball code, and the preset position is: on the human body surface of the surgical part, near the entrance set for the surgery; The handheld device positioning module includes at least one positioning ball, which is referred to as a second positioning ball. The second positioning ball is used to be placed on a handle for ultrasonic image acquisition, and each of the second positioning balls has a unique corresponding ball code. The binocular camera is used to collect a ball position image containing position information of the positioning ball; The touch screen is used to input a mode selection instruction for a surgical mode; The keyboard and mouse are used to input image selection instructions for the medical images to be fused; The display is used to display a multimodal fusion image; The video controller is respectively connected to the medical imaging device, the binocular camera, the display, the touch screen and the keyboard and mouse; The video controller is used to obtain the image selection instruction, select each medical image to be fused corresponding to the image selection instruction from each stored medical image, obtain the image fusion reference plane obtained by preprocessing, and fuse each medical image to be fused on the image fusion reference plane according to the relative positions of each of the first positioning balls on the image fusion reference plane to obtain the multimodal fusion image; The stored medical images are: medical images acquired by medical imaging devices corresponding to the pre-selected surgical mode; The image fusion reference plane is obtained by processing the ball position image containing all the first positioning balls at the preset positions collected by the binocular camera.

2. The multimodal image fusion system according to claim 1, characterized in that: The medical imaging equipment includes: C-arm, ultrasound equipment, endoscope equipment, microscope equipment and image archiving and communication system PACS.

3. A multimodal image fusion method, characterized in that: Applied to the video controller of claim 1, the fusion method comprises: Obtaining an image selection instruction for the medical image to be fused; Selecting each medical image to be fused corresponding to the image selection instruction from each stored medical image, wherein each stored medical image is: a medical image acquired by each medical imaging device corresponding to the pre-selected surgical mode; Acquire the image fusion reference plane obtained by preprocessing, wherein the image fusion reference plane is obtained by processing the ball position image containing all the first positioning balls at the preset position collected by the binocular camera, and the preset position is: the position on the human body surface of the surgical part, near the entrance set for the surgical operation; According to the relative positions of the first positioning balls on the image fusion reference plane, the medical images to be fused are fused on the image fusion reference plane to obtain a multimodal fusion image.

4. The multimodal image fusion method according to claim 3, characterized in that: The process of obtaining the image fusion reference plane is as follows: Acquire the actual position information of each first positioning ball in the human body surface reference plane positioning module placed at a preset position, wherein the preset position is: a position on the human body surface of the surgical part, near the entrance set for the surgical operation, and the actual position information includes: the ball number and volume size of each first positioning ball, and the shortest distance between each first positioning ball; Obtaining a ball position image collected by a binocular camera containing position information of all first positioned balls; According to the actual position information of each of the first positioning balls, the position of each of the first positioning balls in the ball position image is adjusted so that the position of each of the first positioning balls in the ball position image is consistent with the actual position; The adjusted ball position image is determined as the image fusion reference plane.

5. The multimodal image fusion method according to claim 3, characterized in that: Also includes: The multimodal fusion image is output to a display for display.

6. The multimodal image fusion method according to claim 3, characterized in that: The step of fusing the medical images to be fused on the image fusion reference plane according to the relative positions of the first positioning balls on the image fusion reference plane to obtain the multimodal fusion image specifically includes: Get 3D human anatomy images; According to the relative positions of each first positioning sphere on the image fusion reference plane, the 3D human anatomical image and each medical image to be fused are fused on the image fusion reference plane to obtain the multimodal fusion image, wherein the 3D human anatomical image is the bottom layer image in the multimodal fusion image.

7. The multimodal image fusion method according to claim 6, characterized in that: Also includes: The 3D human anatomy image, the multimodal fusion image and each of the medical images to be fused are output to a split screen of a display for display.

8. The multimodal image fusion method according to claim 3, characterized in that: Before obtaining the image selection instruction for the medical image to be fused, the method further includes: Acquire a mode selection instruction for a surgical mode, and determine each medical imaging device corresponding to the surgical mode according to the mode selection instruction; The medical images collected by each of the medical imaging devices are acquired and stored.

Images