Medical image fusion system, method, device

Abstract

The embodiment of the application provides a medical image fusion system, method and device, the medical image fusion system comprises an endoscope, an ultrasonic probe, a processor and a display, and the processor is used for: acquiring a medical image scanning reconstruction model of a target tissue; acquiring an endoscope image collected when the endoscope probes the target tissue; acquiring an ultrasonic image collected when the ultrasonic probe probes the target tissue; acquiring clinical data, determining at least two data in the medical image scanning reconstruction model, the endoscope image and the ultrasonic image as target data according to the clinical data, or displaying a prompt window according to the clinical data and determining at least two data in the medical image scanning reconstruction model, the endoscope image and the ultrasonic image as target data according to user input information; at least fusing the target data to obtain a target fusion image; and displaying the target fusion image through the display. The target fusion image can more accurately reflect the information currently required by the user, so that the operation can be more accurately performed.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a medical image fusion system, method, and device. Background Technology

[0002] Minimally invasive surgery utilizes endoscopy to obtain a high-resolution surgical view of the target area, guiding surgeons in their procedures. It boasts advantages such as minimal invasiveness and rapid recovery, making its development in the field of surgery remarkable. However, due to the limitations of endoscopic perspective and the difficulty in obtaining comprehensive information about organ tissues and predicting the course of internal vessels by imaging the organ's surface, single-modal endoscopic imaging cannot provide complete information to aid surgeons in their judgment. Therefore, surgeons often need to utilize multiple modal imaging information for auxiliary diagnosis and treatment.

[0003] In related technologies, imaging information of multiple modalities is displayed in a split-screen manner, which requires doctors to fill in the gaps in their understanding and fails to establish effective multimodal information fusion. In existing technologies, endoscopic images, ultrasound images, and preoperative scan images are also fused, but they cannot be freely configured and the image display mode cannot be configured according to the actual situation. Summary of the Invention

[0004] This application provides a medical image fusion system, method, and device that can present users with more comprehensive information in real time and more intuitively, so that users can perform minimally invasive surgery more accurately, effectively, and safely.

[0005] In a first aspect, embodiments of this application provide a medical image fusion system, the medical image fusion system including an endoscope, an ultrasound probe, a processor, and a display, wherein the processor is used for:

[0006] Obtain a medical image scan reconstruction model of the target tissue;

[0007] Acquire endoscopic images captured when the endoscope explores the target tissue;

[0008] Acquire ultrasound images when the ultrasound probe examines the target tissue;

[0009] Acquire clinical data, which includes at least one of the following: clinical image data, surgical instrument status, monitoring data, the position information of the endoscope, and the position information of the ultrasound probe;

[0010] Based on the clinical data, at least two of the data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image are determined as target data; or based on the clinical data display prompt window and user input information, at least two of the data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image are determined as target data.

[0011] The target data are fused at least to obtain a target fused image, wherein at least a portion of the image region of the target fused image includes multiple superimposed target data.

[0012] The target fused image is displayed on the monitor.

[0013] Secondly, embodiments of this application provide a medical image fusion method, including:

[0014] Obtain a medical image scan reconstruction model of the target tissue;

[0015] Acquire endoscopic images during endoscopic examination of the target tissue;

[0016] Acquire ultrasound images when an ultrasound probe examines the target tissue;

[0017] Acquire clinical data, which includes at least one of the following: clinical image data, surgical instrument status, monitoring data, the position information of the endoscope, and the position information of the ultrasound probe;

[0018] Based on the clinical data, at least two of the data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image are determined as target data; or based on the clinical data display prompt window and user input information, at least two of the data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image are determined as target data.

[0019] The target data are fused at least to obtain a target fused image, wherein at least a portion of the image region of the target fused image includes multiple superimposed target data.

[0020] The target fused image is displayed on a monitor.

[0021] Thirdly, embodiments of this application provide a medical image fusion device, the medical image fusion device comprising:

[0022] Display interface, used to connect to a monitor;

[0023] A processor for performing the aforementioned medical image fusion method.

[0024] This application provides a medical image fusion system, method, and device. The medical image fusion system includes an endoscope, an ultrasound probe, a processor, and a display. The processor is used to: acquire a medical image scan reconstruction model of a target tissue; acquire endoscopic images captured when the endoscope probes the target tissue; acquire ultrasound images captured when the ultrasound probe probes the target tissue; acquire clinical data, the clinical data including at least one of the following: clinical image data, surgical instrument status, monitoring data, endoscope pose information, and ultrasound probe pose information; determine at least two of the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data, or determine at least two of the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data display prompt window and user input information; fuse at least the target data to obtain a target fused image, at least a portion of the image region of the target fused image including multiple superimposed target data; and display the target fused image through the display. By determining the target data for fusion based on clinical data or by displaying a prompt window based on clinical data and allowing the user to determine the target data for fusion, the resulting fused image can more accurately reflect the information the user currently needs, enabling the user to perform minimally invasive surgery more precisely, effectively, and safely.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the disclosure of the embodiments of this application. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the structure of a medical image fusion system provided in an embodiment of this application;

[0028] Figure 2 This is a schematic diagram of the workflow of the medical image fusion system according to one embodiment of this application;

[0029] Figure 3 This is a schematic flowchart of a medical image fusion method provided in an embodiment of this application;

[0030] Figure 4 This is a schematic diagram of a medical image fusion device provided in one embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0033] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0034] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a medical image fusion system provided in an embodiment of this application.

[0035] like Figure 1 As shown, the medical image fusion system includes an endoscope, an ultrasound probe, a processor, and a display. The processor is connected to the endoscope, ultrasound probe, and display, and this connection may include a wired connection and / or a wireless communication connection.

[0036] The endoscope includes an insertion section and an operating section. The insertion section is used to insert into the patient's site of observation. The insertion section and the operating section can be an integral or detachable structure. The endoscope also includes at least one image sensor (not shown). Exemplarily, the image sensor can be located at the front end of the insertion section or in the operating section. The processor is used to acquire image signals from the endoscope and process them to obtain endoscopic images.

[0037] The medical image fusion system also includes a light source, which can be connected to the endoscope via a beam guide. The light source provides illumination to the area to be observed. The light source can include visible light sources and special light sources. Exemplarily, the visible light source is an LED light source, which can provide multiple monochromatic lights of different wavelength ranges, combinations of multiple monochromatic lights, or a broadband white light source. The special light source can be a laser source corresponding to a fluorescent reagent, such as near-infrared light. In some embodiments, before imaging with the endoscope system, a fluorescent reagent is injected into the area to be observed; the fluorescent reagent absorbs the laser light generated by the laser source and fluoresces.

[0038] The insertion section of the endoscope includes a tube, an image sensor, and an illumination path. The tip of the tube is inserted into the human body and reaches the area to be examined. The illumination path is aligned with a beam guide to direct light generated by the light source onto the area to be examined. The image signal acquired by the image sensor can undergo preliminary signal processing before being sent to the processor for further image processing. This preliminary signal processing includes amplification and filtering. For example, in an electronic endoscope, the image sensor includes a first image sensor and a second image sensor. The optical axes of the first and second sensors can be arranged parallel or at an angle. The left and right image signals acquired by the first and second image sensors correspond to the stereoscopic images observed by the left and right eyes, thus simulating binocular stereoscopic vision. The other end of the operating section is connected to the processor via a cable, transmitting the image signal to the processor for processing. In some embodiments, image data can also be transmitted to the processor wirelessly.

[0039] An ultrasound probe utilizes the piezoelectric effect to convert excitation electrical pulse signals into ultrasonic signals that enter the patient's body. The ultrasound echo signals reflected from the tissue are then converted into electrical signals. A processor can process these electrical signals into an ultrasound image, thus enabling tissue detection. Optionally, the ultrasound probe can be an external ultrasound probe or a laparoscopic ultrasound probe, which can be inserted into the body cavity of a human or animal. Examples include, but are not limited to, conventional transesophageal probes or laparoscopic ultrasound probes.

[0040] It should be noted that the medical image fusion system includes one or more processors. For example, the same processor can acquire image signals from the endoscope and perform signal processing on the image signals acquired from the endoscope to obtain an endoscopic image, and process the electrical signals obtained from the ultrasound echo signals into an ultrasound image. It can also control the display to show the endoscopic image and / or the ultrasound image. Alternatively, a first processor (such as the processor of the endoscope's camera host) can perform signal processing on the image signals acquired from the endoscope to obtain an endoscopic image, and a second processor (such as the processor of the ultrasound device) can process the electrical signals obtained from the ultrasound echo signals into an ultrasound image. The first processor, the second processor, or other processors can control the display to show the endoscopic image and / or the ultrasound image.

[0041] Specifically, the processor of the medical image fusion system is used to: acquire a medical image scan reconstruction model of the target tissue; acquire endoscopic images collected when the endoscope probes the target tissue; acquire ultrasound images collected when the ultrasound probe probes the target tissue; acquire clinical data, the clinical data including at least one of the following: clinical image data, surgical instrument status, monitoring data, endoscope pose information, and ultrasound probe pose information; determine at least two of the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data, or determine at least two of the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data display prompt window and user input information; fuse at least the target data to obtain a target fused image, at least a portion of the image region of the target fused image including multiple superimposed target data; and display the target fused image through the display.

[0042] For example, the target tissues include, but are not limited to, at least one of the following: liver, kidney, heart, spleen, lung, stomach, esophagus, intestine, and uterine cavity. For ease of explanation, the embodiments of this application mainly use the liver as an example.

[0043] Medical image scanning reconstruction models are obtained by processing medical images such as CT (computed tomography) images and / or MR (magnetic resonance imaging) images.

[0044] Original CT and / or MR images are typically two-dimensional slice images, allowing users to visualize and obtain the relative positions of different tissue structures (such as blood vessels). This application's embodiments can segment and reconstruct the structures (such as anatomical structures and vessels) of the target tissue in the two-dimensional slice images using three-dimensional reconstruction technology to obtain a three-dimensional image of the surgical area. The medical image scan reconstruction model includes global information about the patient's surgical area. For example, users can use the medical image scan reconstruction model as a basis for preoperative planning and surgery.

[0045] Optionally, the medical image scan reconstruction model can be obtained and stored before surgery, and the processor can acquire the pre-stored medical image scan reconstruction model during surgery. Alternatively, the processor can acquire and process medical images such as CT images and / or MR images during surgery to obtain the medical image scan reconstruction model. The CT images and / or MR images acquired by the processor can be taken before surgery or during surgery.

[0046] Medical image scanning reconstruction models can reflect the global information of the target organ during surgery. However, these models are reconstructed based on preoperative scan information or low-frequency scan information during surgery, resulting in low real-time performance and an inability to reflect changes in organ tissue in real time during the procedure. Endoscopic and ultrasound images, on the other hand, offer higher real-time performance. Endoscopic images can reflect the surface information of the local area currently being explored by the endoscope, while ultrasound images can reflect the course of blood vessels in the subcutaneous tissue of the explored area. However, the field of view of endoscopic and ultrasound images is limited, and they cannot reflect the overall information of the organ tissue. For example, a user can only determine that there are blood vessels in the current direction of exploration using endoscopic and ultrasound images, but cannot determine the specific location of those blood vessels on the organ. This application's embodiment fuses at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image as target data to obtain a target fusion image. This target fusion image can present users with more comprehensive information in real time and more intuitively, enabling users to perform minimally invasive surgery more accurately, effectively, and safely.

[0047] Specifically, in this embodiment of the application, the target data is determined based on clinical data or the target data is determined by the user through a prompt window based on clinical data. This makes the fused data more in line with clinical needs, and the resulting target fused image can more accurately reflect the information that the user currently needs, so that the user can perform minimally invasive surgery more accurately, effectively, and safely.

[0048] For example, a target fusion image obtained by fusing medical imaging scan reconstruction models and endoscopic images as target data can provide global anatomical information of the target tissue and real-time surface information of the local area currently being explored by the endoscope. This can help users understand the specific lesion and its location, guiding them to more accurately locate the lesion and determine the surgical path and location.

[0049] For example, a target fusion image obtained by fusing medical imaging scan reconstruction models and ultrasound images as target data can provide global anatomical information of the target tissue, and can also provide real-time information on the location, boundaries, and status of adjacent structures such as blood vessels (e.g., blood vessels) and masses within the target tissue. This allows users to easily understand the specific location and orientation of blood vessels, masses, and other structures within the target tissue in real time.

[0050] For example, a target fusion image obtained by fusing endoscopic images and ultrasound images as target data can provide real-time information on the surface and depth of the target tissue, which helps users identify lesions on the surface and inside the target tissue.

[0051] For example, a target fusion image obtained by fusing medical image scanning reconstruction model, endoscopic image and ultrasound image as target data can comprehensively reflect the global anatomical information of the target tissue, the surface information and deep information of the local area of ​​the target tissue, and the specific location of the local area in the target tissue, so as to provide users with more comprehensive and accurate information.

[0052] In other embodiments, when two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image are determined to be target data, the step of fusing at least the target data to obtain a target fused image includes: fusing the target data while not fusing the non-target data to obtain the target fused image; the non-target data is one type of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image other than the target data. For example, when the endoscopic image and the ultrasound image are determined to be target data, the endoscopic image and the ultrasound image are fused to obtain the target fused image.

[0053] In other embodiments, fusing the target data to obtain a target fused image includes: fusing the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image to obtain the target fused image; and when two types of data in the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image are determined to be target data, the transparency of the non-target data in the target fused image is greater than or equal to a transparency threshold; the non-target data is one type of data in the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image other than the target data.

[0054] For example, the three types of data can be fused by default to obtain an initial target fused image. By adjusting the transparency of the non-target data in the initial target fused image, the non-target data in the displayed target fused image can be made completely or partially transparent to highlight the target data. This can avoid or reduce the interference of non-target data on the target data, so that users can clearly view the information of the target data.

[0055] For example, when adjusting target data based on clinical data or operational data, it is not necessary to adjust the fusion process of the three types of data. The adjusted target data can be highlighted by adjusting the transparency of the corresponding data, so as to avoid or reduce the interference of the adjusted non-target data on the target data.

[0056] For example, the processor is further configured to: when the non-target data is determined to be target data, adjust the transparency of the non-target data in the target fused image to be less than the transparency threshold.

[0057] For example, after determining the endoscopic image and the ultrasound image as target data, when adjusting the target data based on clinical data or adjusting the target data including the medical image scanning reconstruction model based on operation, the transparency of the medical image scanning reconstruction model in the target fusion image is adjusted to be equal to the transparency of the endoscopic image and the ultrasound image, and both are less than the transparency threshold, so as to comprehensively reflect the global anatomical information of the target tissue, the surface information and deep information of the local area of ​​the target tissue, and the specific location of the local area in the target tissue, which can provide users with more comprehensive and accurate information.

[0058] For example, after determining the medical image scan reconstruction model, endoscopic image, and ultrasound image as target data, if the target data is adjusted according to clinical data or adjusted according to operation to exclude ultrasound image, the transparency of the ultrasound image in the target fusion image is adjusted to be greater than or equal to the transparency threshold. This is to avoid or reduce the interference of ultrasound image on the medical image scan reconstruction model and endoscopic image in the target fusion image, so that the user can clearly view the information of the medical image scan reconstruction model and endoscopic image.

[0059] Optionally, the order in which the various target data are superimposed in the target fused image is not limited. For example, when the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image are determined as target data, at least a portion of the image region of the medical image scanning reconstruction model can be superimposed on the endoscopic image, and at least a portion of the image region of the ultrasound image can be superimposed on the medical image scanning reconstruction model.

[0060] In some embodiments, when the data to be fused includes the ultrasound image, the fusion of at least the target data to obtain a target fused image includes: superimposing at least a portion of the ultrasound image region onto the medical image scan reconstruction model and / or the endoscopic image.

[0061] For example, overlaying blood flow signal regions and / or mass signal regions from the ultrasound image onto the medical image scan reconstruction model facilitates the user's determination of the location of blood flow and / or mass within the target tissue. Overlaying blood flow signal regions and / or mass signal regions from the ultrasound image onto the endoscopic image facilitates the user's understanding of the distribution of blood flow and / or mass below the surface of the target tissue. Furthermore, it reduces the occlusion of the ultrasound image on the medical image scan reconstruction model and / or the endoscopic image in the target fused image.

[0062] For example, the ultrasound image includes a two-dimensional ultrasound image; for example, at least a portion of the image region of the medical imaging scan reconstruction model and / or the endoscopic image may be superimposed on the two-dimensional ultrasound image.

[0063] For example, the medical image scanning reconstruction model is a three-dimensional reconstruction model, on which at least a portion of the image region of the endoscopy image and / or at least a portion of the image region of the ultrasound image can be superimposed.

[0064] Optionally, when superimposing at least a portion of the image region of the two-dimensional ultrasound image onto the three-dimensional reconstruction model to obtain the target fused image, the processor can further be configured to: superimpose an ultrasound probe identifier onto the target fused image, wherein the position of the ultrasound probe identifier and the exploration direction indicated by the ultrasound probe identifier are determined according to the pose of the ultrasound probe corresponding to the two-dimensional ultrasound image. Based on the ultrasound probe identifier, the user can clearly know the position and orientation of the area currently being explored by the ultrasound probe on the target tissue, thereby allowing the user to know the distribution of structures such as blood vessels and masses indicated by the ultrasound image on the target tissue. Of course, the ultrasound probe identifier can also indicate the position of the ultrasound probe corresponding to the two-dimensional ultrasound image; that is, the ultrasound probe identifier does not need to reflect the exploration direction of the ultrasound probe. Optionally, superimposing at least a portion of the image region of the two-dimensional ultrasound image onto the three-dimensional reconstruction model can include cases where the target data includes a three-dimensional reconstruction model and a two-dimensional ultrasound image but does not include an endoscopic image, or cases where the target data includes a three-dimensional reconstruction model, a two-dimensional ultrasound image, and an endoscopic image; for example, the endoscopic image can be superimposed on the three-dimensional reconstruction model first, and then the two-dimensional ultrasound image can be superimposed on the endoscopic image and the three-dimensional reconstruction model.

[0065] For example, the ultrasound image includes a three-dimensional ultrasound image; for instance, at least a portion of the image region of the medical image scan reconstruction model and / or the endoscopic image can be superimposed on the three-dimensional ultrasound image. Optionally, the ultrasound probe is an area array probe, and the electrical signal processing corresponding to the ultrasound echo signal received by the area array probe can obtain a three-dimensional ultrasound image; or the ultrasound probe is a two-dimensional probe, and multiple two-dimensional ultrasound images can be acquired by a flat scan and the three-dimensional ultrasound image can be reconstructed from the multiple two-dimensional ultrasound images; or multiple two-dimensional ultrasound images and the pose of the ultrasound probe corresponding to each two-dimensional ultrasound image can be acquired, and the three-dimensional ultrasound image can be reconstructed from the multiple two-dimensional ultrasound images based on the pose of the ultrasound probe.

[0066] The three-dimensional ultrasound image is obtained in real time during the operation and can be referred to as an intraoperative three-dimensional ultrasound image.

[0067] Three-dimensional ultrasound images provide more comprehensive anatomical information than two-dimensional ultrasound images, and can also show the spatial relationships between different structures, facilitating the identification and analysis of structures within and between tissues. By fusing three-dimensional ultrasound images with the medical imaging scan reconstruction model and / or the endoscopic images with the three-dimensional ultrasound images, more comprehensive and accurate information can be provided to the user.

[0068] Optionally, the medical image scanning and reconstruction model is a three-dimensional reconstruction model. The step of superimposing at least a portion of the ultrasound image region onto the medical image scanning and reconstruction model includes: superimposing the three-dimensional ultrasound image onto the medical image scanning and reconstruction model according to the pose of the ultrasound probe, wherein the pose of the superimposed three-dimensional ultrasound image is determined according to the pose of the ultrasound probe. This ensures that the pose of the three-dimensional ultrasound image in the target fused image is the same as the current pose of the ultrasound probe, facilitating the user's understanding of the current probe location.

[0069] Optionally, the medical image scanning reconstruction model is a three-dimensional reconstruction model. Superimposing at least a portion of the ultrasound image region onto the medical image scanning reconstruction model includes: determining a three-dimensional ultrasound image of the target structure in the three-dimensional ultrasound image based on the target structure of the target tissue in the endoscopic image; and superimposing the three-dimensional ultrasound image of the target structure onto the medical image scanning reconstruction model. For example, when the surface or interior of the area currently being explored by the endoscope includes target structures such as blood vessels or masses, superimposing a three-dimensional ultrasound image of the target structure onto the medical image scanning reconstruction model allows the user to accurately understand the specific location and orientation of the target structures such as blood vessels or masses on or inside the target tissue area.

[0070] Optionally, the display parameters of the three-dimensional ultrasound image in the target fused image are different from the display parameters of the three-dimensional reconstruction model in the target fused image, so that users can distinguish between the three-dimensional ultrasound image and the three-dimensional reconstruction model in the target fused image.

[0071] For example, users can adjust the target data based on the desired effect of the fused image display. For instance, when observing the 3D reconstruction model is not required, the target data can be adjusted to exclude the 3D reconstruction model; similarly, when observing the 3D ultrasound image is not required, the target data can be adjusted to exclude the 3D ultrasound image.

[0072] For example, the transparency of the 3D ultrasound image in the target fused image is less than the transparency of the 3D reconstructed model in the target fused image. That is, the 3D ultrasound image is highlighted so that the user can observe the information indicated by the 3D ultrasound image.

[0073] In some implementations, determining at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on user input information includes: displaying the corresponding identifiers for each of the medical image scan reconstruction model, the endoscopic image, and the ultrasound image; and determining at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as the target data based on the user's selection of the identifiers. For example, the display parameters of the identifiers corresponding to the target data are different from the display parameters of the identifiers corresponding to non-target data, so that the user can identify which data is the target data. It should be noted that the identifiers can be displayed in the prompt window to allow the user to determine the target data, and the identifiers can also be displayed in other interfaces or windows. For example, the corresponding identifiers for each of the medical image scan reconstruction model, the endoscopic image, and the ultrasound image can be displayed on each interface displayed on the monitor, allowing the user to familiarize themselves with the data corresponding to each identifier and thus select the required data as the target data more quickly and accurately.

[0074] In some implementations, the clinical image data includes the ultrasound image and / or the endoscopic image; determining at least two of the data from the medical image reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: when the clinical image data includes the target structure of the target tissue, determining the medical image reconstruction model and the clinical image data as the target data. For example, when the ultrasound image includes target structures such as blood vessels or masses, the processor automatically determines the medical image reconstruction model and the ultrasound image as target data, or displays a prompt window and determines the medical image reconstruction model and the ultrasound image as target data based on user input information; and fuses the target data to obtain a target fused image. Optionally, the endoscopic image can also be determined as the target data. For example, when the ultrasound image includes the middle hepatic vein, the image region corresponding to the middle hepatic vein in the ultrasound image is superimposed on the medical image reconstruction model, and the user can identify the location and direction of the middle hepatic vein based on the target fused image.

[0075] For example, determining at least two types of data—the medical image reconstruction model, the endoscopic image, and the ultrasound image—as target data based on the clinical data includes: when the clinical image data does not include the target structure, the target data does not include the medical image reconstruction model; and / or, the processor is further configured to: when the clinical image data does not include the target structure, display the clinical image data and the medical image reconstruction model separately, for example, displaying them on different displays, or displaying them in different areas of the same display, or displaying them alternately. When the clinical image data does not include the target structure, the target fusion image does not superimpose the medical image reconstruction model, or the transparency of the medical image reconstruction model is greater than or equal to a transparency threshold, which can prevent the medical image reconstruction model from interfering with the clinical image data and hindering the user's identification of the target structure in the clinical image data.

[0076] In some implementations, the clinical image data includes the ultrasound image; determining at least two types of data from the medical image reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: when the ultrasound image includes the target structure of the target tissue, determining the ultrasound image, the medical image reconstruction model, and / or the endoscopic image as the target data. For example, when the ultrasound image includes target structures such as blood vessels or masses, the processor automatically determines the endoscopic image and the ultrasound image as target data, or displays a prompt window and determines the endoscopic image and the ultrasound image as target data based on user input; and fuses the target data to obtain a target fused image. Optionally, the medical image reconstruction model can also be determined as the target data. For example, when the ultrasound image includes the middle hepatic vein, the image region corresponding to the middle hepatic vein in the ultrasound image is superimposed on the endoscopic image, allowing the user to more comprehensively and accurately identify lesions on the surface and inside the target tissue based on the target fused image.

[0077] In some implementations, the clinical image data includes the endoscopic image; determining at least two of the following data as target data based on the clinical data—the medical image reconstruction model, the endoscopic image, and the ultrasound image—includes: when the endoscopic image includes the target structure of the target tissue, determining the endoscopic image, the medical image reconstruction model, and / or the ultrasound image as the target data. For example, when the surface or interior of the area currently being explored by the endoscope includes target structures such as blood vessels or masses, the processor automatically determines the endoscopic image and the medical image reconstruction model as target data, or displays a prompt window and determines the endoscopic image and the medical image reconstruction model as target data based on user input; and fuses the target data to obtain a target fused image. Optionally, the ultrasound image can also be determined as the target data. For example, when the surface of the area currently being explored by the endoscope includes a mass, by overlaying at least a portion of the image area of ​​the medical image reconstruction model onto the endoscopic image, the user can more comprehensively and accurately identify lesions in the target tissue based on the target fused image.

[0078] In some embodiments, determining at least two of the data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: determining the medical image scan reconstruction model and the endoscopic image and / or the ultrasound image as the target data when the distance between the surgical instrument and the target structure of the target tissue is greater than or equal to a first preset distance.

[0079] When the surgical instruments are far from the target structure, such as the middle hepatic vein, the target fusion image obtained by fusing the medical imaging scan reconstruction model with the endoscopic image and / or the ultrasound image can help the user determine the orientation and location of the target structure on the target tissue.

[0080] For example, the processor can acquire the pose of the surgical instrument and determine the distance between the surgical instrument and the target structure of the target tissue based on the pose of the surgical instrument. Optionally, the surgical instrument is equipped with a pose sensor and / or the operating mechanism of the surgical instrument is equipped with a pose sensor, which can determine the pose of the surgical instrument based on the data from the pose sensor.

[0081] For example, the processor can determine the distance between the surgical instruments and the target structure of the target tissue based on the endoscopic images.

[0082] In some embodiments, determining at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: when the distance between the surgical instrument and the target structure of the target tissue is less than the first preset distance, determining that the target data does not include the medical image scan reconstruction model.

[0083] When the target data does not include the medical image scan reconstruction model, the target fused image does not overlay the medical image scan reconstruction model, or the transparency of the medical image scan reconstruction model is greater than or equal to a transparency threshold. Optionally, the processor is configured to: display the medical image scan reconstruction model separately from the ultrasound image and / or the endoscopic image when the target data does not include the medical image scan reconstruction model.

[0084] When the surgical instruments are close to the target structure, such as the middle hepatic vein, the target fusion image does not superimpose the medical image scanning reconstruction model, or the transparency of the medical image scanning reconstruction model is greater than or equal to the transparency threshold. This can prevent the medical image scanning reconstruction model from obscuring the surgical instruments in the endoscopic image and / or the ultrasound image, thus preventing the user from not being able to see the position and status of the surgical instruments, thereby affecting the surgical operation.

[0085] In other embodiments, the step of determining at least two types of data—the medical image scan reconstruction model, the endoscopic image, and the ultrasound image—as target data based on the clinical data display prompt window and user input information includes: displaying a prompt window when the distance between the surgical instrument and the target structure of the target tissue is less than or equal to a first preset distance, and determining whether the medical image scan reconstruction model is the target data based on user input information. When the surgical instrument and the target structure, such as the middle hepatic vein, are close, the user can determine whether the target fusion image superimposes the medical image scan reconstruction model or whether the transparency of the medical image scan reconstruction model is greater than or equal to a transparency threshold based on the display effect of the target fusion image. For example, if the medical image scan reconstruction model does not obscure the endoscopic image and / or the ultrasound image and can assist the user in analysis, the user can determine the medical image scan reconstruction model as the target data; if the medical image scan reconstruction model obscures the endoscopic image and / or the ultrasound image, the user can determine that the target data does not include the medical image scan reconstruction model.

[0086] In some embodiments, determining at least two types of data—the medical image scan reconstruction model, the endoscopic image, and the ultrasound image—as target data based on the clinical data includes: when the monitoring data meets the emergency state judgment criteria, determining that the target data does not include the medical image scan reconstruction model. Optionally, the processor is configured to: when the target data does not include the medical image scan reconstruction model, display the medical image scan reconstruction model separately from the ultrasound image and / or the endoscopic image.

[0087] For example, the monitoring data includes, but is not limited to, at least one of the following: blood pressure, heart rate, respiratory rate, and blood oxygen saturation; for example, when blood pressure drops to a low level or drops rapidly, the target data is determined to exclude the medical image scan reconstruction model so that users can more clearly and quickly view real-time endoscopic images and / or ultrasound images, thereby speeding up processing efficiency to deal with emergencies.

[0088] In other embodiments, the step of determining at least two types of data—the medical image scan reconstruction model, the endoscopic image, and the ultrasound image—as target data based on the clinical data display prompt window and user input information includes: displaying a prompt window when the monitoring data meets the emergency status judgment conditions, and determining whether the medical image scan reconstruction model is the target data based on user input information. When the monitoring data meets the emergency status judgment conditions, the user can determine whether the target fused image is superimposed with the medical image scan reconstruction model or whether the transparency of the medical image scan reconstruction model is greater than or equal to a transparency threshold based on the display effect of the target fused image and the urgency of the emergency.

[0089] In some implementations, determining at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: determining at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image as the target data based on at least two of the pose relationships between the endoscope and the target tissue, the ultrasound probe and the target tissue, and the endoscope and the ultrasound probe. This makes the target fused image obtained by fusing the target data more suitable for the user's needs in the current exploration state.

[0090] For example, when the distance between the endoscope and the target tissue is less than or equal to a second preset distance, and the distance between the ultrasound probe and the target tissue is less than or equal to a third preset distance and / or the distance between the endoscope and the ultrasound probe is less than or equal to a fourth preset distance, the medical image scan reconstruction model, the endoscope image, and the ultrasound image are determined as the target data. When the distance between the endoscope and the target tissue is relatively close, fusing the medical image scan reconstruction model and the ultrasound image with the endoscope image allows the user to view the structure of the target tissue more precisely for accurate surgery. When the distance between the ultrasound probe and the target tissue is relatively close, fusing the medical image scan reconstruction model and the endoscope image with the ultrasound image allows the user to view the structure of the target tissue more precisely for accurate surgery. When the distance between the endoscope and the ultrasound probe is relatively close, fusing the endoscope image, the ultrasound image, and the medical image scan reconstruction model allows the user to view the structure of the target tissue more precisely for accurate surgery.

[0091] For example, when the distance between the ultrasound probe and the target tissue is greater than the third preset distance and / or the distance between the endoscope and the ultrasound probe is greater than the fourth preset distance, the medical image scan reconstruction model and the endoscopic image are determined to be the target data. When the distance between the ultrasound probe and the target tissue is far and / or the distance between the ultrasound probe and the endoscope is far, the target fused image is not superimposed with the ultrasound image, or the transparency of the ultrasound image is greater than or equal to a transparency threshold, which can prevent the ultrasound image from interfering with the user's viewing of the medical image scan reconstruction model and the endoscopic image.

[0092] In some implementations, when fusing the target data to obtain the target fused image, the poses of the remaining target data are adjusted according to the pose changes of at least one of the target data, and the pose changes of the remaining target data follow the pose changes of the at least one target data.

[0093] For example, the pose change of the target data includes: the pose change of the endoscope image corresponding to the pose change of the endoscope, the pose change of the ultrasound image corresponding to the pose change of the ultrasound probe, and the pose change of the target data under the user's pose adjustment operation.

[0094] For example, changes in the pose of an ultrasound probe will cause corresponding changes in the pose of the ultrasound image. By adjusting the pose of the medical image scanning reconstruction model and the endoscope image to follow the changes in the pose of the ultrasound image, the pose of the target tissue in the target fusion image can follow the changes in the pose of the ultrasound probe, making it easier for users to clearly view the detailed structure of the target tissue.

[0095] For example, by adjusting the medical image scanning reconstruction model and the ultrasound image to follow the pose changes of the endoscopic image, the pose of the target tissue in the target fusion image can change with the pose changes of the endoscope, making it easier for users to clearly view the detailed structure of the target tissue.

[0096] To adjust the pose of other target data based on pose changes of at least one of the target data, point matching can be performed on the target data in the medical image scan reconstruction model, ultrasound image, and endoscopic image. For example, when displaying the scan reconstruction model and the endoscopic image, the target tissue in the scan reconstruction model is matched with the target tissue in the endoscopic image, so that the positions of the corresponding target tissue images overlap, and the viewing angle of the target tissue in the scan reconstruction model observed by the doctor is consistent with that of the target tissue in the endoscopic image. As the endoscopic pose changes, the position and viewing angle of the target tissue in the endoscopic image change, and the system automatically adjusts the scan reconstruction data so that the position and viewing angle of the target tissue in the scan reconstruction model are adjusted according to the position and viewing angle of the target tissue in the endoscopic image, achieving multimodal registration and follow-up. When at least two of the scan reconstruction model, ultrasound image, and endoscopic image are displayed, registration can also be performed according to the method of the above embodiment to achieve multimodal follow-up. In this application, the pose of the reconstruction model, the pose of the ultrasound probe, or the pose of the endoscope can be selected as the main pose, and the other images are adjusted according to the main pose.

[0097] For example, the medical image fusion system also includes an interactive device, which includes, but is not limited to, at least one of the following: a mouse, keyboard, joystick, trackball, touchpad, or a touch-enabled display. The processor acquires user input through the interactive device and, based on the user's pose adjustment operation on at least one of the target data in the target fused image, such as the pose of a medical image scan reconstruction model, adjusts the pose of the remaining target data to follow the pose change of the at least one target data. This allows the pose of the target tissue in the target fused image to follow the pose change of the at least one target data, facilitating clear viewing of the detailed structure of the target tissue by the user.

[0098] In some embodiments, fusing the target data to obtain a target fused image includes: transforming the target data to the same preset space, such as transforming it to the same coordinate system; and fusing the target data in the preset space to obtain a target fused image. This ensures that the reference of each target data is the same, and the structures indicated by each target data in the target fused image can be aligned in the preset space, allowing the user to clearly view the detailed structure of the target organization.

[0099] For example, the same preset space is a preset space based on the endoscope. Compared with medical image scanning reconstruction models and ultrasound images, endoscopic images provide more intuitive information and are more in line with the user's visual experience. By fusing target data based on the endoscope, users can dynamically, intuitively, and accurately view the detailed structure of the target tissue.

[0100] For example, the step of converting the target data to the same preset space includes: obtaining the pose relationship between the medical image scanning reconstruction model and the endoscopic image; obtaining the pose relationship between the endoscopic image and the ultrasound image; and converting at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image to the same preset space based on the pose relationship between the medical image scanning reconstruction model and the endoscopic image, and the pose relationship between the endoscopic image and the ultrasound image.

[0101] For example, the medical image fusion system also includes a locator; see [link to relevant documentation]. Figure 2 The positioning device can acquire the pose of the endoscope relative to the positioning device and the pose of the ultrasound probe relative to the positioning device. Based on the poses of the endoscope and the ultrasound probe, as well as the pose relationships between the endoscope and the endoscope image, and between the ultrasound probe and the ultrasound image, the pose relationship between the endoscope image and the ultrasound image can be determined, thus unifying the endoscope image space and the ultrasound image space. For example, the positioning device can include an optical positioning device or an electromagnetic positioning device. For instance, the endoscope and the ultrasound probe are equipped with optical elements, and the optical positioning device determines the poses of the endoscope and the ultrasound probe by detecting the pose of the optical elements; or the endoscope and the ultrasound probe are equipped with electromagnetic elements, and the electromagnetic positioning device determines the poses of the endoscope and the ultrasound probe by detecting the pose of the electromagnetic elements.

[0102] For example, a medical image fusion system includes an operating mechanism that operates an endoscope and an ultrasound probe, or a user operates the endoscope and ultrasound probe through the operating mechanism; the operating mechanism can acquire the pose of the endoscope and the ultrasound probe; thereby, based on the pose of the endoscope and the ultrasound probe, as well as the pose relationship between the endoscope and the endoscope image, and the pose relationship between the ultrasound probe and the ultrasound image, the pose relationship between the endoscope and the ultrasound image can be determined, thus achieving the unification of the endoscope image space and the ultrasound image space.

[0103] For example, when the ultrasound probe is a laparoscopic ultrasound probe and the endoscopic image includes the laparoscopic ultrasound probe, the positional relationship between the laparoscopic ultrasound probe and the endoscope can be determined based on the positional relationship of the laparoscopic ultrasound probe in the endoscopic image. By combining the positional relationship between the endoscope and the endoscopic image and the positional relationship between the ultrasound probe and the ultrasound image, the positional relationship between the endoscopic image and the ultrasound image can be determined, thereby achieving the unification of the endoscopic image space and the ultrasound image space.

[0104] Optionally, obtaining the pose relationship between the endoscopic image and the ultrasound image includes: displaying the endoscopic image and the ultrasound image on the display; and obtaining the pose relationship between the endoscopic image and the ultrasound image based on the user's registration operation on the endoscopic image and / or the ultrasound image. For example, the processor obtains the user's registration operation on the ultrasound image and / or the endoscopic image through an interactive device, aligns the structures in the ultrasound image with the structures in the endoscopic image, realizes manual registration of the endoscopic image and the ultrasound image, and achieves the unification of the endoscopic image space and the ultrasound image space.

[0105] Optionally, obtaining the pose relationship between the endoscopic image and the ultrasound image includes: registering preset features of the endoscopic image and preset features of the ultrasound image to obtain the pose relationship between the endoscopic image and the ultrasound image. For example, the preset features of the ultrasound image and the endoscopic image include, but are not limited to, at least one of the following: point cloud, corner points, and contour lines. By extracting the preset features of the ultrasound image and the preset features of the endoscopic image, and registering these features, such as point cloud registration and feature point matching, the structures in the ultrasound image can be aligned with the structures in the endoscopic image, achieving automatic registration between the endoscopic image and the ultrasound image, and unifying the endoscopic image space and the ultrasound image space.

[0106] Optionally, obtaining the pose relationship between the medical image scan reconstruction model and the endoscopic image includes: displaying the medical image scan reconstruction model and the endoscopic image on the display; and obtaining the pose relationship between the medical image scan reconstruction model and the endoscopic image based on the user's registration operation on the medical image scan reconstruction model and / or the endoscopic image. For example, the processor obtains the user's registration operation on the medical image scan reconstruction model and / or the endoscopic image through an interactive device, aligns the structure in the medical image scan reconstruction model with the structure in the endoscopic image, and realizes manual registration between the endoscopic image and the medical image scan reconstruction model; furthermore, it can combine the pose relationship between the endoscopic image and the ultrasound image to convert at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image to the same preset space, thereby unifying the preoperative reconstruction space, the endoscopic image space, and the ultrasound image space.

[0107] Optionally, obtaining the pose relationship between the medical image scanning reconstruction model and the endoscopic image includes: extracting preset features of the medical image scanning reconstruction model and extracting preset features of the endoscopic image; registering the preset features of the medical image scanning reconstruction model and the preset features of the endoscopic image to obtain the pose relationship between the medical image scanning reconstruction model and the endoscopic image. For example, the preset features of the medical image scanning reconstruction model and the endoscopic image include, but are not limited to, at least one of the following: point cloud, corner points, and contour lines. By extracting preset features from the medical image scanning reconstruction model and the preset features from the endoscopic image, and registering these features, such as point cloud registration and feature point matching, the structure in the medical image scanning reconstruction model can be aligned with the structure in the endoscopic image, thus achieving automatic registration between the endoscopic image and the medical image scanning reconstruction model. Furthermore, by combining the pose relationship between the endoscopic image and the ultrasound image, at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image can be converted to the same preset space, thereby achieving the unification of the preoperative reconstruction space, the endoscopic image space, and the ultrasound image space.

[0108] Please see Figure 2 During the surgery, at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image can be fused within a preset space to obtain a target fused image. The ultrasound image can be a two-dimensional ultrasound image or a three-dimensional ultrasound image. Displaying this target fused image can provide users with more comprehensive information in real time and more intuitively, enabling users to perform minimally invasive surgery more accurately, effectively, and safely.

[0109] The medical image fusion system provided in this application includes an endoscope, an ultrasound probe, a processor, and a display. The processor is used to: acquire a medical image scan reconstruction model of a target tissue; acquire endoscopic images captured when the endoscope probes the target tissue; acquire ultrasound images captured when the ultrasound probe probes the target tissue; acquire clinical data, the clinical data including at least one of the following: clinical image data, surgical instrument status, monitoring data, endoscope pose information, and ultrasound probe pose information; determine at least two of the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data, or determine at least two of the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data display prompt window and user input information; fuse at least the target data to obtain a target fused image, at least a portion of the image region of the target fused image including multiple superimposed target data; and display the target fused image through the display. By determining the target data for fusion based on clinical data or by displaying a prompt window based on clinical data and allowing the user to determine the target data for fusion, the resulting fused image can more accurately reflect the information the user currently needs, enabling the user to perform minimally invasive surgery more precisely, effectively, and safely.

[0110] Please refer to the foregoing embodiments. Figure 3 , Figure 3 This is a schematic flowchart of a medical image fusion method provided in one embodiment of this application.

[0111] like Figure 3 As shown, the medical image fusion method includes the following steps S110 to S170.

[0112] Step S110: Obtain the medical image scan reconstruction model of the target tissue.

[0113] Step S120: Obtain endoscopic images captured during endoscopic exploration of the target tissue.

[0114] Step S130: Acquire ultrasound images captured when the ultrasound probe examines the target tissue.

[0115] Step S140: Obtain clinical data, which includes at least one of the following: clinical image data, surgical instrument status, monitoring data, endoscope position information, and ultrasound probe position information.

[0116] Step S150: Determine at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data, or determine at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data display prompt window and user input information.

[0117] Step S160: At least the target data is fused to obtain a target fused image, wherein at least a portion of the image region of the target fused image includes multiple superimposed target data.

[0118] Step S170: Display the target fused image on the monitor.

[0119] The specific principles and implementation methods of the medical image fusion method provided in this application are similar to those of the medical image fusion system in the aforementioned embodiments, and will not be repeated here.

[0120] Please refer to the foregoing embodiments. Figure 4 , Figure 4 This is a schematic diagram of a medical image fusion device 60 provided in one embodiment of this application.

[0121] like Figure 4 As shown, the medical image fusion device 60 includes a display interface 61 and a processor 62; the display interface 61 is used to connect to a display, the processor 62 is connected to the display interface 61, and the processor 62 is used to execute the aforementioned medical image fusion method.

[0122] Optionally, the display interface 61 can be connected to the monitor via a cable, or the display interface 61 can communicate with the monitor wirelessly.

[0123] Optionally, the medical image fusion device can be a device independent of the endoscope's camera host and ultrasound device, or the medical image fusion device can be the same device as the endoscope's camera host, or the medical image fusion device can be the same device as the ultrasound device, or the medical image fusion device can be the same device as both the endoscope's camera host and the ultrasound device.

[0124] The specific principles and implementation methods of the medical image fusion device provided in this application embodiment are similar to those of the medical image fusion system in the aforementioned embodiment, and will not be repeated here.

[0125] It should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application.

[0126] It should also be understood that the term “and / or” as used in this application and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0127] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A medical image fusion system, characterized in that, The medical image fusion system includes an endoscope, an ultrasound probe, a processor, and a display. The processor is used for: Obtain a medical image scan reconstruction model of the target tissue; Acquire endoscopic images captured when the endoscope explores the target tissue; Acquire ultrasound images when the ultrasound probe examines the target tissue; Acquire clinical data, which includes at least one of the following: clinical image data, surgical instrument status, monitoring data, the position information of the endoscope, and the position information of the ultrasound probe; Based on the clinical data, at least two of the data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image are determined as target data; or based on the clinical data display prompt window and user input information, at least two of the data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image are determined as target data. The target data are fused at least to obtain a target fused image, wherein at least a portion of the image region of the target fused image includes multiple superimposed target data. The target fused image is displayed on the monitor.

2. The medical image fusion system according to claim 1, characterized in that, The process of fusing at least the target data to obtain a target fused image includes: The medical image scan reconstruction model, the endoscopic image, and the ultrasound image are fused to obtain the target fused image; and and When two types of data in the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image are determined to be target data, the transparency of the non-target data in the target fusion image is greater than or equal to a transparency threshold; the non-target data is one type of data in the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image other than the target data.

3. The medical image fusion system according to claim 2, characterized in that, The processor is also used for: When the non-target data is identified as target data, the transparency of the non-target data identified as target data in the target fused image is adjusted to be less than the transparency threshold.

4. The medical image fusion system according to claim 1, characterized in that, When the data to be fused includes the ultrasound image, the process of fusing at least the target data to obtain a target fused image includes: At least a portion of the ultrasound image region is superimposed on the medical image scan reconstruction model and / or the endoscopic image.

5. The medical image fusion system according to claim 4, characterized in that, The medical image scanning reconstruction model is a three-dimensional reconstruction model, and the ultrasound image includes a two-dimensional ultrasound image; When superimposing at least a portion of the two-dimensional ultrasound image onto the three-dimensional reconstruction model to obtain the target fused image, the processor is further configured to: An ultrasound probe identifier is superimposed on the target fused image, and the position of the ultrasound probe identifier and the exploration direction indicated by the ultrasound probe identifier are determined according to the pose of the ultrasound probe corresponding to the two-dimensional ultrasound image.

6. The medical image fusion system according to claim 4, characterized in that, The medical image scanning reconstruction model is a three-dimensional reconstruction model, and the ultrasound image includes a three-dimensional ultrasound image. The superposition of at least a portion of the ultrasound image onto the medical image scanning reconstruction model includes: Based on the target structure of the target tissue in the endoscopic image, a three-dimensional ultrasound image of the target structure is determined in the three-dimensional ultrasound image; A three-dimensional ultrasound image of the target structure is superimposed on the medical imaging scan reconstruction model; Alternatively, the three-dimensional ultrasound image can be superimposed on the medical image scanning reconstruction model according to the pose of the ultrasound probe, and the pose of the superimposed three-dimensional ultrasound image is determined according to the pose of the ultrasound probe.

7. The medical image fusion system according to claim 6, characterized in that, The display parameters of the three-dimensional ultrasound image in the target fusion image are different from the display parameters of the three-dimensional reconstruction model in the target fusion image.

8. The medical image fusion system according to any one of claims 1-7, characterized in that, The step of determining at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image as target data based on user input information includes: Display the corresponding identifiers for the medical image scan reconstruction model, the endoscopic image, and the ultrasound image; Based on the user's selection of the identifier, at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image are determined as the target data.

9. The medical image fusion system according to any one of claims 1-7, characterized in that, The clinical image data includes the ultrasound images and / or the endoscopic images; The step of determining at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: When the clinical image data includes the target structure of the target tissue, the medical image scan reconstruction model and the clinical image data are identified as the target data.

10. The medical image fusion system according to claim 9, characterized in that, The step of determining at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: When the clinical image data does not include the target structure, the target data does not include the medical image scan reconstruction model; And / or, the processor is further configured to: display the clinical image data and the medical image scan reconstruction model separately when the clinical image data does not include the target structure.

11. The medical image fusion system according to any one of claims 1-7, characterized in that, The clinical image data includes the ultrasound images; The step of determining at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: when the ultrasound image includes the target structure of the target tissue, determining the ultrasound image, the medical image scanning reconstruction model, and / or the endoscopic image as the target data; and / or The clinical image data includes the endoscopic image; determining at least two of the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: when the endoscopic image includes the target structure of the target tissue, determining the endoscopic image, the medical image scanning reconstruction model, and / or the ultrasound image as the target data.

12. The medical image fusion system according to any one of claims 1-7, characterized in that, The step of determining at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: When the distance between the surgical instrument and the target structure of the target tissue is greater than or equal to a first preset distance, the medical image scanning reconstruction model, the endoscopic image, and / or the ultrasound image are determined as the target data.

13. The medical image fusion system according to any one of claims 1-7, characterized in that, The step of displaying a prompt window based on the clinical data and determining at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on user input information includes: displaying a prompt window when the distance between the surgical instrument and the target structure of the target tissue is less than or equal to a first preset distance, and determining whether the medical image scan reconstruction model is the target data based on user input information; Alternatively, determining at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: when the distance between the surgical instrument and the target structure of the target tissue is less than the first preset distance, determining that the target data does not include the medical image scan reconstruction model.

14. The medical image fusion system according to any one of claims 1-7, characterized in that, The step of displaying a prompt window based on the clinical data and determining at least two types of data among the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on user input information includes: displaying a prompt window when the monitoring data meets the emergency state judgment conditions, and determining whether the medical image scan reconstruction model is the target data based on user input information; Alternatively, determining at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: when the monitoring data meets the emergency state judgment conditions, determining that the target data does not include the medical image scanning reconstruction model.

15. The medical image fusion system according to any one of claims 1-7, characterized in that, The step of determining at least two types of data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image as target data based on the clinical data includes: Based on at least two of the following relationships: the pose relationship between the endoscope and the target tissue, the pose relationship between the ultrasound probe and the target tissue, and the pose relationship between the endoscope and the ultrasound probe, at least two of the following data are determined as the target data: the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image.

16. The medical image fusion system according to claim 15, characterized in that, The determination of at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image as the target data based on at least two of the following: the pose relationship between the endoscope and the target tissue, the pose relationship between the ultrasound probe and the target tissue, and the pose relationship between the endoscope and the ultrasound probe; including at least one of the following: When the distance between the endoscope and the target tissue is less than or equal to a second preset distance, and the distance between the ultrasound probe and the target tissue is less than or equal to a third preset distance and / or the distance between the endoscope and the ultrasound probe is less than or equal to a fourth preset distance, the medical image scanning reconstruction model, the endoscope image, and the ultrasound image are determined to be the target data. When the distance between the ultrasound probe and the target tissue is greater than the third preset distance and / or the distance between the endoscope and the ultrasound probe is greater than the fourth preset distance, the medical image scanning reconstruction model and the endoscope image are determined to be the target data.

17. The medical image fusion system according to any one of claims 1-7, characterized in that, The process of fusing at least the target data to obtain a target fused image includes: Convert the target data to the same preset space; In the preset space, the target data is fused to obtain a target fused image.

18. The medical image fusion system according to claim 17, characterized in that, The step of converting the target data to the same preset space includes: Obtain the pose relationship between the medical image scan reconstruction model and the endoscopic image; Obtain the pose relationship between the endoscopic image and the ultrasound image; Based on the pose relationship between the medical image scanning reconstruction model and the endoscopic image, and the pose relationship between the endoscopic image and the ultrasound image, at least two types of data from the medical image scanning reconstruction model, the endoscopic image, and the ultrasound image are converted to the same preset space.

19. The medical image fusion system according to claim 18, characterized in that, The step of obtaining the pose relationship between the medical image scan reconstruction model and the endoscopic image includes: displaying the medical image scan reconstruction model and the endoscopic image on the display; obtaining the pose relationship between the medical image scan reconstruction model and the endoscopic image based on the user's registration operation on the medical image scan reconstruction model and / or the endoscopic image; and / or The step of obtaining the pose relationship between the endoscopic image and the ultrasound image includes: displaying the endoscopic image and the ultrasound image on the display; and obtaining the pose relationship between the endoscopic image and the ultrasound image based on the user's registration operation on the endoscopic image and / or the ultrasound image.

20. The medical image fusion system according to claim 18, characterized in that, The step of obtaining the pose relationship between the medical image scanning reconstruction model and the endoscopic image includes: extracting preset features of the medical image scanning reconstruction model and extracting preset features of the endoscopic image; registering the preset features of the medical image scanning reconstruction model and the preset features of the endoscopic image to obtain the pose relationship between the medical image scanning reconstruction model and the endoscopic image; and / or The step of obtaining the pose relationship between the endoscopic image and the ultrasound image includes: registering preset features of the endoscopic image and preset features of the ultrasound image to obtain the pose relationship between the endoscopic image and the ultrasound image.

21. The medical image fusion system according to claim 17, characterized in that, The same preset space is a preset space based on the endoscope.

22. The medical image fusion system according to claim 17, characterized in that, When fusing the target data to obtain the target fused image, the pose of the remaining target data is adjusted according to the pose change of at least one of the target data, and the pose change of the remaining target data follows the pose change of the at least one target data.

23. The medical image fusion system according to claim 22, characterized in that, The pose changes of the target data include: the pose changes of the endoscope image corresponding to the pose changes of the endoscope, the pose changes of the ultrasound image corresponding to the pose changes of the ultrasound probe, and the pose changes of the target data under the user's pose adjustment operation.

24. A medical image fusion method, characterized in that, The method includes: Obtain a medical image scan reconstruction model of the target tissue; Acquire endoscopic images during endoscopic examination of the target tissue; Acquire ultrasound images when an ultrasound probe examines the target tissue; Acquire clinical data, which includes at least one of the following: clinical image data, surgical instrument status, monitoring data, the position information of the endoscope, and the position information of the ultrasound probe; Based on the clinical data, at least two of the data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image are determined as target data; or based on the clinical data display prompt window and user input information, at least two of the data from the medical image scan reconstruction model, the endoscopic image, and the ultrasound image are determined as target data. The target data are fused at least to obtain a target fused image, wherein at least a portion of the image region of the target fused image includes multiple superimposed target data. The target fused image is displayed on a monitor.

25. A medical image fusion device, characterized in that, The medical image fusion device includes: Display interface, used to connect to a monitor; A processor for performing the medical image fusion method of claim 24.

Images