Image fusion method, device, system, computer device and storage medium

Abstract

The application relates to an image fusion method, device, system, computer equipment and storage medium. The method comprises the following steps: acquiring a preoperative scanning image, an intraoperative endoscope image and an intraoperative ultrasound image of a target region of a patient, fusing the preoperative scanning image and the intraoperative endoscope image to generate a first fusion image, fusing the intraoperative ultrasound image and the intraoperative endoscope image to generate a second fusion image, and finally, superimposing the first fusion image and the second fusion image into the intraoperative endoscope image to generate a target fusion image. The method can realize accurate fusion of three different modalities and types of images, i.e. the preoperative scanning image, the intraoperative ultrasound image and the intraoperative endoscope image, the quality of the obtained fusion image is high, and the quality and accuracy of the fusion image are improved.

Description

Technical Field

[0001] This application relates to the field of medical testing technology, and in particular to an image fusion method, apparatus, system, computer equipment, and storage medium. Background Technology

[0002] Computed tomography (CT), magnetic resonance imaging (MRI), ultrasound imaging (US), and laparoscopic ultrasonography (LUS) are widely used in computer-aided diagnosis and surgical navigation. Especially in minimally invasive surgery, fused images, obtained by fusing preoperative and intraoperative images, provide a more intuitive view of the structure and location of various organs and tissues; therefore, fused images are widely used for surgical navigation and localization.

[0003] Currently, preoperative and postoperative fused images typically include preoperative CT / MRI images, intraoperative ultrasound images, and endoscopic images. However, because these three types of images are of different modalities and types, accurate fusion is difficult, thus affecting the quality of the fused image. Therefore, providing a method that can accurately fuse CT / MRI images, ultrasound images, and endoscopic images has become a pressing technical problem in the field of medical testing. Summary of the Invention

[0004] Based on this, it is necessary to provide an image fusion method, device, system, computer equipment, computer-readable storage medium, and computer program product that can achieve the fusion of three different modalities of images—preoperative CT / MRI images, intraoperative ultrasound images, and intraoperative endoscopic images—and improve the quality and accuracy of the fused images, in order to address the aforementioned technical problems.

[0005] In a first aspect, this application provides an image fusion method, the method comprising:

[0006] Acquire preoperative scan images, intraoperative endoscopic images, and intraoperative ultrasound images of the patient's target area;

[0007] The preoperative scan images and intraoperative endoscopic images are fused to generate the first fused image;

[0008] The intraoperative ultrasound image and the intraoperative endoscopic image are fused to generate a second fused image;

[0009] The first fused image and the second fused image are superimposed on the intraoperative endoscopic image to generate the target fused image.

[0010] In one embodiment, preoperative scan images and intraoperative endoscopic images are fused to generate a first fused image, including:

[0011] A first point cloud model is generated based on the preoperative scan images, and a second point cloud model is generated based on the intraoperative endoscopic images.

[0012] The first point cloud model and the second point cloud model are fused to obtain the first fused image.

[0013] In one embodiment, generating a first point cloud model based on preoperative scan images includes:

[0014] The preoperative scan image was segmented using image segmentation technology to obtain a first segmented image; the first segmented image includes organ and tissue information of the target region;

[0015] The first segmented image is reconstructed using 3D reconstruction technology to obtain the first 3D model;

[0016] The first 3D model is transformed to obtain the first point cloud model.

[0017] In one embodiment, generating a second point cloud model based on intraoperative endoscopic images includes:

[0018] Image segmentation technology is used to segment intraoperative endoscopic images to obtain a second segmented image; the second segmented image includes target organ information of the target region;

[0019] Obtain the pixel value and depth information of each pixel in the second segmented image;

[0020] A second point cloud model is generated based on the pixel value and depth information of each pixel.

[0021] In one embodiment, the first point cloud model and the second point cloud model are fused to obtain a first fused image, including:

[0022] Determine a preset number of first registration objects from the first point cloud model;

[0023] Determine the second registration object corresponding to the first registration object from the second point cloud model;

[0024] Calculate the registration matrix based on the first and second registration objects;

[0025] The first point cloud model and the second point cloud model are fused according to the registration matrix to obtain the first fused image.

[0026] In one embodiment, intraoperative ultrasound images and intraoperative endoscopic images are fused to generate a second fused image, including:

[0027] Intraoperative ultrasound images were segmented to obtain a third segmented image; the third segmented image included information about blood vessels and tumors in the target region.

[0028] The third segmented image is reconstructed using 3D reconstruction technology to obtain the second 3D model;

[0029] The second 3D model is transformed to obtain the third point cloud model;

[0030] The second point cloud model and the third point cloud model are fused to generate a second fused image.

[0031] In one embodiment, fusing the second point cloud model and the third point cloud model to generate a second fused image includes:

[0032] Obtain the first position information of the second point cloud model in the preset world coordinate system;

[0033] Obtain the second position information of the third point cloud model in the world coordinate system;

[0034] Based on the first and second location information, the second point cloud model and the third point cloud model are fused to obtain the second fused image.

[0035] Secondly, this application also provides an image fusion apparatus, which includes:

[0036] The acquisition module is used to acquire preoperative scan images, intraoperative endoscopic images, and intraoperative ultrasound images of the patient's target area;

[0037] The first fusion module is used to fuse preoperative scan images with intraoperative endoscopic images to generate a first fused image.

[0038] The second fusion module is used to fuse intraoperative ultrasound images with intraoperative endoscopic images to generate a second fused image.

[0039] The generation module is used to overlay the first fused image and the second fused image onto the intraoperative endoscopic image to generate the target fused image.

[0040] Thirdly, this application also provides an image fusion system, which includes an endoscope, an ultrasound device, and a processing device; the processing device is communicatively connected to both the endoscope and the ultrasound device.

[0041] Endoscopic equipment is used to acquire intraoperative endoscopic images of the target area of ​​the patient and send the intraoperative endoscopic images to the processing equipment;

[0042] Ultrasound equipment is used to acquire intraoperative ultrasound images of the target area and send the intraoperative ultrasound images to the processing equipment;

[0043] The processing device is used to receive intraoperative endoscopic images transmitted by the endoscopic device and intraoperative ultrasound images transmitted by the ultrasound device.

[0044] The processing device is also used to acquire preoperative scan images of the target area, and to fuse the preoperative scan images with intraoperative endoscopic images to generate a first fused image; to fuse the intraoperative ultrasound images with intraoperative endoscopic images to generate a second fused image; and to superimpose the first fused image and the second fused image onto the intraoperative endoscopic image to generate a target fused image.

[0045] In one embodiment, the system also includes a tracking device;

[0046] Tracking devices are used to track endoscopic and ultrasound equipment in real time.

[0047] Fourthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method steps described in any of the embodiments of the first aspect.

[0048] Fifthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the method steps of any of the embodiments of the first aspect described above.

[0049] Sixthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the method steps of any of the embodiments of the first aspect described above.

[0050] The aforementioned image fusion method, apparatus, system, computer equipment, storage medium, and computer program product acquire preoperative scan images, intraoperative endoscopic images, and intraoperative ultrasound images of the patient's target area. They then fuse the preoperative scan images with the intraoperative endoscopic images to generate a first fused image, and fuse the intraoperative ultrasound images with the intraoperative endoscopic images to generate a second fused image. Finally, the first and second fused images are superimposed onto the intraoperative endoscopic images to generate the target fused image. This enables accurate fusion of three different modalities and types of images—preoperative scan images, intraoperative ultrasound images, and intraoperative endoscopic images—and produces high-quality fused images, thus improving the quality and accuracy of the fused images. Attached Figure Description

[0051] Figure 1 This is an application environment diagram of the image fusion method in one embodiment;

[0052] Figure 2 This is a flowchart illustrating an image fusion method in one embodiment;

[0053] Figure 3 This is a flowchart illustrating the image fusion method in another embodiment;

[0054] Figure 4 This is a flowchart illustrating the image fusion method in another embodiment;

[0055] Figure 5 This is a flowchart illustrating the image fusion method in another embodiment;

[0056] Figure 6 This is a flowchart illustrating the image fusion method in another embodiment;

[0057] Figure 7 This is a flowchart illustrating the image fusion method in another embodiment;

[0058] Figure 8 This is a flowchart illustrating the image fusion method in another embodiment;

[0059] Figure 9 This is a schematic diagram of the positioning structure of the endoscope and ultrasound equipment in one embodiment;

[0060] Figure 10 This is a structural block diagram of an image fusion device in one embodiment;

[0061] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] Minimally invasive surgery offers unparalleled advantages over traditional open surgery, including less trauma, faster recovery, and fewer postoperative complications. With rapid advancements in medical technology and the advent of technologies like the electronic 3D laparoscope and the da Vinci surgical robot, the concept of "minimally invasive" has permeated all areas of surgery, bringing benefits to a wide range of patients. Laparoscopic surgery is a major form of minimally invasive surgery. However, due to the limited field of vision of the laparoscope and the complex and varied structures within the abdominal cavity, even experienced surgeons find it difficult to accurately locate target organs. Furthermore, tumors and important blood vessels are often deeply embedded within organs and tissues, making it impossible to "see through" the internal structures with a laparoscope. Relying solely on the surgeon's experience makes it difficult to correctly separate tumor tissue from normal tissue. Undercutting of the tumor can lead to its spread and recurrence, while overcutting can cause extensive loss of normal tissue, hindering the patient's rapid recovery.

[0064] Therefore, the common practice is to fuse preoperative CT / MRI images, intraoperative ultrasound images, and intraoperative endoscopic images. By fusing these images, the structure, location, surface information, internal blood vessels, and tumors of various organs and tissues can be displayed more intuitively. This method is widely used for surgical navigation and localization.

[0065] The advantages and disadvantages of the above three modalities of images are analyzed as follows:

[0066] (1) Preoperative CT / MR images are convenient for surgical planning and provide a clear representation of the surface and outline of organs, but they are not as accurate as ultrasound images in locating tumors and key blood vessels inside organs.

[0067] (2) Intraoperative endoscopic images act as the doctor's "eyes" during surgery, with the characteristics of visual intuitiveness. However, laparoscopic images have a limited field of view and cannot "see through" to observe the location of tumors and blood vessels inside organs and tissues.

[0068] (3) Intraoperative ultrasound images can achieve real-time imaging and show very obvious tumor and blood vessel imaging, but the imaging quality is poor, the signal-to-noise ratio is low, and it is not easy to interpret.

[0069] Among related technologies, the multimodal image fusion techniques used in endoscopic surgery mainly include: fusing preoperative CT / MRI images with intraoperative ultrasound images and then fusing them with intraoperative endoscopic images. Furthermore, the preoperative CT / MRI images and intraoperative ultrasound images are registered and fused using image feature-based methods, which do not yield intuitive results and lead to poor final image quality.

[0070] In view of this, this application proposes a preoperative and intraoperative multimodal image registration and fusion technology. Combining the advantages of the three image modalities mentioned above, firstly, three-dimensional reconstruction and surgical planning results are obtained based on preoperative CT / MR segmentation; during surgery, endoscopic images are acquired using endoscopic equipment, and the surface of the organ is segmented (surface features); during surgery, ultrasound images are acquired using ultrasound equipment, and deep blood vessels and tumors of the organ are segmented (deep features); finally, the three segmentation results are fused together using techniques such as registration, camera calibration, and coordinate system integration, and displayed on the endoscopic image. The method proposed in this application achieves accurate navigation and positioning of the target organ and its internal tumors and blood vessels by registering and fusing preoperative and intraoperative multimodal images, and by using ultrasound equipment to achieve real-time registration and fusion. This effectively reduces surgical time and improves surgical success rate. In addition, preoperative surgical plans can be successfully imported into intraoperative image data, and the relative positional relationships between organ surfaces, internal organs (tumors, blood vessels), and organs can be accurately located, enabling true intraoperative surgical navigation.

[0071] The image fusion method proposed in this application has the following technical advantages:

[0072] (1) By fusing preoperative CT / MRI images, intraoperative ultrasound images and intraoperative endoscopic images, accurate localization of organ parenchyma and key tissues inside the organ (tumor, blood vessels, nerves, bone tissue, etc.) can be achieved, which can effectively improve surgical accuracy, reduce the risk of tumor recurrence, and reduce the occurrence of surgical complications.

[0073] (2) It can quickly locate the relative position of the target organ and its surrounding organs and tissues, effectively reducing the operation time, thereby reducing the patient's pain and relieving the doctor's fatigue during long operation.

[0074] (3) It can display the target organ (surface features) and its internal tumor blood vessels (deep features) in real time using augmented reality (AR), and import the preoperative surgical plan (such as cutting range, depth, and entry path) into the intraoperative image; it can also display the lesion area and the area to be cut intuitively and accurately, avoid undersegmentation of tumor tissue and normal tissue, and achieve true surgical navigation.

[0075] (4) The preoperative planning results can be overlaid on the endoscopic image using augmented reality (AR) technology. Any organ can be specified for enhanced display, or any organ can be hidden to focus only on the target organ. The transparency of the enhanced display effect can also be adjusted, and different features of different modal images can be displayed with different colors and transparency.

[0076] (5) The image fusion method proposed in the embodiments of this application has universality:

[0077] a) It is applicable to a variety of minimally invasive surgical procedures, including general surgery, general surgery, gastroenterology, neurosurgery, gynecology, and orthopedics.

[0078] b) It is also applicable to various electronic endoscopes, such as laparoscopes, colonoscopes, hysteroscopes, bronchoscopes, brain endoscopes and other endoscopic equipment;

[0079] c) The image fusion method involved in this image fusion method can adopt a series of superimposed display methods such as video superimposition, optical superimposition (wearing AR glasses devices), projection superimposition, and 3D image superimposition.

[0080] The technical solutions involved in the embodiments of this application will be described below in conjunction with the application scenarios.

[0081] The image fusion method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the processing device 101 is communicatively connected to the endoscope device 102, the ultrasound device 103, and the display device 104. The display device 104 may include a single monitor to display the content of different devices; it can display content in separate areas or overlay content, etc. Alternatively, the display device may include multiple monitors, each displaying content from different devices. The processing device 101 can be a computing device with computational capabilities, such as a computer workstation or a local server.

[0082] In one embodiment, such as Figure 2 As shown, an image fusion method is provided, which can be applied to... Figure 1 Taking the processing equipment in the example, the following steps are included:

[0083] Step 201: Acquire preoperative scan images, intraoperative endoscopic images, and intraoperative ultrasound images of the patient's target area.

[0084] Among them, the preoperative scan images can be scanned images obtained by scanning the patient's target area with medical imaging equipment before the operation, including but not limited to CT scan images, MR scan images, etc.; the intraoperative endoscopic images can be raw images acquired in real time during the operation by moving the endoscopic equipment inside the patient's target area; the intraoperative ultrasound images can be ultrasound images acquired in real time during the operation by moving the ultrasound equipment inside the patient's target area.

[0085] Optionally, the processing device can acquire preoperative scan images of the patient's target area from a medical imaging device, a storage device connected to the medical imaging device, or a server connected to the medical imaging device. During the surgical operation on the patient's target area, the processing device can acquire intraoperative endoscopic images of the patient's target area in real time from an endoscope and intraoperative ultrasound images of the patient's target area in real time from an ultrasound device.

[0086] Optionally, at least one first intermediate device may be included between the processing device and the endoscopic device. This first intermediate device can process intraoperative endoscopic images acquired in real-time by the endoscopic device and then send them to the processing device, or it can forward intraoperative endoscopic images acquired in real-time by the endoscopic device to the processing device. Similarly, at least one second intermediate device may be included between the processing device and the ultrasound device. This second intermediate device can process intraoperative ultrasound images acquired in real-time by the ultrasound device and then send them to the processing device, or it can forward intraoperative ultrasound images acquired in real-time by the ultrasound device to the processing device. It should be noted that the number and function of the first intermediate device and the number and function of the second intermediate device are not limited in this embodiment. Furthermore, the first intermediate device and the second intermediate device may be the same device or different devices.

[0087] Step 202: The preoperative scan image and the intraoperative endoscopic image are fused to generate the first fused image.

[0088] The preoperative scan images are obtained by scanning the target area of ​​the patient, covering the target organ and its adjacent tissues. These images determine the relative positions of the organs, which can then be used to develop preoperative plans, such as determining the cutting range, depth, and incision path. The intraoperative endoscopic images are acquired by the endoscopic device as it moves within the target area, including the target organ. Image fusion of the preoperative scan images and the intraoperative endoscopic images aims to match the target organ in the intraoperative endoscopic images with the corresponding organ in the preoperative scan images; that is, to match the intraoperative endoscopic images to the corresponding positions in the preoperative scan images, resulting in a fused image.

[0089] Optionally, preoperative scan images and intraoperative endoscopic images can be input into a preset first fusion algorithm to obtain a first fused image. The first fused image includes information such as organ surface details, relative positional relationships between organs, and preoperative planning.

[0090] Optionally, the preoperative scan image and the intraoperative endoscopic image can be converted to make the converted preoperative scan image and the converted intraoperative endoscopic image the same type of image data, such as three-dimensional image data, three-dimensional point cloud data, etc.; then, the converted preoperative scan image and the converted intraoperative endoscopic image are subjected to image fusion processing to generate a first fused image.

[0091] It should be noted that traditional image fusion algorithms can be used to perform image fusion. Therefore, the specific implementation principle of traditional image fusion algorithms will not be described in detail in this application embodiment.

[0092] Step 203: The intraoperative ultrasound image and the intraoperative endoscopic image are fused to generate a second fused image.

[0093] Intraoperative ultrasound images are images acquired by ultrasound equipment as it moves within the target area, including the target organ and tumor tissue within the target area.

[0094] Optionally, intraoperative ultrasound images and intraoperative endoscopic images can be input into a preset second fusion algorithm to obtain a second fused image. This second fused image primarily includes information about tumors and blood vessels within the organ.

[0095] Optionally, the intraoperative ultrasound image and the intraoperative endoscopy image can be converted to make the converted intraoperative ultrasound image and the converted intraoperative endoscopy image the same type of image data, such as three-dimensional image data, three-dimensional point cloud data, etc.; then, the converted intraoperative ultrasound image and the converted intraoperative endoscopy image are subjected to image fusion processing to generate a second fused image.

[0096] It should be noted that traditional image fusion algorithms can be used to perform image fusion. Therefore, the specific implementation principle of traditional image fusion algorithms will not be described in detail in this application embodiment.

[0097] Step 204: Overlay the first fused image and the second fused image onto the intraoperative endoscopic image to generate the target fused image.

[0098] Optionally, the first fused image and the second fused image can be superimposed on the intraoperative endoscopic image using augmented reality methods. Alternatively, they can be superimposed using a series of superimposed display methods such as video superimposition, optical superimposition (e.g., wearing AR glasses), projection superimposition, and 3D image superimposition. This application does not limit the specific methods used in this embodiment.

[0099] In addition, the superimposed target fusion image includes all the information from the preoperative scan image, intraoperative endoscopic image, and intraoperative ultrasound image, which can be used for precise intraoperative navigation and positioning.

[0100] In the above image fusion method, preoperative scan images, intraoperative endoscopic images, and intraoperative ultrasound images of the patient's target area are acquired. The preoperative scan images and intraoperative endoscopic images are fused to generate a first fused image, and the intraoperative ultrasound images and intraoperative endoscopic images are fused to generate a second fused image. Finally, the first fused image and the second fused image are superimposed on the intraoperative endoscopic image to generate the target fused image. This method can accurately fuse three different modalities and types of images—preoperative scan images, intraoperative ultrasound images, and intraoperative endoscopic images—and the resulting fused images are of high quality, thus improving the quality and accuracy of the fused images.

[0101] In an optional embodiment of this application, the above-mentioned display device is used to display at least one of preoperative scan images, intraoperative endoscopic images, intraoperative ultrasound images, and the above-mentioned target fusion images. Optionally, when there is only one display device, these images can be displayed in a segmented manner. When there are multiple display devices, each display device can display at least one of the above-mentioned images. Each display device can display the same image or different images; this application does not limit this.

[0102] Figure 3 This is a flowchart illustrating an image fusion method in another embodiment. This embodiment relates to an optional implementation process where a processing device fuses preoperative scan images with intraoperative endoscopic images to generate a first fused image. Based on the above embodiment, as... Figure 3 As shown, step 202 above includes:

[0103] Step 301: Generate a first point cloud model based on the preoperative scan images, and generate a second point cloud model based on the intraoperative endoscopic images.

[0104] In other words, in this embodiment, both the preoperative scan image and the intraoperative endoscopic image are converted into the same type of point cloud model. Since the preoperative scan image is a tomographic image acquired by a medical imaging device, and the intraoperative endoscopic image is a two-dimensional planar image acquired by an endoscopic camera or endoscopic probe, the modalities of the preoperative scan image and the intraoperative endoscopic image are different. Therefore, image conversion algorithms can be preset for the preoperative scan image and the intraoperative endoscopic image respectively, so that the preoperative scan image is input into the first image conversion algorithm to obtain the first point cloud model, and the intraoperative endoscopic image is input into the second image conversion algorithm to obtain the second point cloud model.

[0105] Step 302: Fuse the first point cloud model and the second point cloud model to obtain the first fused image.

[0106] Optionally, a traditional image fusion algorithm can be used to fuse the first point cloud model and the second point cloud model to obtain the first fused image.

[0107] In this embodiment, a first point cloud model is generated based on the preoperative scan image, and a second point cloud model is generated based on the intraoperative endoscopic image. Then, the first point cloud model and the second point cloud model are fused to obtain a first fused image. That is, in this embodiment, the preoperative scan image and the intraoperative endoscopic image in different modalities are converted into point cloud models of the same modality, and then the first point cloud model and the second point cloud model in the same modality are subjected to image fusion processing to obtain the first fused image. Since multiple images of the same modality are easy to fuse, the accuracy and efficiency of image fusion can be greatly improved.

[0108] Figure 4 This is a flowchart illustrating an image fusion method in another embodiment. This embodiment relates to an optional implementation process where the processing device generates a first point cloud model based on preoperative scan images. Building upon the above embodiments, as shown... Figure 4 As shown, step 301 above includes:

[0109] Step 401: The preoperative scan image is segmented using image segmentation technology to obtain the first segmented image.

[0110] The first segmented image includes organ and tissue information of the target region. Optionally, the organ and tissue information includes organ label information and organ location information. The first segmented image may also include the relative positional relationship of each organ in the current region.

[0111] Optionally, image segmentation techniques can be used to segment the target organ and its adjacent organs in the preoperative scan image to obtain a first segmented image. Image segmentation is a traditional and relatively mature image processing technique; therefore, its specific principles will not be discussed in detail here.

[0112] Furthermore, after obtaining the first segmented image, the organs in the first segmented image can be identified, and different labels can be assigned to different organs or tissues, thereby obtaining the relative positional relationship between the organs.

[0113] Step 402: The first segmented image is reconstructed using three-dimensional reconstruction technology to obtain the first three-dimensional model.

[0114] Among them, 3D reconstruction technology is also a traditional and relatively mature image processing technology. The specific principles of image reconstruction will not be discussed in detail here.

[0115] Step 403: Convert the first 3D model to obtain the first point cloud model.

[0116] For those skilled in the art, converting 3D models into point clouds is a common image processing technique, which will not be described in detail here.

[0117] In this embodiment, the preoperative scan image is segmented using image segmentation technology to obtain a first segmented image; then, the first segmented image is reconstructed using 3D reconstruction technology to obtain a first 3D model; the first 3D model is converted to obtain a first point cloud model; wherein, the first segmented image includes organ and tissue information of the target region; the image conversion method provided in this embodiment can convert the preoperative scan image into a point cloud model, realize the conversion of image modes, and the conversion effect of the image conversion method provided in this embodiment is good.

[0118] Figure 5 This is a flowchart illustrating an image fusion method in another embodiment. This embodiment relates to an optional implementation process where the processing device generates a second point cloud model based on intraoperative endoscopic images. Building upon the above embodiments, as... Figure 5 As shown, step 301 above includes:

[0119] Step 501: The intraoperative endoscopic image is segmented using image segmentation technology to obtain a second segmented image.

[0120] The second segmented image includes target organ information of the target region, which may include the surface information of the organ.

[0121] For example, during laparoscopic surgery, several trocars can be created on the surface of the patient's abdomen and filled with carbon dioxide gas to obtain sufficient surgical space. The laparoscope tip is then inserted into the abdominal cavity through the trocars, and an appropriate angle is selected to maximize the exposure of the target organ within the laparoscope's field of vision. Intraoperative endoscopic images are then acquired and sent to a processing device.

[0122] After obtaining the intraoperative endoscopic image, the processing device can use an image segmentation algorithm to segment the intraoperative endoscopic image to obtain the second segmented image; optionally, the second segmented image can be a surface mask image, which includes the surface information of the organ.

[0123] Step 502: Obtain the pixel value and depth information of each pixel in the second segmented image.

[0124] Optionally, a preset feature extraction algorithm can be used to extract the pixel values ​​and depth information of each pixel in the second segmented image.

[0125] Step 503: Generate a second point cloud model based on the pixel values ​​and depth information of each pixel.

[0126] Optionally, a three-dimensional reconstruction can be performed based on the pixel values ​​and depth information of each pixel to generate a second point cloud model corresponding to the intraoperative endoscopic image.

[0127] In this embodiment, the intraoperative endoscopic image is segmented using image segmentation technology to obtain a second segmented image; the pixel value and depth information of each pixel in the second segmented image are obtained; then, a second point cloud model is generated based on the pixel value and depth information of each pixel; wherein, the second segmented image includes target organ information of the target region; the image conversion method provided in this embodiment can convert the intraoperative endoscopic image into a point cloud model, realize the conversion of image modality, and the conversion effect of the image conversion method provided in this embodiment is good.

[0128] Figure 6 This is a flowchart illustrating an image fusion method in another embodiment. This embodiment relates to an optional implementation process where a processing device fuses a first point cloud model and a second point cloud model to obtain a first fused image. Based on the above embodiment, such as... Figure 6 As shown, step 302 above includes:

[0129] Step 601: Determine a preset number of first registration objects from the first point cloud model.

[0130] Optionally, the first registration object can be a point, line, or surface in the first point cloud model.

[0131] When fusing the first point cloud model and the second point cloud model, image fusion can be achieved by registering the first point cloud model and the second point cloud model. The registration can be done using point-based registration or surface-based registration, etc.

[0132] For example, when using a point-based registration algorithm, several key points can be selected on the first point cloud model and the second point cloud model respectively, and then the registration matrix can be obtained through the correspondence between point pairs. That is, a preset number of first registration objects are determined from the first point cloud model, and the first registration objects are the features of the first key points in the first point cloud model.

[0133] Step 602: Determine the second registration object corresponding to the first registration object from the second point cloud model.

[0134] For example, after obtaining the features of a preset number of first key points in the first point cloud model, the features of multiple second key points corresponding to the multiple first key points in the first point cloud model can be obtained from the second point cloud model to obtain a second registration object corresponding to the first registration object of the first point cloud model.

[0135] Step 603: Calculate the registration matrix based on the first registration object and the second registration object.

[0136] Optionally, the registration matrix between the first and second registration objects can be obtained by performing operations such as rotation, translation, geometric correction, and projection transformation on the first and second registration objects.

[0137] Step 604: The first point cloud model and the second point cloud model are fused according to the registration matrix to obtain the first fused image.

[0138] Optionally, the first point cloud model can be fused to the second point cloud model based on the registration matrix to obtain the first fused image.

[0139] In this embodiment, a preset number of first registration objects are determined from the first point cloud model, and then a second registration object corresponding to the first registration object is determined from the second point cloud model. Next, a registration matrix is ​​calculated based on the first and second registration objects, and the first and second point cloud models are fused according to the registration matrix to obtain a first fused image. This enables the fusion processing between preoperative reconstructed images and intraoperative endoscopic reconstructed images, improving the feasibility of fused images and the effect of image fusion.

[0140] In an optional embodiment of this application, when the processing device fuses the first point cloud model and the second point cloud model to obtain the first fused image, it can also adopt another implementation method, namely, when registering the first point cloud model and the second point cloud model, it can also adopt a surface-based registration method. The surface-based registration method refers to registering using the surfaces of the first point cloud model and the second point cloud model to obtain a registration matrix. Then, based on the registration matrix, the first point cloud model is fused to the second point cloud model, so that it can be fused and displayed on the intraoperative endoscopic image to obtain a fused image.

[0141] Figure 7 This is a flowchart illustrating an image fusion method in another embodiment. This embodiment relates to an optional implementation process whereby a processing device fuses intraoperative ultrasound images and intraoperative endoscopic images to generate a second fused image. Based on the above embodiment, such as... Figure 7 As shown, step 203 above includes:

[0142] Step 701: Perform image segmentation processing on the intraoperative ultrasound image to obtain the third segmented image.

[0143] The third segmented image includes information on blood vessels and tumors in the target region.

[0144] For example, during laparoscopic surgery on a patient, an ultrasound probe can be inserted through a puncture site, and then an intraoperative ultrasound image at an appropriate angle can be selected so that the ultrasound image includes the target organ and tumor tissue. The acquired intraoperative ultrasound image is then sent to a processing device.

[0145] After obtaining the intraoperative ultrasound image, the processing device can use an image segmentation algorithm to segment the intraoperative ultrasound image to obtain the third segmented image; optionally, the third segmented image can be a segmentation mask image, which includes internal tissue information such as blood vessels and tumors.

[0146] Step 702: The third segmented image is reconstructed using three-dimensional reconstruction technology to obtain the second three-dimensional model.

[0147] Similarly, 3D reconstruction technology is a traditional and relatively mature image processing technology, and the specific principles of image reconstruction will not be discussed in detail here.

[0148] Step 703: Convert the second 3D model to obtain the third point cloud model.

[0149] Similarly, for those skilled in the art, converting 3D models into point clouds is a common image processing technique, which will not be described in detail here.

[0150] Step 704: Fuse the second point cloud model and the third point cloud model to generate a second fused image.

[0151] Alternatively, a traditional image fusion algorithm can be used to fuse the second point cloud model and the third point cloud model to obtain a second fused image.

[0152] In this embodiment, a third segmented image is obtained by segmenting the intraoperative ultrasound image; a second three-dimensional model is obtained by reconstructing the third segmented image using three-dimensional reconstruction technology; the second three-dimensional model is then converted to obtain a third point cloud model; subsequently, the second and third point cloud models are fused to generate a second fused image; wherein, the third segmented image includes information on blood vessels and tumors in the target region; that is, in this embodiment, intraoperative ultrasound images and intraoperative endoscopic images in different modalities can be converted into point cloud models of the same modality, and then the second and third point cloud models in the same modality are fused to obtain a second fused image. Since multiple images of the same modality are easy to fuse, the accuracy and efficiency of image fusion can be greatly improved.

[0153] Figure 8 This is a flowchart illustrating an image fusion method in another embodiment. This embodiment relates to an optional implementation process where a processing device fuses a second point cloud model and a third point cloud model to generate a second fused image. Based on the above embodiment, such as... Figure 8 As shown, step 704 above includes:

[0154] Step 801: Obtain the first position information of the second point cloud model in the preset world coordinate system.

[0155] Optionally, when an endoscopic device (such as the laparoscope mentioned above) moves in real time into the patient's target area, a positioning device can be integrated into the endoscopic device to accurately locate it and obtain real-time position information. Optionally, this positioning device can be an optical positioning device, an electromagnetic positioning device, etc. Preferably, an optical array can be attached to the end of the endoscopic device, and an optical tracker can be activated to track and record the spatial coordinates of the endoscopic device.

[0156] Since the second point cloud model is generated from intraoperative endoscopic images acquired in real time by the endoscopic device, and these intraoperative endoscopic images correspond to the current location of the endoscopic device, the first position information corresponding to the second point cloud model can be determined using the current position information of the endoscopic device. Specifically, the current position information of the endoscopic device can be determined by tracking the optical array attached to the end of the endoscopic device using an optical tracker.

[0157] Furthermore, the preset world coordinate system can be any coordinate system different from the coordinate system of the optical tracker, or it can be the coordinate system of the optical tracker. If the preset world coordinate system is the coordinate system of the optical tracker, then the position information of the second point cloud model obtained by the optical tracker is the first position information of the second point cloud model in the preset world coordinate system. If the preset world coordinate system is not the coordinate system of the optical tracker, then after obtaining the position information of the second point cloud model through the optical tracker, it is necessary to convert the position information of the second point cloud model in the coordinate system of the optical tracker into the first position information in the preset world coordinate system according to the transformation relationship between the coordinate system of the optical tracker and the preset world coordinate system.

[0158] Step 802: Obtain the second position information of the third point cloud model in the world coordinate system.

[0159] Optionally, when the ultrasound device (such as the ultrasound probe mentioned above) moves in real time into the patient's target area, a positioning device can be integrated into the ultrasound device to accurately locate it and obtain real-time position information. This positioning device can be an optical positioning device, an electromagnetic positioning device, or the like. Preferably, an electromagnetic tracking sensor can be attached to the front end of the ultrasound device, and the electromagnetic tracker can be activated to record the spatial coordinates of the ultrasound probe.

[0160] Optionally, such as Figure 9 As shown, an optical array 1 can be attached to the end of the endoscope device to track and record the spatial coordinates of the endoscope device using an optical tracker. Alternatively, an optical array 2 can be attached to the end of the ultrasound device to track and record the spatial coordinates of the ultrasound device using an optical tracker.

[0161] Optionally, if the preset world coordinate system is the coordinate system where the optical tracker is located, then the current position information of the ultrasound device can be obtained through the optical tracker. This current position information is the position information of the third point cloud model corresponding to the intraoperative ultrasound image acquired by the ultrasound device, which is the second position information of the third point cloud model in the world coordinate system.

[0162] Optionally, if the preset world coordinate system is not the coordinate system of the optical tracker, then after obtaining the position information of the third point cloud model through the optical tracker, it is also necessary to convert the position information of the third point cloud model in the coordinate system of the optical tracker into the second position information in the preset world coordinate system according to the transformation relationship between the coordinate system of the optical tracker and the preset world coordinate system.

[0163] Step 803: Based on the first location information and the second location information, the second point cloud model and the third point cloud model are fused to obtain the second fused image.

[0164] In this embodiment, the first position information of the second point cloud model in a preset world coordinate system is obtained; and the second position information of the third point cloud model in the world coordinate system is obtained; based on the first and second position information, the second point cloud model and the third point cloud model are fused to obtain a second fused image. In this embodiment, by unifying the second point cloud model and the third point cloud model into a preset world coordinate system, and then performing fusion processing on the second point cloud model and the third point cloud model according to the unified position information, the matching degree of each organ, tissue, blood vessel, etc. after fusion is high, the fusion error is small, and thus the accuracy of the obtained fused second fused image is high, which can improve the fusion effect of the fused image.

[0165] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0166] Based on the same inventive concept, this application also provides an image fusion apparatus for implementing the image fusion method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image fusion apparatus embodiments provided below can be found in the limitations of the image fusion method described above, and will not be repeated here.

[0167] In one embodiment, such as Figure 10 As shown, an image fusion device 1000 is provided, including: an acquisition module 1001, a first fusion module 1002, a second fusion module 1003, and a generation module 1004, wherein:

[0168] The acquisition module 1001 is used to acquire preoperative scan images, intraoperative endoscopic images, and intraoperative ultrasound images of the patient's target area.

[0169] The first fusion module 1002 is used to fuse preoperative scan images with intraoperative endoscopic images to generate a first fused image.

[0170] The second fusion module 1003 is used to fuse intraoperative ultrasound images and intraoperative endoscopic images to generate a second fused image.

[0171] The generation module 1004 is used to overlay the first fused image and the second fused image onto the intraoperative endoscopic image to generate the target fused image.

[0172] In one embodiment, the first fusion module 1002 includes a generation unit and a fusion unit; wherein, the generation unit is used to generate a first point cloud model based on the preoperative scan image and generate a second point cloud model based on the intraoperative endoscopic image; the first fusion unit is used to fuse the first point cloud model and the second point cloud model to obtain a first fused image.

[0173] In one embodiment, the aforementioned generation unit is specifically used to segment a preoperative scan image using image segmentation technology to obtain a first segmented image; to reconstruct the first segmented image using three-dimensional reconstruction technology to obtain a first three-dimensional model; and to convert the first three-dimensional model to obtain a first point cloud model; wherein the first segmented image includes organ and tissue information of the target region.

[0174] In one embodiment, the aforementioned generation unit is specifically used to segment the intraoperative endoscopic image using image segmentation technology to obtain a second segmented image; to acquire the pixel value and depth information of each pixel in the second segmented image; and to generate a second point cloud model based on the pixel value and depth information of each pixel; wherein the second segmented image includes target organ information of the target region.

[0175] In one embodiment, the first fusion unit is specifically used to determine a preset number of first registration objects from the first point cloud model; determine a second registration object corresponding to the first registration object from the second point cloud model; calculate a registration matrix based on the first registration object and the second registration object; and fuse the first point cloud model and the second point cloud model according to the registration matrix to obtain a first fused image.

[0176] In one embodiment, the second fusion module 1003 includes a segmentation unit, a reconstruction unit, a conversion unit, and a second fusion unit; wherein, the segmentation unit is used to perform image segmentation processing on the intraoperative ultrasound image to obtain a third segmented image; the third segmented image includes vascular and tumor information of the target region; the reconstruction unit is used to reconstruct the third segmented image using three-dimensional reconstruction technology to obtain a second three-dimensional model; the conversion unit is used to convert the second three-dimensional model to obtain a third point cloud model; and the second fusion unit is used to fuse the second point cloud model and the third point cloud model to generate a second fused image.

[0177] In one embodiment, the second fusion unit is specifically used to obtain the first position information of the second point cloud model in a preset world coordinate system; obtain the second position information of the third point cloud model in the world coordinate system; and fuse the second point cloud model and the third point cloud model based on the first position information and the second position information to obtain a second fused image.

[0178] Each module in the aforementioned image fusion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0179] In one embodiment, an image fusion system is provided, such as Figure 1 As shown, the image fusion system includes an endoscope, an ultrasound device, and a processing device; the processing device is communicatively connected to both the endoscope and the ultrasound device.

[0180] The system includes an endoscope for acquiring intraoperative endoscopic images of the target area and sending them to a processing device; an ultrasound device for acquiring intraoperative ultrasound images of the target area and sending them to a processing device; a processing device for receiving intraoperative endoscopic images from the endoscope and intraoperative ultrasound images from the ultrasound device; additionally, the processing device is used to acquire preoperative scan images of the target area and fuse the preoperative scan images with the intraoperative endoscopic images to generate a first fused image; fuse the intraoperative ultrasound images with the intraoperative endoscopic images to generate a second fused image; and superimpose the first fused image and the second fused image onto the intraoperative endoscopic image to generate a target fused image.

[0181] The image fusion system provided in this application embodiment has similar implementation principles and technical effects for the processing device as described in the various embodiments of the image fusion method above, and will not be repeated here.

[0182] In an optional embodiment of this application, the image fusion system may further include a tracking device for real-time tracking of the endoscopic device and the ultrasound device. Optionally, there may be one or more tracking devices; that is, one tracking device can be used to jointly achieve real-time tracking of the endoscopic device and the ultrasound device; of course, one tracking device can be used to achieve real-time tracking of the endoscopic device, and another tracking device can be used to achieve real-time tracking of the ultrasound device; this embodiment of the application does not limit this.

[0183] Optionally, the tracking device may include an optical tracker and an electromagnetic tracker. The electromagnetic tracker is unaffected by light obstruction and can "penetrate the human body" to track the position of electromagnetic sensors on the surface of organs, thus allowing the spatial position information of the ultrasound device to be tracked. Optical trackers have the advantage of real-time accuracy, but they are more sensitive to obstruction and cannot "penetrate the human body." Therefore, optical trackers need to track small balls attached to an optical array at the endoscope tip to record the spatial coordinate information of the endoscope device.

[0184] For example, refer to Figure 9 An optical array 1 can be attached to the end of the endoscope, allowing an optical tracker to track a small ball on the array and obtain the spatial coordinates of the endoscope in the tracker's coordinate system. Conversely, an electromagnetic sensor is attached to the front of the ultrasound device, and an electromagnetic tracker and optical array 2 are attached to its end. The electromagnetic tracker receives signals from the electromagnetic sensor, thus obtaining the spatial coordinates of the ultrasound device in the tracker's (optical array 2) coordinate system. The optical tracker then tracks the small ball on the optical array 2, obtaining its spatial coordinates in the tracker's coordinate system. Finally, based on these coordinates, a coordinate transformation is performed to obtain the ultrasound device's spatial coordinates in the tracker's coordinate system. This allows for real-time tracking of both the endoscope and ultrasound devices.

[0185] In one embodiment, a computer device is provided, which may be the processing device described above, and its internal structure diagram may be as shown below. Figure 10As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores image data such as preoperative scan images, intraoperative endoscopic images, and intraoperative ultrasound images. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an image fusion method.

[0186] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0187] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the image fusion method in any of the above embodiments.

[0188] The computer device provided in this application embodiment has a similar implementation principle and technical effect to the above method embodiment, and will not be described again here.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the image fusion method in any of the above embodiments.

[0190] The computer-readable storage medium provided in this embodiment is similar in principle and technical effect to the method embodiment described above, and will not be repeated here.

[0191] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the image fusion method in any of the above embodiments.

[0192] The computer program product provided in this embodiment has a similar implementation principle and technical effect to the method embodiment described above, and will not be repeated here.

[0193] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0194] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0195] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0196] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image fusion method, characterized in that, The method includes: Acquire preoperative scan images, intraoperative endoscopic images, and intraoperative ultrasound images of the patient's target area; A first point cloud model is generated based on the preoperative scan image, and a second point cloud model is generated based on the intraoperative endoscopic image. A preset number of first registration objects are determined from the first point cloud model. A second registration object corresponding to the first registration object is determined from the second point cloud model. A registration matrix is ​​calculated based on the first registration object and the second registration object. The first point cloud model and the second point cloud model are fused according to the registration matrix to obtain a first fused image. Wherein, the first point cloud model and the second point cloud model are point cloud models of the same modality. The first registration object is a feature of a first key point in the first point cloud model, and the second registration object is a feature of a second key point in the second point cloud model. The intraoperative ultrasound image and the intraoperative endoscopic image are fused to generate a second fused image; The first fused image and the second fused image are superimposed on the intraoperative endoscopic image to generate the target fused image.

2. The method according to claim 1, characterized in that, The step of generating the first point cloud model based on the preoperative scan image includes: The preoperative scan image is segmented using image segmentation technology to obtain a first segmented image; the first segmented image includes organ and tissue information of the target region; The first segmented image is reconstructed using 3D reconstruction technology to obtain a first 3D model; The first 3D model is converted to obtain the first point cloud model.

3. The method according to claim 1, characterized in that, The step of generating a second point cloud model based on the intraoperative endoscopic images includes: The intraoperative endoscopic image is segmented using image segmentation technology to obtain a second segmented image; the second segmented image includes target organ information of the target region; Obtain the pixel value and depth information of each pixel in the second segmented image; The second point cloud model is generated based on the pixel values ​​and depth information of each pixel.

4. The method according to claim 1, characterized in that, The step of fusing the intraoperative ultrasound image and the intraoperative endoscopic image to generate a second fused image includes: The intraoperative ultrasound image is segmented to obtain a third segmented image; the third segmented image includes information about blood vessels and tumors in the target region. The third segmented image is reconstructed using 3D reconstruction technology to obtain a second 3D model; The second 3D model is transformed to obtain the third point cloud model; The second point cloud model and the third point cloud model are fused to generate the second fused image.

5. The method according to claim 4, characterized in that, The step of fusing the second point cloud model and the third point cloud model to generate the second fused image includes: Obtain the first position information of the second point cloud model in the preset world coordinate system; Obtain the second position information of the third point cloud model in the world coordinate system; Based on the first location information and the second location information, the second point cloud model and the third point cloud model are fused to obtain the second fused image.

6. An image fusion apparatus, characterized in that, The device includes: The acquisition module is used to acquire preoperative scan images, intraoperative endoscopic images, and intraoperative ultrasound images of the patient's target area; A first fusion module is configured to generate a first point cloud model based on the preoperative scan image and a second point cloud model based on the intraoperative endoscopic image; determine a preset number of first registration objects from the first point cloud model; determine a second registration object corresponding to the first registration object from the second point cloud model; calculate a registration matrix based on the first registration object and the second registration object; and fuse the first point cloud model and the second point cloud model according to the registration matrix to obtain a first fused image; wherein the first point cloud model and the second point cloud model are point cloud models of the same modality; the first registration object is a feature of a first key point in the first point cloud model, and the second registration object is a feature of a second key point in the second point cloud model; The second fusion module is used to fuse the intraoperative ultrasound image and the intraoperative endoscopic image to generate a second fused image. The generation module is used to overlay the first fused image and the second fused image onto the intraoperative endoscopic image to generate a target fused image.

7. An image fusion system, characterized in that, The image fusion system includes an endoscope, an ultrasound device, and a processing device; the processing device is communicatively connected to both the endoscope and the ultrasound device. The endoscopic device is used to acquire intraoperative endoscopic images of the target area of ​​the patient and send the intraoperative endoscopic images to the processing device; The ultrasound device is used to acquire intraoperative ultrasound images of the target area and send the intraoperative ultrasound images to the processing device; The processing device is used to receive the intraoperative endoscopic images sent by the endoscopic device and the intraoperative ultrasound images sent by the ultrasound device. The processing device is further configured to acquire a preoperative scan image of the target area, and fuse the preoperative scan image with the intraoperative endoscopic image to generate a first fused image; and fuse the intraoperative ultrasound image with the intraoperative endoscopic image to generate a second fused image; The first fused image and the second fused image are superimposed on the intraoperative endoscopic image to generate a target fused image. The generation of the first fused image includes generating a first point cloud model based on the preoperative scan image and generating a second point cloud model based on the intraoperative endoscopic image; and determining a preset number of first registration objects from the first point cloud model. Determine the second registration object corresponding to the first registration object from the second point cloud model; Calculate the registration matrix based on the first registration object and the second registration object; The first point cloud model and the second point cloud model are fused according to the registration matrix to obtain the first fused image; wherein the first point cloud model and the second point cloud model are point cloud models of the same modality; the first registration object is the feature of the first key point in the first point cloud model, and the second registration object is the feature of the second key point in the second point cloud model.

8. The system according to claim 7, characterized in that, The system also includes tracking devices; The tracking device is used to track the endoscope and the ultrasound device in real time.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images