A real-time surgical navigation method and system based on image fusion

Abstract

The present application relates to the field of image data processing technology, and in particular to a real-time surgical navigation method and system based on image fusion. The method includes: obtaining a three-dimensional annotation model of a target surgical site of a target patient before surgery, the three-dimensional annotation model including image information of the target surgical site at different viewing angles and target annotation information of the image information; when performing surgery on the target surgical site of the target patient, obtaining a first image of the target surgical site at the current moment through an endoscope; determining the target image information corresponding to the first image from the three-dimensional annotation model, and determining the first annotation information corresponding to the target image information from the three-dimensional annotation model; performing image fusion on the first annotation information and the first image to obtain a second image, and displaying the second image through a display device, so as to perform real-time navigation of the surgery through the display device. Through the above technical solution, navigation of the surgical process can be effectively realized.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to a real-time surgical navigation method and system based on image fusion. Background Art

[0002] When medical personnel perform surgery on a patient, such as laparoscopic surgery, they can determine the location of the laparoscopic surgery and observe the situation during the laparoscopic surgery in real time through an endoscope to achieve surgical navigation, so that the medical personnel can perform surgery on the patient.

[0003] In the related art, for example, in the Chinese patent application document with publication number CN118383871A, a navigation assistance system for uterine cavity surgical field of view based on MRI image calibration is provided. The position of the surgical area is calibrated on the MRI uterine cavity inner wall digital model by a calibration unit, and the spatial relative position of the surgical area on the MRI uterine cavity inner wall digital model relative to the uterine horns on both sides is determined, expressed in vector form, and the left uterine horn relative vector and the right uterine horn relative vector are obtained; after the operation begins, a hysteroscopic device with a multi-point ranging function is used to perform a scanning measurement of the uterine cavity inner wall, and the modeling unit establishes an intraoperative digital model of the uterine cavity inner wall based on the point data of the uterine cavity inner wall obtained by the scanning measurement, identifies the intraoperative uterine horn position, and determines the intraoperative uterine horn coordinates; the deformation calculation unit calculates the relative position of the surgical point in the intraoperative uterine cavity inner wall digital model based on the MRI relative vector of the surgical point and the intraoperative coordinates of the uterine horn; the navigation unit determines the relationship between the surgical point and the current hysteroscope field of view, and prompts the surgeon through AR technology.

[0004] When navigating an operation, the related technology determines the relationship between the operation point and the current hysteroscope's field of view, and prompts the surgeon through AR technology. The surgeon performing the operation can only judge the operation situation through the intraoperative laparoscopic images. It takes a long time to judge the operation situation only through the intraoperative laparoscopic images. Therefore, the related technology is difficult to effectively realize navigation of the operation process. Summary of the invention

[0005] In order to overcome the problem in related technologies that it is difficult to effectively realize navigation of the surgical process, the present application provides a real-time surgical navigation method and system based on image fusion.

[0006] According to a first aspect of an embodiment of the present application, a real-time surgical navigation method based on image fusion is provided, comprising: obtaining a three-dimensional annotation model of a target surgical site of a target patient before surgery, the three-dimensional annotation model comprising image information of the target surgical site at different viewing angles and target annotation information of the image information; the target annotation information comprises at least one of organ annotation information, tissue annotation information, blood vessel annotation information, bone annotation information and lesion annotation information of the image information; when performing surgery on the target surgical site of the target patient, obtaining a first image of the target surgical site at a current moment through an endoscope; determining target image information corresponding to the first image from the three-dimensional annotation model, and determining first annotation information corresponding to the target image information from the three-dimensional annotation model; performing image fusion on the first annotation information and the first image to obtain a second image, and displaying the second image through a display device, so as to perform real-time navigation of the surgery through the display device.

[0007] In this way, when performing surgery on the target surgical site of the target patient, a first image of the target surgical site at the current moment is obtained through an endoscope; target image information corresponding to the first image is determined from the three-dimensional annotation model, and first annotation information corresponding to the target image information is determined from the three-dimensional annotation model; the first annotation information and the first image can be fused to obtain a second image, and the second image can be displayed through a display device to perform real-time navigation of the surgery through the display device, and the target annotation information includes at least one of organ annotation information, tissue annotation information, blood vessel annotation information, bone annotation information and lesion annotation information of the image information, thereby being able to provide a more valuable reference for the surgical process and effectively realize navigation of the surgical process.

[0008] Optionally, the three-dimensional annotation model is acquired in the following manner: obtaining a three-dimensional reconstruction model of the target surgical site of the target patient before the operation, the three-dimensional annotation model including image information of the target surgical site at different viewing angles; inputting the three-dimensional reconstruction model into a pre-trained target annotation model to obtain the three-dimensional annotation model output by the target annotation model; wherein the target annotation model is used to annotate the image information at different viewing angles in the three-dimensional reconstruction model with target annotation information, so as to output the three-dimensional annotation model.

[0009] In this way, a 3D reconstructed model of the target surgical part of the target patient before the operation is obtained, and the 3D reconstructed model is input into a pre-trained target annotation model to obtain the 3D annotation model output by the target annotation model. This can realize automatic annotation of the 3D reconstructed model. Compared with manual annotation of target annotation information, this can improve annotation efficiency and reduce manual labor intensity.

[0010] Optionally, the target annotation model is obtained by training in the following manner: obtaining a pre-constructed initial annotation model, and obtaining a training sample set and a verification sample set for the initial annotation model, the training sample set including multiple preset three-dimensional reconstruction models, and the verification sample set including multiple preset three-dimensional annotation models corresponding to the multiple preset three-dimensional reconstruction models; using the training sample set as the input of the initial annotation model, and using the verification sample set as the output of the initial annotation model, training the initial annotation model, and obtaining the target annotation model.

[0011] In this way, by training the initial annotation model, a target annotation model for annotating the three-dimensional reconstructed model can be obtained, so that the obtained target annotation model can be used to subsequently implement rapid annotation of the three-dimensional reconstructed model.

[0012] Optionally, the target annotation model is acquired in the following manner: receiving a target organ type for the target surgical site specified by a user; determining a target annotation model corresponding to the target organ type from multiple preset annotation models corresponding to multiple preset organ types; wherein the preset annotation model is obtained by training based on image information of the surgical site of the corresponding preset organ type and annotation information of the image information, and different preset annotation models correspond to different preset organ types.

[0013] In this way, by determining the target annotation model corresponding to the target organ type from multiple preset annotation models corresponding to multiple preset organ types, the target annotation model can be matched with the organ type of the target surgical site, making the annotation information in the obtained three-dimensional annotation model corresponding to the three-dimensional reconstruction model more accurate.

[0014] Optionally, the method also includes: when the target image information corresponding to the first image has been determined, replacing the target image information in the three-dimensional annotation model according to the first image to obtain the target three-dimensional annotation model; displaying the target three-dimensional annotation model through a display device; and displaying the target three-dimensional annotation model and the second image in different display areas of the display device.

[0015] In this way, the target image information in the three-dimensional annotation model is replaced according to the first image to obtain the target three-dimensional annotation model. For example, the target image information in the three-dimensional annotation model is replaced with the first image. Since the first image is collected during the operation, it can better reflect the situation of the surgical site at the current moment. Therefore, the obtained target three-dimensional annotation model can better reflect the actual situation of the surgical site of the target patient, so that the doctor can make an actual judgment based on the target three-dimensional model.

[0016] Furthermore, the three-dimensional annotation model is obtained by annotating the three-dimensional reconstructed model before the operation. Compared with the first image, the three-dimensional annotation model can better reflect the overall situation of the target surgical site of the target patient. The target image information in the three-dimensional annotation model is replaced according to the first image to obtain the target three-dimensional annotation model. The target three-dimensional model can take into account both real-time and integrity, so as to be better used for navigation during the surgical process.

[0017] In addition, compared to scanning the target surgical site and re-constructing the three-dimensional model during surgery, the target image information in the three-dimensional annotation model is replaced according to the first image to obtain the target three-dimensional annotation model. The obtained target three-dimensional annotation model can accurately reflect the actual image information of the target surgical site, reduce the amount of calculation required for model construction, and reduce the operating pressure of the terminal device; and can reduce the time required for model construction, which helps to advance the surgical process as soon as possible.

[0018] Optionally, the first image includes multiple sub-images, the first annotation information includes multiple sub-annotation information, and the target image information in the three-dimensional annotation model is replaced according to the first image to obtain the target three-dimensional annotation model, including: determining an annotated sub-image from the multiple sub-images of the first image, the annotated sub-image being a sub-image in the first image having corresponding sub-annotation information; and replacing the part corresponding to the target image information in the three-dimensional annotation model with the annotated sub-image to obtain the target three-dimensional annotation model.

[0019] Optionally, determining target image information corresponding to the first image from the three-dimensional annotation model includes: determining an evaluation value of the target pixel point based on the pixel value of the target pixel point in the first image and the pixel value of the first pixel point, the evaluation value being used to characterize the probability that the target pixel point belongs to noise, the first pixel point being other pixel points within a preset range of the target pixel point; filtering target pixel points in the first image whose evaluation values ​​are greater than a preset evaluation threshold to obtain a third image after filtering; and determining target image information corresponding to the third image from the three-dimensional annotation model.

[0020] In this way, by replacing the portion corresponding to the target image information in the 3D annotation model with the annotated sub-image, the target 3D annotation model can be quickly obtained, thereby reducing the time required to obtain the target 3D annotation model.

[0021] Optionally, an evaluation value of the target pixel is determined based on the pixel value of the target pixel in the first image and the pixel value of the first pixel, including: determining a first information entropy based on the pixel values ​​of all pixels within a preset range of the target pixel, and determining a second information entropy based on multiple pixel values ​​of multiple first pixels corresponding to the target pixel, so as to determine a first difference value between the first information entropy and the second information entropy; determining a second difference value in pixel value between the target pixel and the first pixel, and obtaining a first sum of multiple second difference values ​​based on multiple second difference values ​​corresponding to the multiple first pixels; multiplying the first difference value by the first sum to obtain a first product value, and obtaining the evaluation value of the target pixel based on the first product value.

[0022] Optionally, obtaining the evaluation value of the target pixel point based on the first multiplication value includes: determining a third difference value in grayscale value between a previous pixel point and a next pixel point for pixels within a preset range of the target pixel point; determining a second sum of multiple third difference values ​​corresponding to multiple pixels within the preset range, determining a first ratio of the second sum to the first multiplication value, and performing an exponential operation on the first ratio using an exponential function to obtain an exponential operation value; and using the reciprocal of the exponential operation value as the evaluation value of the target pixel point.

[0023] According to a second aspect of an embodiment of the present application, a real-time surgical navigation system based on image fusion is provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the steps of the real-time surgical navigation method based on image fusion provided in the first aspect of the present application are implemented.

[0024] The technical solution provided by the embodiments of the present application may include the following beneficial effects: when performing surgery on a target surgical site of a target patient, a first image of the target surgical site at the current moment is obtained through an endoscope; target image information corresponding to the first image is determined from a three-dimensional annotation model, and first annotation information corresponding to the target image information is determined from the three-dimensional annotation model; the first annotation information and the first image can be fused to obtain a second image, and the second image can be displayed through a display device to perform real-time navigation of the surgery through the display device, and the target annotation information includes at least one of organ annotation information, tissue annotation information, blood vessel annotation information, bone annotation information and lesion annotation information of the image information, thereby being able to provide a more valuable reference for the surgical process and effectively realize navigation of the surgical process.

[0025] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0027] Figure 1 is a flow chart of a real-time surgical navigation method based on image fusion according to an exemplary embodiment;

[0028] Figure 2 It is a structural schematic diagram of a real-time surgical navigation system based on image fusion according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0030] First, a brief introduction to the application scenario of the embodiment of the present application is given. In the application scenario of the present application, in order to realize the navigation of the surgical process, the image of the surgical site is currently mainly acquired through an endoscope, and the image captured by the endoscope is displayed through a display device so that the doctor can understand the situation of the surgical site.

[0031] However, when judging the surgical situation only by images collected by the endoscope, it takes a long time and may be difficult to obtain a relatively accurate judgment result, which affects the efficiency and effect of the operation. It can be seen that it is difficult to achieve effective navigation of the surgical process in related technologies.

[0032] In response to the above technical problems, an embodiment of the present application provides a real-time surgical navigation method and system based on image fusion. When performing surgery on a target surgical site of a target patient, a first image of the target surgical site at the current moment is obtained through an endoscope; target image information corresponding to the first image is determined from a three-dimensional annotation model, and first annotation information corresponding to the target image information is determined from the three-dimensional annotation model; the first annotation information and the first image can be image fused to obtain a second image, and the second image can be displayed through a display device to perform real-time navigation of the surgery through the display device, wherein the target annotation information includes at least one of organ annotation information, tissue annotation information, blood vessel annotation information, bone annotation information and lesion annotation information of the image information, thereby being able to provide a more valuable reference for the surgical process and effectively realize navigation of the surgical process.

[0033] Figure 1 is a flowchart of a real-time surgical navigation method based on image fusion according to an exemplary embodiment, which can be applied to terminal devices such as Figure 1 As shown, the method includes the following steps.

[0034] In step S101, a three-dimensional annotated model of a target surgical site of a target patient before surgery is obtained.

[0035] The three-dimensional annotation model includes image information of the target surgical site at different viewing angles and target annotation information of the image information; the target annotation information includes at least one of organ annotation information, tissue annotation information, blood vessel annotation information, bone annotation information and lesion annotation information of the image information.

[0036] Here, by obtaining the three-dimensional annotation model of the target surgical site of the target patient before the operation, it is helpful to subsequently utilize the target annotation information in the three-dimensional annotation model. The target annotation information can provide a more valuable reference for the surgical personnel during the operation; and the annotation of the image information of the target surgical site at different perspectives is completed before the operation, which can shorten the time required for the surgeon to make situation judgments during the operation, thereby completing the operation on the target patient faster and more accurately.

[0037] In step S102 , when performing surgery on a target surgical site of a target patient, a first image of the target surgical site at the current moment is acquired through an endoscope.

[0038] When performing surgery on a patient, a channel can be established for an endoscope so that the surgical site in the patient's body can be observed by placing the endoscope channel into the patient's body.

[0039] Endoscope is one of the commonly used medical devices. It includes a flexible part, a light source and a set of lenses. It can be used to provide channels and lighting for observing or operating body cavities, hollow organs and body pipes. The endoscope can enter the body through the natural orifices of the human body or surgical incisions to obtain images of the target location so that doctors can observe the situation at the surgical site.

[0040] When it is determined that the endoscope has reached the target surgical site, a first image of the target surgical site at the current moment can be acquired through the endoscope; for example, when observing the patient's stomach, an image of the patient's stomach can be acquired through the endoscope.

[0041] In step S103, target image information corresponding to the first image is determined from the three-dimensional annotation model, and first annotation information corresponding to the target image information is determined from the three-dimensional annotation model.

[0042] The three-dimensional annotation model of the target surgical site of the target patient before the operation includes image information of the target surgical site at different viewing angles and target annotation information of the image information; the image information of the target surgical site at different viewing angles may be image information corresponding to different areas of the target surgical site.

[0043] Since the image information in the three-dimensional annotation model corresponds to the target surgical site of the target patient, and the first image captured by the endoscope is captured at the target surgical site of the target patient, the first image can be matched with the image information of the target surgical site in the three-dimensional annotation model at different perspectives, and the target image information matching the first image can be determined from the multiple image information of the three-dimensional annotation model.

[0044] Among them, matching the first image with the image information of the target surgical site in the three-dimensional annotation model at different viewing angles can be achieved through feature point matching and histogram matching; for example, algorithms such as SIFT (Scale-invariant feature transform) and SURF (Speeded Up Robust Features) can be used to achieve matching between images.

[0045] When determining the target image information corresponding to the first image in the three-dimensional annotation model, since the three-dimensional annotation model includes target annotation information for the image information, the first annotation information corresponding to the target image information in the three-dimensional annotation model can also be determined.

[0046] Here, since the first image corresponds to the target image information, for example, the first image may be an image of the first area of ​​the target surgical site during surgery, and the target image information may be an image of the first area of ​​the target surgical site before surgery. By using the correspondence between the image information before surgery and the target annotation information in the three-dimensional annotation model, the correspondence between the image information of the target surgical site during surgery and the target annotation information can be determined, thereby achieving acquisition of target annotation information of different areas of the target surgical site.

[0047] In step S104, the first annotation information and the first image are fused to obtain a second image, and the second image is displayed through a display device so as to perform real-time navigation of the surgery through the display device.

[0048] For example, the endoscope can be communicatively connected to a terminal device, the terminal device can fuse the first annotation information and the first image to obtain a second image, the terminal device can send the second image to a display device, and display the second image through the display device; or, the second image can be displayed on a display device set on the terminal device to achieve real-time navigation of the operation.

[0049] The second image obtained after fusion includes not only the information of the target surgical site in the first image at the current moment, but also the annotation information of the area of ​​the target surgical site photographed by the endoscope at the current moment, which can provide valuable reference for the surgery through the annotation information.

[0050] For example, corresponding positions in the second image may be respectively labeled with text information such as “right internal carotid artery”, “left internal carotid artery”, and “basilar artery”, so as to facilitate the surgeon to understand the conditions of the arteries near the target surgical site.

[0051] The target labeling information includes at least one of organ labeling information, tissue labeling information, blood vessel labeling information, bone labeling information and lesion labeling information. Since the second image is obtained by performing image fusion processing based on the first labeling information, the second image can provide the surgeon with at least one of the organ labeling information, tissue labeling information, blood vessel labeling information, bone labeling information and lesion labeling information, thereby shortening the time required for the surgeon to make judgments. Therefore, it helps to shorten the duration of the operation and reduce the labor intensity of the surgeon. Moreover, since it can provide information other than the images collected by the endoscope, it can also improve the accuracy of the operation.

[0052] Through the real-time surgical navigation method based on image fusion provided in the embodiment of the present application, when performing surgery on the target surgical site of a target patient, a first image of the target surgical site at the current moment is obtained through an endoscope; target image information corresponding to the first image is determined from the three-dimensional annotation model, and first annotation information corresponding to the target image information is determined from the three-dimensional annotation model; the first annotation information and the first image can be fused to obtain a second image, and the second image can be displayed through a display device, so as to perform real-time navigation of the surgery through the display device.

[0053] The target annotation information includes at least one of the organ annotation information, tissue annotation information, blood vessel annotation information, bone annotation information and lesion annotation information of the image information, thereby being able to provide a more valuable reference for the surgical process and effectively realize navigation of the surgical process.

[0054] In one embodiment, the three-dimensional annotation model is acquired in the following manner: acquiring a three-dimensional reconstruction model of the target surgical site of the target patient before the operation, the three-dimensional annotation model including image information of the target surgical site at different viewing angles; inputting the three-dimensional reconstruction model into a pre-trained target annotation model to obtain the three-dimensional annotation model output by the target annotation model; wherein the target annotation model is used to annotate the image information at different viewing angles in the three-dimensional reconstruction model with target annotation information to output the three-dimensional annotation model.

[0055] The construction of a three-dimensional reconstruction model of the patient's surgical site before surgery can provide a valuable reference for the formulation of the patient's surgical plan and risk assessment.

[0056] In order to obtain a three-dimensional reconstructed model of the target surgical site of the target patient before the operation, a CT (Computed Tomography) scan can be performed on the target surgical site before the operation to obtain multiple CT images, and the multiple CT images can be three-dimensionally reconstructed to obtain a three-dimensional reconstructed model of the target surgical site of the target patient.

[0057] Alternatively, before the operation, MRI (Magnetic Resonance Imaging) can be performed on the target surgical site of the target patient to obtain an MRI image, and a three-dimensional reconstructed model of the target surgical site of the target patient can be obtained by performing three-dimensional reconstruction on multiple MRI images. The three-dimensional reconstruction can convert the two-dimensional image data into a three-dimensional reconstructed model.

[0058] In addition, multiple images obtained by means of positron emission tomography and ultrasonic imaging can be reconstructed in three dimensions to obtain a three-dimensional reconstructed model of the target surgical site. The embodiments of the present application do not impose any restrictions on the images used for three-dimensional reconstruction.

[0059] By inputting the three-dimensional reconstruction model into a pre-trained target annotation model, it is possible to implement the annotation of target annotation information of image information at different viewing angles in the three-dimensional reconstruction; for example, it is possible to implement the annotation of blood vessels around a target area of ​​a target surgical site of a target patient, or it is possible to implement the annotation of a lesion area in a target area of ​​a target surgical site of a target patient; or it is possible to implement the annotation of organ information in a target area of ​​a target surgical site of a target patient.

[0060] In this way, a 3D reconstructed model of the target surgical part of the target patient before the operation is obtained, and the 3D reconstructed model is input into a pre-trained target annotation model to obtain the 3D annotation model output by the target annotation model. This can realize automatic annotation of the 3D reconstructed model. Compared with manual annotation of target annotation information, this can improve annotation efficiency and reduce manual labor intensity.

[0061] In one embodiment, the target annotation model is obtained by training in the following manner: obtaining a pre-constructed initial annotation model, and obtaining a training sample set and a verification sample set for the initial annotation model, the training sample set including multiple preset three-dimensional reconstruction models, and the verification sample set including multiple preset three-dimensional annotation models corresponding to the multiple preset three-dimensional reconstruction models; using the training sample set as the input of the initial annotation model, and using the verification sample set as the output of the initial annotation model, training the initial annotation model to obtain the target annotation model.

[0062] For example, the preset three-dimensional reconstruction model may be a model that is not annotated with target annotation information, and the preset three-dimensional annotation model may be a model obtained after annotating image information in the three-dimensional reconstruction model with target annotation information.

[0063] The model structure of the initial annotation model may be, for example, a CNN (Convolutional Neural Network), a DNN (Deep Neural Network), or a network model of other structures, but this application does not impose any limitation on this.

[0064] In this way, by training the initial annotation model, a target annotation model for annotating the three-dimensional reconstructed model can be obtained, so that the obtained target annotation model can be used to subsequently implement rapid annotation of the three-dimensional reconstructed model.

[0065] In one embodiment, the target annotation model is acquired in the following manner: receiving a target organ type for the target surgical site specified by a user; determining a target annotation model corresponding to the target organ type from a plurality of preset annotation models corresponding to a plurality of preset organ types; wherein the preset annotation model is obtained by training based on image information of the surgical site of the corresponding preset organ type and annotation information of the image information, and different preset annotation models correspond to different preset organ types.

[0066] Different body parts correspond to different organ types. For example, the patient's head and neck correspond to different organ types. Since the blood vessels in body parts of different organ types are distributed differently, the characteristics of body parts of different organ types when not diseased are different, or the characteristics of the lesion areas in body parts of different organ types are different, or the surrounding organs of body parts of different organ types are distributed differently.

[0067] In order to improve the specificity of the annotation of target annotation information on the three-dimensional reconstructed model, thereby improving the accuracy of the annotation of target annotation information on the three-dimensional reconstructed model, the target organ type for the target surgical site specified by the user can be received; for example, the terminal device can display an organ type list for the user to select the target organ type from the organ type list; or, the terminal device can display a three-dimensional model of the human body, and the terminal device can receive the user's specified operation on the body part in the three-dimensional model of the human body to determine the corresponding target annotation model according to the specified body part.

[0068] The preset annotation model is obtained by training based on the image information of the surgical site of the corresponding preset organ type and the annotation information of the image information. Different preset annotation models are obtained by training based on the image information of the surgical site of different organ types and the annotation information of the image information. For example, the preset annotation model corresponding to the head can be obtained by training based on the image information of the head and the annotation information of the image information of the head.

[0069] In this way, by determining the target annotation model corresponding to the target organ type from multiple preset annotation models corresponding to multiple preset organ types, the target annotation model can be matched with the organ type of the target surgical site, making the annotation information in the obtained three-dimensional annotation model corresponding to the three-dimensional reconstruction model more accurate.

[0070] In one embodiment, when the target image information corresponding to the first image has been determined, the target image information in the three-dimensional annotation model can be replaced according to the first image to obtain the target three-dimensional annotation model; the target three-dimensional annotation model can be displayed through a display device; and the target three-dimensional annotation model and the second image can be displayed in different display areas of the display device.

[0071] The conditions of the same area of ​​the target surgical site of the target patient may be different before and during the operation, for example, the lesion area may expand during the operation; or, the conditions of the same area of ​​the target surgical site of the target patient during three-dimensional reconstruction before the operation may be different from the conditions immediately before the operation, for example, the surgical site may have vascular dilation during the operation.

[0072] The target image information in the three-dimensional annotation model is replaced according to the first image to obtain a target three-dimensional annotation model. For example, the target image information in the three-dimensional annotation model is replaced with the first image. Since the first image is collected during the operation, it can better reflect the situation of the surgical site at the current moment. Therefore, the obtained target three-dimensional annotation model can better reflect the actual situation of the surgical site of the target patient, so that the doctor can make a judgment on the actual situation based on the target three-dimensional model.

[0073] Furthermore, the three-dimensional annotation model is obtained by annotating the three-dimensional reconstructed model before the operation. Compared with the first image, the three-dimensional annotation model can better reflect the overall situation of the target surgical site of the target patient. The target image information in the three-dimensional annotation model is replaced according to the first image to obtain the target three-dimensional annotation model. The target three-dimensional model can take into account both real-time and integrity, so as to be better used for navigation during the surgical process.

[0074] In addition, compared to scanning the target surgical site and re-constructing the three-dimensional model during surgery, the target image information in the three-dimensional annotation model is replaced according to the first image to obtain the target three-dimensional annotation model. The obtained target three-dimensional annotation model can accurately reflect the actual image information of the target surgical site, reduce the amount of calculation required for model construction, and reduce the operating pressure of the terminal device; and can reduce the time required for model construction, which helps to advance the surgical process as soon as possible.

[0075] In one embodiment, the first image includes multiple sub-images, the first annotation information includes multiple sub-annotation information, and the target image information in the three-dimensional annotation model is replaced according to the first image to obtain the target three-dimensional annotation model, including: determining an annotated sub-image from the multiple sub-images of the first image, the annotated sub-image being a sub-image in the first image having the corresponding sub-annotation information; and replacing the part corresponding to the target image information in the three-dimensional annotation model with the annotated sub-image to obtain the target three-dimensional annotation model.

[0076] The annotation of image information is usually targeted at the key focus areas in the surgical site, such as the lesion area in the surgical site. Compared with non-key focus areas such as non-lesion areas in the surgical site, the key focus areas are more likely to change during surgery, or changes in the key focus areas during surgery have a greater impact on the surgery. Therefore, timely understanding of the situation of key focus areas including lesion areas is helpful for the progress of the surgery.

[0077] The first image includes multiple sub-images, and the first annotation information includes multiple sub-annotation information. Since the first image corresponds to target image information, the target image information in the three-dimensional reconstructed model corresponds to the first annotation information, and multiple sub-images in the first image can correspond to multiple sub-annotation information in the first annotation information. The multiple sub-images of the first image respectively correspond to multiple areas of the first image.

[0078] For example, the first image may include 5 sub-images, namely B1, B2, B3, B4 and B5, and the first annotation information may include 4 sub-annotation information, namely D1, D2, D3 and D4, wherein D1 and D2 are annotation information for sub-image B1, D3 is annotation information for sub-image B2, and D4 is annotation information for sub-image B3. Then sub-images B1, B2 and B3 are annotated sub-images, and sub-images B4 and B5 are unannotated sub-images.

[0079] Sub-images B1, B2 and B3 can be used to replace corresponding parts in the target image information respectively. For example, sub-image B1 can correspond to image information N1 in the target image information, sub-image B2 can correspond to image information N2 in the target image information, and sub-image B3 can correspond to image information N3 in the target image information. By replacing the part corresponding to the target image information in the three-dimensional annotation model with the annotated sub-image, the target three-dimensional annotation model can be quickly obtained, thereby reducing the time taken to obtain the target three-dimensional annotation model.

[0080] In one embodiment, target image information corresponding to the first image is determined from the three-dimensional annotation model, including: determining an evaluation value of the target pixel point based on the pixel value of the target pixel point in the first image and the pixel value of the first pixel point, the evaluation value being used to characterize the probability that the target pixel point belongs to noise, the first pixel point being other pixel points within a preset range of the target pixel point; filtering the target pixel points in the first image whose evaluation values ​​are greater than a preset evaluation threshold to obtain a third image after filtering; and determining the target image information corresponding to the third image from the three-dimensional annotation model.

[0081] The preset range can be, for example, a 5×5 range, which includes the target pixel and 24 first pixel points other than the target pixel; or, the preset range can also be a 7×7 range, which includes the target pixel and 48 first pixel points other than the target pixel.

[0082] The target pixel point is filtered, including: filtering the target pixel point through a target filter, where the target filter can be any one of the following: a mean filter, a median filter, a Gaussian filter, a bilateral filter, and a convolution filter.

[0083] By filtering the target pixel points in the first image whose evaluation values ​​are greater than the preset evaluation threshold, if the evaluation value of the target pixel points after the previous filtering process is still greater than the preset evaluation threshold, the target pixel points can be filtered for the next time.

[0084] For isolated pixels caused by noise in an image, for example, there is only one target pixel with an evaluation value greater than a preset evaluation threshold within a preset range. After the previous filtering process on the target pixel, the noise in the image can be effectively avoided.

[0085] As for non-isolated pixels caused by noise in the image, for example, there are multiple target pixels within the preset range whose evaluation values ​​are greater than the preset evaluation threshold. After the previous filtering process on the target pixel, the grayscale values ​​of other pixels around the target pixel may have changed. After filtering the target pixel again, the existence of noise in the image can be further avoided.

[0086] In the case where the evaluation value ranges from 0 to 1, for example, the evaluation value may be a normalized value, and the preset evaluation threshold may be between 0.6 and 0.8.

[0087] In this way, filtering is performed on the target pixel points in the first image whose evaluation values ​​are greater than the preset evaluation threshold to obtain a third image after filtering. This can effectively avoid the influence of noise in the first image on image information matching, thereby obtaining more accurate target image information.

[0088] In one embodiment, an evaluation value of a target pixel is determined based on a pixel value of the target pixel in a first image and a pixel value of the first pixel, including: determining a first information entropy based on the pixel values ​​of all pixels within a preset range of the target pixel, and determining a second information entropy based on multiple pixel values ​​of multiple first pixels corresponding to the target pixel, so as to determine a first difference value between the first information entropy and the second information entropy; determining a second difference value in pixel value between the target pixel and the first pixel, and obtaining a first sum of multiple second difference values ​​based on multiple second difference values ​​corresponding to the multiple first pixels; multiplying the first difference value by the first sum to obtain a first product value, and obtaining the evaluation value of the target pixel based on the first product value.

[0089] Information entropy can characterize the complexity of the grayscale values ​​corresponding to the pixels in the pixel set in terms of grayscale level; for example, the larger the information entropy, the higher the complexity of the grayscale values ​​corresponding to the pixels in the pixel set in terms of grayscale level, for example, the types of grayscale levels are more diverse; on the contrary, the smaller the information entropy, the lower the complexity of the grayscale values ​​corresponding to the pixels in the pixel set in terms of grayscale level, for example, the types of grayscale levels are relatively single.

[0090] For multiple pixel points included in a preset range, the texture complexity of the preset range can be characterized by information entropy; for example, the texture complexity of the entire preset range can be characterized by the first information entropy; and the texture complexity of the pixel points other than the target pixel point within the preset range, that is, the texture complexity of multiple first pixel points within the preset range, can be characterized by the second information entropy.

[0091] The information entropy can be determined based on the grayscale of the grayscale value corresponding to the pixel point in the pixel point set, and the frequency of occurrence of different grayscale levels. For example, for the grayscale values ​​corresponding to 25 pixels within the preset range of 5×5, there are 5 grayscale levels A1, A2, A3, A4 and A5, and the frequencies of occurrence of different grayscale levels are different. For example, the frequency of occurrence of grayscale level A1 is 4 / 25, the frequency of occurrence of grayscale level A2 is 10 / 25, the frequency of occurrence of grayscale level A3 is 6 / 25, the frequency of occurrence of grayscale level A4 is 3 / 25, and the frequency of occurrence of grayscale level A5 is 2 / 25. The information entropy can be determined based on the frequency of occurrence of different grayscale levels, and this application will not go into details here.

[0092] Through the first difference value between the first information entropy and the second information entropy, since the first information entropy is determined based on the grayscale values ​​of all pixels within the preset range, and the second information entropy is determined based on the grayscale values ​​of multiple first pixels within the preset range, the first difference value can reflect the contribution degree of the target pixel point to the texture complexity of the preset range, or the first difference value can characterize the probability that the target pixel point is a pixel point caused by noise in the preset range.

[0093] The first difference value between the first information entropy and the second information entropy can be, for example, the absolute value of the difference between the first information entropy and the second information entropy; or, the absolute value of the difference between the first information entropy and the second information entropy can be determined, and the ratio of the absolute value of the difference to the first information entropy is used as the first difference value between the first information entropy and the second information entropy.

[0094] By using the second difference value in grayscale value between the target pixel and the first pixel, the degree of difference between the target pixel and other pixels around the target pixel can be characterized. The second difference value can, for example, be the absolute value of the difference between the grayscale value of the target pixel and the grayscale value of the first pixel. By using the first sum of multiple second difference values ​​corresponding to multiple first pixels, the difference between the target pixel and other surrounding pixels can be reflected as a whole, or the first sum can characterize the probability that the target pixel is noise.

[0095] In this way, the first difference value is multiplied by the first sum to obtain the first product value. Both the first difference value and the first sum can better characterize the probability that the target pixel point is noise. The first product value can characterize the probability that the target pixel point is noise. Therefore, the evaluation value of the target pixel point is obtained according to the first product value, so that the obtained evaluation value can better characterize the probability that the target pixel point is noise.

[0096] The following uses an exemplary calculation formula as an example to explain the process of obtaining the first multiplication value in the embodiment of the present application: , where F is the first multiplication value of the target pixel, norm is the normalization function, G is the number of first pixels within the preset range of the target pixel, g is a positive integer less than or equal to G, M is the pixel value of the target pixel, and H g is the g-th first pixel point within the preset range of the target pixel point, P1 is the first information entropy, P2 is the second information entropy, Indicates the absolute value operation.

[0097] In one embodiment, the evaluation value of the target pixel point is obtained according to the first multiplication value, including: for the pixel points within a preset range of the target pixel point, determining a third difference value in grayscale value between the previous pixel point and the next pixel point; determining a second sum of multiple third difference values ​​of multiple pixel points within the preset range, determining a first ratio of the second sum and the first multiplication value, and using an exponential function to perform an exponential operation on the first ratio to obtain an exponential operation value; and using the reciprocal of the exponential operation value as the evaluation value of the target pixel point.

[0098] For example, when the size of the preset range is 3×3, the pixel located in the first row and first column within the preset range may be the previous pixel of the pixel located in the first row and second column within the preset range. The degree of difference between the pixels can be characterized by the third difference value in grayscale value between the previous pixel and the next pixel. The second difference value can, for example, be the absolute value of the difference between the grayscale value of the previous pixel and the grayscale value of the next pixel.

[0099] By respectively corresponding to the second sum of multiple third difference values ​​of multiple pixels within the preset range, the difference in grayscale values ​​of pixels within the preset range can be considered as a whole, thereby reflecting the texture complexity of the preset range where the target pixel is located through the second sum.

[0100] By using an exponential function to perform an exponential operation on the first ratio to obtain an exponential operation value, and using the reciprocal of the exponential operation value as the evaluation value of the target pixel point, the normalization processing of the first ratio can be achieved. The first product value can also be combined with the second sum to obtain the evaluation value of the target pixel point, so as to better characterize the probability that the target pixel point belongs to noise through the evaluation value.

[0101] Figure 2 FIG. 1 is a schematic diagram showing a real-time surgical navigation system based on image fusion according to an exemplary embodiment. Figure 2 The real-time surgical navigation system based on image fusion includes: a processor 1100 and a memory 1200, wherein the memory 1200 stores computer program instructions, and when the computer program instructions are executed by the processor 1100, all or part of the steps of the real-time surgical navigation method based on image fusion in the present application are implemented.

[0102] It should be understood that the features of some embodiments of the various applications described herein may be combined with each other, unless specifically stated otherwise.

[0103] Although terms such as "first", "second", and "third" may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Instead, these terms are only used to distinguish one component, part, region, layer, or section from another component, part, region, layer, or section. Therefore, without departing from the teachings of the examples described herein, the first component, part, region, layer, or section mentioned in the examples may also be referred to as the second component, part, region, layer, or section.

[0104] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description herein, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0105] In addition, the word "exemplary" is used herein to mean serving as an example, instance, or diagram. Any aspect or design described herein as "exemplary" is not necessarily to be understood as being advantageous over other aspects or designs. Instead, the use of the word exemplary is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or".

[0106] Likewise, although the present application has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. With particular regard to the various functions performed by the components (e.g., elements, resources, etc.) described above, unless otherwise noted, the terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific functions of the described components, even if not structurally equivalent to the disclosed structures.

[0107] Additionally, while particular features of the present application may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be desired and advantageous for any given or particular application.

[0108] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application, and the specification and embodiments are only considered as exemplary.

[0109] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A system for image fusion of medical images, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a method for image fusion of medical images is implemented, wherein the method for image fusion of medical images comprises: Acquire a three-dimensional annotated model of a target surgical site of a target patient before surgery, wherein the three-dimensional annotated model includes image information of the target surgical site at different viewing angles and target annotation information of the image information; the target annotation information includes at least one of organ annotation information, tissue annotation information, blood vessel annotation information, bone annotation information, and lesion annotation information of the image information; When performing surgery on the target surgical site of the target patient, acquiring a first image of the target surgical site at a current moment through an endoscope; Determining target image information corresponding to the first image from the three-dimensional annotation model includes: Determining an evaluation value of the target pixel point according to a pixel value of the target pixel point in the first image and a pixel value of the first pixel point includes: Determine a first information entropy according to the pixel values ​​of all pixels within a preset range of the target pixel, and determine a second information entropy according to multiple pixel values ​​of multiple first pixels corresponding to the target pixel, so as to determine a first difference value between the first information entropy and the second information entropy; Determine a second difference value in pixel value between the target pixel point and the first pixel point, and obtain a first sum of the plurality of second difference values ​​according to the plurality of second difference values ​​corresponding to the plurality of first pixel points; Multiplying the first difference value by the first sum to obtain a first multiplication value, and obtaining an evaluation value of the target pixel point according to the first multiplication value; The evaluation value is used to characterize the probability that the target pixel point belongs to noise, and the first pixel point is other pixel points within a preset range of the target pixel point; Performing filtering processing on target pixel points in the first image whose evaluation values ​​are greater than a preset evaluation threshold value to obtain a third image after filtering processing; Determining target image information corresponding to the third image from the three-dimensional annotation model; and determining first annotation information corresponding to the target image information from the three-dimensional annotation model; The first annotation information and the first image are fused to obtain a second image, and the second image is displayed through a display device.

2. The system for medical image fusion according to claim 1, characterized in that: The three-dimensional annotation model is obtained by: Acquire a three-dimensional reconstructed model of a target surgical site of a target patient before surgery, wherein the three-dimensional annotated model includes image information of the target surgical site at different viewing angles; Inputting the three-dimensional reconstruction model into a pre-trained target annotation model to obtain a three-dimensional annotation model output by the target annotation model; The target annotation model is used to annotate the target annotation information of the image information at different viewing angles in the three-dimensional reconstruction model, so as to output the three-dimensional annotation model.

3. The system for medical image fusion according to claim 2, characterized in that: The target annotation model is trained by the following method: Acquire a pre-built initial annotation model, and acquire a training sample set and a verification sample set for the initial annotation model, wherein the training sample set includes a plurality of preset 3D reconstruction models, and the verification sample set includes a plurality of preset 3D annotation models corresponding to the plurality of preset 3D reconstruction models; The training sample set is used as the input of the initial annotation model, the verification sample set is used as the output of the initial annotation model, the initial annotation model is trained, and a target annotation model is obtained.

4. The system for medical image fusion according to claim 2, characterized in that: The target annotation model is obtained in the following way: receiving a target organ type for the target surgical site specified by a user; Determining a target annotation model corresponding to the target organ type from a plurality of preset annotation models corresponding to a plurality of preset organ types; The preset annotation model is obtained by training according to the image information of the surgical site of the corresponding preset organ type and the annotation information of the image information, and different preset annotation models correspond to different preset organ types.

5. The medical image fusion system according to claim 1, characterized in that: The method further comprises: When target image information corresponding to the first image has been determined, the target image information in the three-dimensional annotation model is replaced according to the first image to obtain a target three-dimensional annotation model; The target three-dimensional annotated model is displayed through a display device; the target three-dimensional annotated model and the second image are displayed in different display areas of the display device.

6. The medical image fusion system according to claim 5, characterized in that: The first image includes a plurality of sub-images, the first annotation information includes a plurality of sub-annotation information, and the target image information in the three-dimensional annotation model is replaced according to the first image to obtain the target three-dimensional annotation model, including: Determine a marked sub-image from a plurality of sub-images of the first image, wherein the marked sub-image is a sub-image in the first image having corresponding sub-marking information; The part corresponding to the target image information in the 3D annotation model is replaced with the annotated sub-image to obtain the target 3D annotation model.

7. The medical image fusion system according to claim 1, characterized in that: The step of obtaining the evaluation value of the target pixel point according to the first multiplication value includes: For the pixel points within the preset range of the target pixel point, determining a third difference value in grayscale value between a previous pixel point and a next pixel point; Determine a second sum of a plurality of third difference values ​​corresponding to a plurality of pixel points within a preset range, determine a first ratio of the second sum to the first multiplication value, and perform an exponential operation on the first ratio using an exponential function to obtain an exponential operation value; The reciprocal of the exponential operation value is used as the evaluation value of the target pixel point.

Images