Thyroid endoscopic surgery navigation method and system based on MR fusion AF technology

Abstract

The invention provides a thyroid endoscopic surgery navigation method and system based on MR fusion AF technology, and the method comprises the steps: obtaining a plurality of image types under four-dimensional CT scanning, ultrasonic scanning and a fluorescent endoscope, receiving a real-time surgery video stream of a patient under an endoscopic surgery, the method comprises the steps of selecting a plurality of image types, selecting a target operation video frame from the image types, determining corresponding image defect features, intelligently screening out a corresponding target image from the plurality of image types based on the image defect features, performing image reconstruction with the target operation video frame, generating a target operation picture, and projecting the target operation picture to intelligent glasses of a surgeon. According to the method, the reconstruction effect of the target operation picture is comprehensively improved, the accuracy and reliability of thyroid endoscope operation navigation are improved, and a mixed-reality and multi-dimensional real-time operation view is provided for a surgeon, so that the surgeon can perform the operation more accurately, the damage to surrounding healthy tissues is reduced, and the safety and efficiency of the operation are improved.

Description

Technical Field

[0001] The present application relates to the technical field of thyroid laparoscopic surgery navigation. Specifically, the present application relates to a thyroid laparoscopic surgery navigation method and system based on MR fusion AF technology. Background Art

[0002] With the development of computer technology and the continuous advancement of surgical navigation techniques, computer-assisted surgical navigation is becoming increasingly widely used in surgical procedures. This technology significantly improves surgical accuracy and safety by providing precise surgical positioning and guidance. However, in actual surgical scenarios, such as laparoscopic thyroid surgery, surgeons still face several challenges and limitations.

[0003] First, the target surgical area is often obscured, preventing the surgeon from directly visualizing the thyroid lesion. In these cases, the surgeon must rely on information provided by navigation software to determine the precise location of the thyroid lesion. Furthermore, when the thyroid lesion is near vital nerves and blood vessels, the surgeon must exercise extreme caution during surgery to avoid damaging these tissues.

[0004] Furthermore, current navigation solutions for laparoscopic thyroid surgery rely primarily on registering the patient's preoperative CT scans with the actual surgical image to mark the position of surgical instruments in CT space. However, this approach still suffers from issues such as blurring, high noise levels, incomplete surgical angles, incomplete thyroid lesion tissue, incomplete matching with fused image information, and incomplete image fusion. These issues lead to poor surgical navigation accuracy and place high demands on the surgeon's hand-eye coordination.

[0005] In the technical solution with application number 202211536498.4, although it segments the key surgical areas in the preoperative CT image and selects anatomical landmarks, installs passive tracking tools on the patient's surgical area and endoscope respectively, uses a calibrated probe, uses anatomical landmarks to align the real patient with the three-dimensional model, calibrates the parameters of the endoscope under the navigation system, determines the transformation matrix of the passive tracking tool coordinate system of the patient's surgical area relative to the endoscope coordinate system, uses a pinhole model to simulate imaging, and projects and renders the three-dimensional model of the key surgical areas segmented preoperatively and processes it using distortion parameters to obtain a virtual image, which is then fused and displayed with the image captured in real time by the endoscope, so as to achieve the fusion of the virtual image and the real image of the endoscope and present them on the same screen, thereby reducing the doctor's judgment time during the operation and reducing the difficulty of hand-eye coordination. However, this technical solution cannot completely solve the defects existing in the surgical screen and has limitations. Summary of the Invention

[0006] The main purpose of this application is to provide a thyroid laparoscopic surgery navigation method and system based on MR fusion AF technology to comprehensively improve the image fusion effect, improve the accuracy of thyroid laparoscopic surgery navigation, and thus reduce the dependence on the surgeon's hand-eye coordination.

[0007] In order to achieve the above-mentioned purpose of the invention, the present application provides a thyroid laparoscopic surgery navigation method based on MR fusion AF technology, comprising:

[0008] Obtain multi-dimensional images of the patient's neck tissues and organs through four-dimensional CT scanning;

[0009] Obtaining an overall image of the thyroid gland, an image of thyroid lesions, an image of lymph nodes, and an image of venous connections surrounding the lymph nodes of the patient obtained through ultrasound scanning;

[0010] Based on the autofluorescence technology, a full-view image of the patient's thyroid gland is obtained under a fluorescence laparoscope;

[0011] receiving a real-time surgical video stream of the patient undergoing laparoscopic surgery, selecting a target surgical video frame from the real-time surgical video stream, and determining image defect features of the target surgical video frame;

[0012] According to the image defect characteristics of the target surgical video frame, at least one image is selected from the multidimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node perivascular image, and the overall thyroid development image to obtain a target image;

[0013] After reconstructing the target image and the target surgical video frame, a target surgical scene is obtained;

[0014] The target surgical image is projected onto the surgeon's smart glasses to assist the surgeon in performing the surgery through intelligent navigation.

[0015] Preferably, based on the image defect characteristics of the target surgical video frame, at least one image is selected from the multidimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node perivascular image, and the overall thyroid development image to obtain the target image, including:

[0016] Extracting image features related to the image defect features from the multidimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node perivascular image, and the overall thyroid development image, respectively, to obtain an image feature set, wherein the image feature set includes a plurality of image features;

[0017] Screening out image features that can optimize the image defect features from the image feature set to obtain target image features;

[0018] An image corresponding to the target image feature is determined to obtain a target image.

[0019] Preferably, after reconstructing the target image and the target surgical video frame, a target surgical scene is obtained, including:

[0020] Adjusting the size of the target image to be consistent with the size of the target surgical video frame to obtain a standard target image;

[0021] Dividing the standard target image into a plurality of rectangular image blocks of the same size to obtain a plurality of first rectangular image blocks;

[0022] Dividing the target surgical video frame into a plurality of rectangular image blocks of the same size to obtain a plurality of second rectangular image blocks, wherein the size of the first rectangular image blocks is the same as the size of the second rectangular image blocks;

[0023] performing similarity calculations on each of the first rectangular image blocks and a plurality of the second rectangular image blocks to obtain similarity calculation results;

[0024] According to the similarity calculation result, selecting a first target rectangular image block having a similarity greater than a preset similarity from the plurality of first rectangular image blocks;

[0025] Marking the blocked image area of ​​the target surgical video frame, determining a matching first target rectangular image block based on the adjacent second rectangular image block of the blocked image area and the similarity calculation result, and obtaining a second target rectangular image block;

[0026] The blocked image area of ​​the target surgical video frame is replaced by the adjacent first rectangular image block of the second target rectangular image block to obtain the target surgical picture.

[0027] Preferably, after reconstructing the target image and the target surgical video frame, a target surgical scene is obtained, including:

[0028] Setting a Gaussian kernel of the target image, wherein the Gaussian kernel is a matrix weighted based on a Gaussian distribution, and the matrix includes a plurality of Gaussian weights;

[0029] Performing normalized weight processing on the Gaussian kernel of the target image to obtain a target Gaussian kernel, wherein the sum of all Gaussian weights in the target Gaussian kernel is equal to 1;

[0030] For each pixel in the target image, multiply the pixel values ​​in the neighborhood of each pixel by the corresponding Gaussian weight in the target Gaussian kernel, and sum them up to obtain a Gaussian weight map corresponding to the target image;

[0031] After weighted multiplication of the target image and the corresponding Gaussian weight map, a weighted map corresponding to the target image is obtained;

[0032] Inputting the target surgical video frame into a pre-trained convolutional neural network to obtain a convolutional graph corresponding to the target surgical video frame;

[0033] The target surgical scene is obtained by fusing the weighted graph corresponding to the target image with the convolution graph corresponding to the target surgical video frame.

[0034] Furthermore, after reconstructing the target image and the target surgical video frame to obtain the target surgical scene, the method further includes:

[0035] Segmenting the real-time surgical video stream into multiple surgical video frames according to a timeline;

[0036] respectively determining the surgical position of the thyroid lesion tissue in each of the surgical video frames;

[0037] generating a surgical trajectory of the thyroid lesion tissue based on a time node of each surgical video frame and a surgical position of the thyroid lesion tissue;

[0038] marking the boundary of the thyroid lesion tissue in the thyroid lesion image;

[0039] predicting the real-time dynamic position of the thyroid lesion tissue to be removed based on the surgical trajectory and boundaries of the thyroid lesion tissue;

[0040] The real-time dynamic position is marked in the target surgical screen.

[0041] Furthermore, after obtaining the multi-dimensional image of the patient's neck tissues and organs through four-dimensional CT scanning, the method further includes:

[0042] Acquiring a reference multi-dimensional image having a resolution greater than a preset resolution;

[0043] Performing dimensionality reduction processing on the multidimensional image to obtain a first image;

[0044] Performing dimensionality reduction processing on the reference multidimensional image to obtain a second image;

[0045] extracting first image parameters of the first image and second image parameters of the second image respectively;

[0046] Calculating a structural similarity index between the first image and the second image based on the first image parameter and the second image parameter;

[0047] Determining whether the structural similarity index is greater than a preset structural similarity index;

[0048] If yes, performing the step of selecting at least one image from the multi-dimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node peripheral vein image, and the overall thyroid development image to obtain a target image;

[0049] If not, the multi-dimensional image is subjected to multi-level filtering and histogram equalization processing, and then input into a convolutional neural network until the structural similarity index corresponding to the output image is greater than a preset structural similarity index.

[0050] Furthermore, after obtaining the multi-dimensional image of the patient's neck tissues and organs through four-dimensional CT scanning, the method further includes:

[0051] Grayscale conversion, normalization, noise removal and edge detection are performed on the multidimensional image.

[0052] Preferably, the target surgical images include: a simulated surgical field of view reconstructed by four-dimensional CT, a simulated surgical field of view reconstructed by ultrasound images, a surgical field of view reconstructed by thyroid tissue under autofluorescence, and a real-time dynamic surgical field of view.

[0053] Preferably, the overall thyroid development image marks the thyroid gland, parathyroid glands, fluorescence intensity, range and boundaries.

[0054] This application also provides a thyroid laparoscopic surgery navigation system based on MR fusion AF technology, including:

[0055] The first acquisition module is used to obtain a multi-dimensional image of the patient's neck tissue and organs obtained through a four-dimensional CT scan;

[0056] A second acquisition module is used to obtain a thyroid panorama image, a thyroid lesion image, a lymph node image, and an image of the venous system surrounding the lymph nodes of the patient obtained by ultrasound scanning;

[0057] A third acquisition module is configured to acquire a full-view image of the patient's thyroid gland under a fluorescence laparoscope based on autofluorescence technology;

[0058] a determination module, configured to receive a real-time surgical video stream of the patient undergoing laparoscopic surgery, select a target surgical video frame from the real-time surgical video stream, and determine image defect features of the target surgical video frame;

[0059] a screening module, configured to screen at least one image from the multidimensional image, the panorama image of the thyroid gland, the image of the thyroid lesion, the lymph node image, the image of the venous vessels surrounding the lymph node, and the panorama image of the thyroid gland according to the image defect characteristics of the target surgical video frame, to obtain a target image;

[0060] An image reconstruction module, configured to reconstruct the target image and the target surgical video frame to obtain a target surgical picture;

[0061] The projection module is used to project the target surgical image onto the surgeon's smart glasses to assist the surgeon in performing the surgery through intelligent navigation.

[0062] The present application provides a thyroid laparoscopic surgical navigation method and system based on MR fusion AF technology. First, by obtaining a variety of image types under four-dimensional CT scans, ultrasound scans, and fluorescence laparoscopy, it can provide multi-dimensional images of neck tissue organs and detailed information about the thyroid gland and its surrounding tissues. Based on autofluorescence technology, the full-view development image of the thyroid gland under fluorescence laparoscopy can more clearly identify the thyroid tissue. Secondly, according to the image defect characteristics of the target surgical video frame, the corresponding target image is intelligently screened from a variety of image types, and the image is reconstructed with the target surgical video frame to generate a target surgical screen to comprehensively improve the reconstruction effect of the target surgical screen. At the same time, this fusion of multimodal images helps the surgeon understand the patient's condition from different angles and different levels, improve the accuracy of diagnosis, and thus improve the accuracy and reliability of thyroid laparoscopic surgical navigation. Finally, the reconstructed target surgical screen is projected onto the surgeon's smart glasses, providing the surgeon with a mixed reality, multi-dimensional real-time surgical field of view. The surgeon can obtain clearer visual information during the operation, thereby performing the surgical operation more accurately, reducing damage to surrounding healthy tissues, and improving the safety and efficiency of the operation. In summary, this application provides an innovative auxiliary means for thyroid laparoscopic surgery by combining multimodal imaging, autofluorescence imaging, real-time video processing, image reconstruction and mixed reality technology, significantly improving the accuracy, safety and efficiency of the surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flowchart of a thyroid laparoscopic surgery navigation method based on MR fusion AF technology according to an embodiment of the present application;

[0064] Figure 2 This is a flowchart of a thyroid laparoscopic surgery navigation method based on MR fusion AF technology according to another embodiment of the present application;

[0065] Figure 3 This is a flowchart of a thyroid laparoscopic surgery navigation method based on MR fusion AF technology according to another embodiment of the present application;

[0066] Figure 4 This is a flowchart of a thyroid laparoscopic surgery navigation method based on MR fusion AF technology according to another embodiment of the present application;

[0067] Figure 5This is a schematic block diagram of the structure of a thyroid laparoscopic surgery navigation system based on MR fusion AF technology according to an embodiment of the present application.

[0068] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0070] Mixed reality (MR) is an emerging immersive technology that blends the physical and digital worlds, enabling interaction between real-world and virtual world elements through 3D holograms. MR technology establishes an interactive feedback loop between the virtual and real worlds, enhancing the user experience's sense of reality.

[0071] AF (Auto-Fluorescence) is a technology that uses the property of endogenous fluorescent substances in biological tissues to emit fluorescence under the excitation of light of a specific wavelength for imaging and detection. Specifically, there are many endogenous fluorescent substances in biological tissues (such as collagen, elastic fibers, NADH, lipofuscin, etc.). When these substances are excited by light of a specific wavelength, they absorb light energy and transition to an excited state, and then release energy in the form of fluorescent signals. These fluorescent signals can be detected to reflect the pathological state of biological tissues. In the medical field, autofluorescence technology can be applied to cancer diagnosis, that is, by stimulating autofluorescence in human tissues to diagnose diseases such as cancer. For example, human tissues undergoing malignant changes will undergo spectral changes during fluorescence examination, which manifests as a lack of fluorescence, so that highly suspicious lesion areas can be identified in real time.

[0072] refer to Figure 1 In one embodiment, the present application provides a thyroid laparoscopic surgery navigation method based on MR fusion AF technology, the method comprising:

[0073] S11, obtaining a multi-dimensional image of the patient's neck tissues and organs through a four-dimensional CT scan;

[0074] S12, obtaining an overall image of the thyroid gland, an image of thyroid lesions, an image of lymph nodes, and an image of venous connections surrounding the lymph nodes of the patient obtained by ultrasound scanning;

[0075] S13. Obtaining a full-view image of the patient's thyroid gland under a fluorescence laparoscope based on autofluorescence technology;

[0076] S14, receiving a real-time surgical video stream of the patient undergoing laparoscopic surgery, selecting a target surgical video frame from the real-time surgical video stream, and determining image defect features of the target surgical video frame;

[0077] S15. Filtering at least one image from the multidimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node perivascular image, and the overall thyroid development image based on the image defect characteristics of the target surgical video frame to obtain a target image;

[0078] S16, reconstructing the target image and the target surgical video frame to obtain a target surgical scene;

[0079] S17: Projecting the target surgical image onto the surgeon's smart glasses to assist the surgeon in performing the surgery through intelligent navigation.

[0080] This embodiment can provide a series of innovative auxiliary means for surgical operations by combining a variety of advanced medical imaging and image processing technologies. The following is a detailed explanation of this embodiment:

[0081] Four-dimensional CT (4DCT) scanning is an advanced imaging technology that can capture the movement and changes of organs at different time points, generating multidimensional images. This multidimensional image provides detailed and dynamic information about the tissues and organs in the neck, encompassing both temporal and spatial information and capturing dynamic changes in tissue.

[0082] Ultrasound scanning is a non-invasive imaging technique used to obtain images of the thyroid gland, thyroid lesions, lymph nodes, and the choroidal structures surrounding the lymph nodes. The thyroid gland image provides information about the overall structure of the thyroid gland, while the thyroid lesion image reveals the specific location and characteristics of lesions within the thyroid gland. The lymph node image shows the status of lymph nodes that may be associated with thyroid disease, and the choroidal structures surrounding the lymph nodes provide detailed information about the tissues surrounding the lymph nodes. These images are helpful in assessing the condition of the thyroid gland and lymph node abnormalities.

[0083] This embodiment can use an imaging device capable of capturing near-infrared autofluorescence, such as the Fluobeam LX. This device can excite and capture autofluorescence emitted by thyroid tissue. It can capture autofluorescence images of thyroid tissue using a fluorescence laparoscope, analyze the autofluorescence images to identify the boundaries, structure, and any abnormal areas of the thyroid gland, and use the autofluorescence images obtained under the fluorescence laparoscope as a panoramic thyroid imaging image. This panoramic thyroid imaging image can display the entire thyroid gland, including normal and abnormal tissue. This autofluorescence technology can enhance the visibility of thyroid tissue, facilitate the identification and location of thyroid lesions, and enable physicians to utilize the tissue's own fluorescence properties for surgical navigation without the use of additional fluorescent markers.

[0084] Alternatively, the OPTO-CAM2100 endoscopic fluorescence camera system can be used. Its fluorescence light source must be equipped with near-infrared excitation light of a specific wavelength, typically around 805nm. Furthermore, the camera system must be able to capture autofluorescence signals at a corresponding wavelength, such as 835nm, for imaging. During surgery, the fluorescence laparoscope system is connected to the surgical equipment and turned on. Parameters such as the fluorescence gain are adjusted, typically to around 25. The surgical light is turned off to prevent interference from other light sources with autofluorescence imaging, but the operating room lights can remain on to allow for an overall view of the surgical field. When the fluorescence laparoscope enters the thyroid surgical area, the 805nm near-infrared light it emits illuminates the thyroid tissue, which, upon excitation, produces autofluorescence at 835nm. The camera system captures this fluorescence signal, processes it, and displays it in the surgical field. For example, in the multi-mode fluorescence mode of the endoscope, the upper left image on the screen can be developed in green, the middle left image can be developed in fluorescent white, and the lower left image can be developed in blue. The main image is a conventional laparoscope image under white light, and the three small images are fluorescent images, thus presenting a full-view image of the thyroid tissue and surrounding tissues including the parathyroid glands.

[0085] In one embodiment, the global thyroid imaging image marks the thyroid gland, parathyroid glands, fluorescence intensity, range, and boundaries.

[0086] Autofluorescence can also identify the parathyroid glands, as parathyroid tissue emits a unique fluorescence when illuminated by a specific wavelength of light. This fluorescence helps surgeons distinguish the parathyroid glands from surrounding thyroid tissue, fat, muscle, and lymph nodes during surgery.

[0087] For example, parathyroid tissue can emit autofluorescence with a maximum fluorescence intensity at 820 nm when illuminated by light of a wavelength of 785 nm. This fluorescence intensity can be used as an important parameter for identifying parathyroid glands.

[0088] The extent describes how widespread the thyroid tissue is or is affected. For example, in the context of thyroiditis or goiter, the extent might refer to the size of the affected thyroid tissue or the area involved. In ultrasound or other imaging reports, the extent helps doctors understand the spread of the disease.

[0089] The border is the demarcation between a thyroid nodule or other lesion and the surrounding normal tissue, as assessed by imaging techniques. Nodules with clear, regular borders are generally more likely to be benign, whereas nodules with irregular, fuzzy borders may indicate malignancy.

[0090] During surgery, the laparoscopic surgery system captures the patient's real-time surgical video stream. From this real-time surgical video stream, the system can automatically select target surgical video frames containing key surgical information, such as surgical video frames that locate thyroid lesions.

[0091] A pre-trained image processing model is used to perform feature analysis on the selected target surgical video frames to identify image defect characteristics such as blur, reflection, or occlusion. This information is crucial for subsequent image processing and analysis.

[0092] In one embodiment, the image defect characteristics include low image clarity, high noise, low contrast, obscured image areas, incomplete surgical angles, low clarity and integrity of thyroid lesion tissue, incomplete matching with fused image information, incomplete image information fusion, etc.

[0093] According to the image defect characteristics of the target surgical video frame, at least one target image that can optimize the image defect characteristics of the target surgical video frame is screened out from previously acquired multidimensional images, thyroid panorama images, etc. The target image will be used to match and reconstruct the target surgical video frame to improve the target surgical video frame.

[0094] Finally, the selected target image is reconstructed with the target surgical video frame to generate the target surgical image, thereby improving the accuracy and reliability of surgical navigation. For example, if the target surgical video frame has an image defect characterized by incomplete surgical angles, a panoramic thyroid image that can reflect the surgical image from multiple angles is selected from the acquired multi-dimensional image and panoramic thyroid image as the target image. This image is then spliced ​​with the target surgical video frame to create the target surgical image.

[0095] Image reconstruction is the process of generating an image from a series of measurements (e.g., raw data from X-rays, CT scans, MRI, or PET scans). These measurements are often affected by noise and incomplete data, so algorithms are needed to restore clear, accurate images. The goal of image reconstruction is to recover as detailed an image as possible from limited information to facilitate diagnosis or treatment planning. Methods for image reconstruction may include:

[0096] Sinusoidal image domain filtering method: such as the filtered back projection (FBP) algorithm, which performs Fourier transform on the collected projection data, then applies a filter, and finally performs inverse Fourier transform to obtain the image.

[0097] Iterative reconstruction method: Optimizes image reconstruction through multiple iterative processes, which can better handle noise and artifacts.

[0098] Projection domain CT image reconstruction method: Use deep learning models to directly learn and reconstruct images from raw projection data, which can handle incomplete or noisy data problems.

[0099] In one embodiment, the target surgical image includes: a simulated surgical field reconstructed by four-dimensional CT, a simulated surgical field reconstructed by ultrasound images, a surgical field reconstructed by thyroid tissue under autofluorescence, and a real-time dynamic surgical field.

[0100] This embodiment projects the reconstructed target surgical image onto the surgeon's smart glasses, allowing the surgeon to view the target surgical image in real time during the operation, thereby performing the surgery more accurately.

[0101] For example, suppose a patient requires thyroid tumor removal surgery. First, a 4D CT scan acquires dynamic images of the neck tissue and organs, followed by an ultrasound scan to obtain detailed images of the thyroid gland and lymph nodes. Before surgery, a Fluobeam LX device is used to stimulate and capture autofluorescence emitted by the thyroid tissue, and images of the thyroid gland are acquired under a fluorescence laparoscope. During the surgery, the laparoscopic surgical video stream is received in real time, and key video frames are selected. By analyzing the image defect characteristics of these video frames, the target image is selected from the previous images and reconstructed with the video frames to produce a clear target surgical image. Finally, this surgical image is projected onto the surgeon's smart glasses, allowing the surgeon to see the precise location of the thyroid tumor and surrounding structures in real time, enabling precise surgical resection. This technical solution improves surgical safety and efficiency while reducing patient risk. This enables a mixed reality fusion display of the preoperative imaging model and the intraoperative video stream.

[0102] The present application provides a thyroid laparoscopic surgical navigation method based on MR fusion AF technology. First, by obtaining a variety of image types under four-dimensional CT scans, ultrasound scans, and fluorescence laparoscopy, it can provide multi-dimensional images of neck tissue organs and detailed information about the thyroid gland and its surrounding tissues. Based on autofluorescence technology, the full-view development image of the thyroid gland under fluorescence laparoscopy can more clearly identify the thyroid tissue. Secondly, according to the image defect characteristics of the target surgical video frame, the corresponding target image is intelligently screened from a variety of image types, and the image is reconstructed with the target surgical video frame to generate a target surgical screen to comprehensively improve the reconstruction effect of the target surgical screen. At the same time, the fusion of the multimodal images helps the surgeon understand the patient's condition from different angles and different levels, improve the accuracy of diagnosis, and thus improve the accuracy and reliability of thyroid laparoscopic surgical navigation. Finally, the reconstructed target surgical screen is projected onto the surgeon's smart glasses, providing the surgeon with a mixed reality, multi-dimensional real-time surgical field of view. The surgeon can obtain clearer visual information during the operation, thereby performing the surgical operation more accurately, reducing damage to surrounding healthy tissues, and improving the safety and efficiency of the operation. In summary, this application provides an innovative auxiliary means for thyroid laparoscopic surgery by combining multimodal imaging, autofluorescence imaging, real-time video processing, image reconstruction and mixed reality technology, significantly improving the accuracy, safety and efficiency of the surgery.

[0103] refer to Figure 2 In one embodiment, based on the image defect characteristics of the target surgical video frame, at least one image is selected from the multidimensional image, the thyroid panorama image, the thyroid lesion image, the lymph node image, the lymph node perivascular image, and the thyroid panorama image to obtain the target image, including:

[0104] S151, extracting image features related to the image defect features from the multidimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node perivascular image, and the overall thyroid development image, respectively, to obtain an image feature set, the image feature set comprising a plurality of image features;

[0105] S152, filtering out image features that can optimize the image defect features from the image feature set to obtain target image features;

[0106] S153: Determine the image corresponding to the target image feature to obtain the target image.

[0107] In this embodiment, image features related to image defect features (such as edges, corners, textures, etc.) can be extracted from multidimensional images, thyroid panorama images, thyroid lesion images, lymph node images, lymph node perivascular images, and thyroid panorama development images through image processing algorithms (such as SI FT, SURF, ORB, etc.) to form an image feature set.

[0108] Image features that can optimize image defect characteristics are screened from the image feature set to obtain target image features. Specifically, by analyzing the extracted image features, features related to image defect characteristics (such as blur, noise, occlusion, etc.) are identified. Then, feature selection algorithms (such as principal component analysis (PCA) and feature importance assessment) are used to select the features most helpful in optimizing image defects, forming the target image feature set.

[0109] Finally, the filtered target image features are matched with the original image to find the target image corresponding to the target image features. The target image is an image filtered out from multidimensional images, thyroid panorama images, etc., so as to accurately filter out the required target image, which is conducive to the subsequent comprehensive improvement of the reconstruction effect of the target surgical picture.

[0110] refer to Figure 3 In one embodiment, after reconstructing the target image and the target surgical video frame, a target surgical scene is obtained, including:

[0111] S161, adjusting the size of the target image to be consistent with the size of the target surgical video frame to obtain a standard target image;

[0112] S162, dividing the standard target image into a plurality of rectangular image blocks of the same size to obtain a plurality of first rectangular image blocks;

[0113] S163: Divide the target surgical video frame into a plurality of rectangular image blocks of the same size to obtain a plurality of second rectangular image blocks, wherein the size of the first rectangular image blocks is the same as the size of the second rectangular image blocks;

[0114] S164: perform similarity calculation on each of the first rectangular image blocks and the plurality of second rectangular image blocks to obtain similarity calculation results;

[0115] S165: Filtering, based on the similarity calculation result, from the plurality of first rectangular image blocks, a first target rectangular image block having a similarity greater than a preset similarity;

[0116] S166, marking the blocked image area of ​​the target surgical video frame, and determining a matching first target rectangular image block based on the adjacent second rectangular image blocks of the blocked image area and the similarity calculation result, to obtain a second target rectangular image block;

[0117] S167 . Use the adjacent first rectangular image block of the second target rectangular image block to replace the blocked image area of ​​the target surgical video frame to obtain a target surgical picture.

[0118] This embodiment resizes the target image to match the size of the target surgical video frame, producing a standardized target image. Because the video stream and medical image may originate from different imaging devices, their resolutions and sizes may differ. By resizing the images, the pixel sizes of the two images are matched, facilitating subsequent image processing and analysis.

[0119] Then, both the standard target image and the target surgical video frame are evenly segmented into multiple rectangular image blocks of equal size, obtaining first rectangular image blocks corresponding to the multiple standard target images and second rectangular image blocks corresponding to the multiple target surgical video frames. Image segmentation divides the image into smaller regions so that each region can be processed and analyzed independently. This image segmentation can be performed based on a fixed-size grid.

[0120] By calculating the similarity between each first rectangular image block and multiple second rectangular image blocks respectively (such as using the structural similarity index SSIM, mean square error MSE, etc.), the degree of matching between each first rectangular image block and each second rectangular image block can be evaluated, providing a basis for subsequent image block screening.

[0121] By setting a similarity threshold, a first target rectangular image block having a similarity greater than a preset similarity can be screened out from a plurality of first rectangular image blocks to select an image block that best matches the surgical video frame.

[0122] Finally, in surgical videos, certain areas may be obscured by blood, tissue, or other reasons. Therefore, it is necessary to mark the obscured image area of ​​the target surgical video frame and, based on the adjacent second rectangular image blocks of the obscured image area and the similarity calculation results, determine a first target rectangular image block similar to the adjacent second rectangular image block to obtain a second target rectangular image block. The obscured image area of ​​the target surgical video frame is then replaced with the adjacent first rectangular image block of the second target rectangular image block to obtain the target surgical image. By replacing the obscured image area, the visibility and information content of the surgical video can be improved.

[0123] For example, suppose a patient is undergoing thyroid surgery, where the doctor needs to accurately identify and remove the thyroid tumor. During surgery, the surgical video stream may not clearly show the tumor boundary due to blood obstruction. Preoperative CT images can be resized to the same resolution as the surgical video and divided into small blocks. The similarity between the CT image blocks and the image blocks of the surgical video frame is calculated. The CT image blocks that best match the image blocks of the video frame are selected, and the occluded areas in the surgical video are identified and replaced with matching CT image blocks to reveal the obscured anatomical structures.

[0124] This embodiment can replace the blocked area, allowing doctors to obtain a clearer surgical field of view and reduce the risk of accidental damage to surrounding tissues. Accurate navigation information can help doctors identify the target area more quickly, shorten the operation time, reduce uncertainty and potential damage during the operation, and help patients recover after surgery.

[0125] refer to Figure 4 In one embodiment, after reconstructing the target image and the target surgical video frame, a target surgical scene is obtained, including:

[0126] S261, setting a Gaussian kernel of the target image, where the Gaussian kernel is a matrix weighted based on Gaussian distribution, and the matrix includes multiple Gaussian weights;

[0127] S262, performing normalized weight processing on the Gaussian kernel of the target image to obtain a target Gaussian kernel, where the sum of all Gaussian weights in the target Gaussian kernel is equal to 1;

[0128] S263. For each pixel in the target image, multiply the pixel values ​​in the neighborhood of each pixel by the corresponding Gaussian weight in the target Gaussian kernel, and sum them up to obtain a Gaussian weight map corresponding to the target image;

[0129] S264: performing weighted multiplication on the target image and the corresponding Gaussian weight map to obtain a weighted map corresponding to the target image;

[0130] S265: Input the target surgical video frame into a pre-trained convolutional neural network to obtain a convolutional graph corresponding to the target surgical video frame;

[0131] S266: Fusing the weighted graph corresponding to the target image with the convolution graph corresponding to the target surgical video frame to obtain a target surgical image.

[0132] This embodiment involves using a Gaussian kernel to process a target image and combining it with a convolutional neural network (CNN) to enhance the quality of surgical video frames, thereby improving the accuracy of surgical navigation.

[0133] Specifically, a Gaussian kernel is a matrix weighted by a Gaussian distribution, where the weight of each element is determined by the Gaussian distribution. The Gaussian kernel is used for image smoothing by making the value of each pixel in the image depend not only on itself but also on the values ​​of surrounding pixels, which are weighted by the Gaussian distribution.

[0134] The Gaussian kernel of the target image is normalized and weighted to obtain the target Gaussian kernel, so that the sum of all Gaussian weights in the target Gaussian kernel is equal to 1. This normalization ensures that the brightness of the image does not change during the subsequent weighted averaging process, but only the distribution of pixel values ​​changes to highlight certain features.

[0135] For each pixel in the target image, the pixel values ​​in its neighborhood are multiplied by the corresponding Gaussian weight in the target Gaussian kernel, and the sum is calculated to obtain the Gaussian weight map corresponding to the target image. Gaussian blurring of the target image can reduce image noise while retaining the edge features of the image.

[0136] The target image is multiplied with its corresponding Gaussian weight map to generate a weighted map. This map further enhances the image's features, making it more suitable for subsequent processing. Simultaneously, the target surgical video frame is fed into a pre-trained convolutional neural network to generate the corresponding convolutional map. This convolutional neural network extracts deep features from the video frames, facilitating understanding of the image's content.

[0137] Finally, the weighted graph of the target image is fused with the convolutional graph of the target surgical video frame to produce the target surgical image. This fusion process combines the image features processed by the Gaussian kernel with the deep features extracted by the convolutional neural network, generating a clear and rich image for surgical navigation.

[0138] For example, suppose a patient is undergoing thyroid surgery, and the doctor needs to accurately identify and remove the thyroid tumor. During the operation, the surgical video may not clearly show the tumor boundary due to blood obstruction. Through the above technical solution: Gaussian kernel is applied to the preoperative CT image to reduce noise and enhance edge features, the processed CT image is fused with the current frame in the surgical video to generate an image containing more information, and a convolutional neural network is used to extract features from the surgical video frame, such as the edge and texture information of the tumor. The extracted features are fused with the CT image to generate a clear target surgical picture, guiding the doctor to accurately remove the tumor.

[0139] This embodiment enhances image edge and texture features, allowing doctors to more accurately identify tumor boundaries and minimize damage to healthy tissue. Precise navigation information helps doctors more quickly identify target areas, shortening surgical time. It also reduces uncertainty and potential injuries during surgery, aiding patient recovery.

[0140] In one embodiment, after reconstructing the target image and the target surgical video frame to obtain the target surgical scene, the method further includes:

[0141] Segmenting the real-time surgical video stream into multiple surgical video frames according to a timeline;

[0142] respectively determining the surgical position of the thyroid lesion tissue in each of the surgical video frames;

[0143] generating a surgical trajectory of the thyroid lesion tissue based on a time node of each surgical video frame and a surgical position of the thyroid lesion tissue;

[0144] marking the boundary of the thyroid lesion tissue in the thyroid lesion image;

[0145] predicting the real-time dynamic position of the thyroid lesion tissue to be removed based on the surgical trajectory and boundaries of the thyroid lesion tissue;

[0146] The real-time dynamic position is marked in the target surgical screen.

[0147] This embodiment analyzes real-time surgical video streams and combines them with multidimensional medical imaging data to accurately locate and dynamically track thyroid lesions, thereby assisting decision-making during surgery. Specifically, a real-time surgical video stream is a continuous sequence of images. By segmenting the video stream along the timeline, individual images representing different time points during the surgical process, i.e., video frames, can be obtained. For example, the ffmpeg library can be used to extract video frames.

[0148] Image processing techniques, such as feature point detection and tracking algorithms, are used to determine the surgical location of the resected thyroid lesion in each frame of the surgical video. Preoperative medical imaging data, such as CT and MRI images, can also be combined to improve positioning accuracy.

[0149] By recording the positional changes of the resected thyroid lesion in consecutive video frames, its trajectory during surgery can be generated. Simultaneously, image segmentation algorithms, such as deep learning-based segmentation models, are used to automatically identify and mark the boundaries of the thyroid lesion. Boundary marking is crucial for precise lesion removal and minimizes damage to healthy tissue.

[0150] Finally, the real-time dynamic position of the thyroid lesion tissue to be removed is predicted based on the surgical trajectory and boundary of the thyroid lesion tissue. For example, the real-time position of the thyroid lesion tissue is predicted by combining the surgical trajectory and boundary information, such as Kalman filtering or deep learning models. The predicted real-time position information of the thyroid lesion tissue is superimposed on the target surgical screen to superimpose virtual information (such as the position of the thyroid lesion tissue) on the doctor's field of view, providing the doctor with intuitive navigation information.

[0151] For example, suppose a patient is about to undergo thyroid tumor removal surgery. The doctor first obtains multidimensional and panoramic images of the thyroid gland through a four-dimensional CT scan and ultrasound examination. During the operation, the doctor wears AR glasses and watches the surgical video stream in real time. The surgical video stream is segmented into video frames in real time, with each frame representing a moment of the operation. In each video frame, image processing technology can be used to determine the location of the resected thyroid tumor. The surgical trajectory of the thyroid tumor is generated based on the position changes of the resected thyroid tumor in consecutive video frames. An AI algorithm can be used to mark the tumor boundary in the ultrasound image. Based on this surgical trajectory and boundary information, the real-time dynamic position of the thyroid tumor to be removed is predicted. The predicted location of the thyroid tumor to be removed is marked in real time on the surgical screen displayed by the smart glasses, guiding the surgeon to accurately remove the thyroid tumor. The surgeon can track the location of the thyroid tumor in real time during the operation, improving the accuracy and safety of the operation.

[0152] In one embodiment, after obtaining a multi-dimensional image of the patient's neck tissues and organs through a four-dimensional CT scan, the method further includes:

[0153] Acquiring a reference multi-dimensional image having a resolution greater than a preset resolution;

[0154] Performing dimensionality reduction processing on the multidimensional image to obtain a first image;

[0155] Performing dimensionality reduction processing on the reference multidimensional image to obtain a second image;

[0156] extracting first image parameters of the first image and second image parameters of the second image respectively;

[0157] Calculating a structural similarity index between the first image and the second image based on the first image parameter and the second image parameter;

[0158] Determining whether the structural similarity index is greater than a preset structural similarity index;

[0159] If yes, performing the step of selecting at least one image from the multi-dimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node peripheral vein image, and the overall thyroid development image to obtain a target image;

[0160] If not, the multi-dimensional image is subjected to multi-level filtering and histogram equalization processing, and then input into a convolutional neural network until the structural similarity index corresponding to the output image is greater than a preset structural similarity index.

[0161] This embodiment obtains a reference multidimensional image with a resolution greater than a preset resolution and performs dimensionality reduction processing on the multidimensional image and the reference multidimensional image, respectively, to obtain a first image corresponding to the multidimensional image and a second image corresponding to the reference multidimensional image. The preset resolution can be customized. The reference multidimensional image can be obtained from a high-resolution medical imaging device, such as CT or MRI, which provides rich anatomical and pathological information. Dimensionality reduction processing can be performed through image compression or resolution reduction to make the image more suitable for real-time processing and analysis.

[0162] First image parameters of the first image and second image parameters of the second image are extracted. These image parameters may include features such as pixel values, texture, shape, and signal intensity distribution. Simultaneously, a structural similarity index is calculated between the two images based on the first and second image parameters. This structural similarity index is a measure of the visual similarity between the two images, taking into account image brightness, contrast, and structural information.

[0163] In one embodiment, when calculating the structural similarity index between the first image and the second image based on the first image parameter and the second image parameter, the pixel mean of the first image can be calculated based on the first image parameter to obtain the first pixel mean, and the pixel mean of the second image can be calculated based on the second image parameter to obtain the second pixel mean; then, the ratio of the first pixel mean to the second pixel mean is calculated, and the ratio is added to a first constant to obtain a first value, and the first constant can be set to 0.9; in addition, the variance of the first image is calculated based on all pixel values ​​of the first image parameter to obtain a first variance, and the variance of the second image is calculated based on all pixel values ​​of the second image parameter to obtain a second variance; finally, the sum of the squares of the first variance and the second variance is calculated, and the sum of the squares of the first variance and the second variance is added to a second constant to obtain a second value, and the second constant can be set to 0.95. The ratio of the first value to the second value is calculated to obtain the structural similarity index between the first image and the second image.

[0164] It is determined whether the calculated structural similarity index is greater than a preset structural similarity index. If the structural similarity index is greater than a preset threshold, it means that the two images are visually similar enough to be used for surgical navigation.

[0165] If the structural similarity index does not meet the requirements, the multidimensional image is subjected to multi-level filtering and histogram equalization to improve image quality and enhance the effectiveness of subsequent processing. The processed image is then input into a pre-trained convolutional neural network until the structural similarity index corresponding to the output image exceeds the preset structural similarity index. Convolutional neural networks can extract deep features from images and perform nonlinear mapping to improve image recognition and classification capabilities.

[0166] For example, a surgeon first acquires a high-resolution CT image and then uses dimensionality reduction to generate two images. By calculating the structural similarity index of these two images, the surgeon finds that their similarity is insufficient for surgical navigation. Therefore, the surgeon applies multi-layer filtering and histogram equalization to the images and feeds them into a convolutional neural network. After several iterations, the final output image meets the preset structural similarity index threshold and is selected as the target image for surgical navigation.

[0167] This embodiment can accurately analyze the image quality of multidimensional images through structural similarity index comparison, and after multi-level filtering, histogram equalization and convolutional neural network processing, ensure the high quality of surgical navigation images, so that doctors can more accurately identify and remove tumors; at the same time, improving the clarity and contrast of the image also helps to avoid damage to surrounding healthy tissues.

[0168] In one embodiment, after obtaining a multi-dimensional image of the patient's neck tissues and organs through a four-dimensional CT scan, the method further includes:

[0169] Grayscale conversion, normalization, noise removal and edge detection are performed on the multidimensional image.

[0170] The image processing performed on the multi-dimensional image in this embodiment is an important step in medical image analysis, which helps to improve the quality of the image and provide more accurate data for subsequent image analysis and reconstruction.

[0171] Grayscaling converts a color image into a grayscale image. Color images typically contain three color channels: RGB. Grayscaling combines this color information into a single channel, reflecting the image's brightness. Grayscaling simplifies the image data and reduces the complexity of subsequent processing, while preserving important structural information.

[0172] Normalization is the process of scaling the pixel values ​​of an image so that they are distributed within a uniform range, such as [0, 1]. This helps eliminate differences in brightness and contrast between different images, making subsequent image processing and analysis more consistent and effective.

[0173] Denoising is the process of removing noise from an image to improve image quality. Noise refers to random variations in the image that are not part of the original scene and can be caused by factors such as interference in the sensor or during transmission. Filters such as Gaussian filtering, median filtering, or non-local means filtering can effectively remove noise while preserving the important features of the image.

[0174] Edge detection is the process of identifying edges in an image, i.e., the outlines of objects or the boundaries between different areas. Edge detection is achieved by identifying pixels in an image with significant brightness changes. Edge detection algorithms include Canny edge detection and the Sobel operator.

[0175] This embodiment can grayscale, normalize, remove noise and perform edge detection on the multidimensional image. The grayscale and normalization processes simplify the data and reduce the computational burden of subsequent processing. The processed images can also better reflect the actual tissue structure and pathological changes. In addition, the visual quality of the image is also improved, which helps to analyze the image content more accurately.

[0176] refer to Figure 5 The present application also provides a thyroid laparoscopic surgery navigation system based on MR fusion AF technology, comprising:

[0177] The first acquisition module 11 is used to obtain a multi-dimensional image of the patient's neck tissue and organs obtained by four-dimensional CT scanning;

[0178] The second acquisition module 12 is used to obtain the patient's thyroid panorama image, thyroid lesion image, lymph node image and lymph node surrounding vein image obtained by ultrasound scanning;

[0179] A third acquisition module 13 is configured to acquire a full-view image of the patient's thyroid gland under a fluorescence laparoscope based on autofluorescence technology;

[0180] a determination module 14, configured to receive a real-time surgical video stream of the patient undergoing laparoscopic surgery, select a target surgical video frame from the real-time surgical video stream, and determine image defect features of the target surgical video frame;

[0181] a screening module 15 for screening at least one image from the multidimensional image, the panorama image of the thyroid gland, the image of the thyroid lesion, the lymph node image, the image of the venous vessels surrounding the lymph node, and the panorama image of the thyroid gland according to the image defect characteristics of the target surgical video frame, to obtain a target image;

[0182] An image reconstruction module 16 is configured to reconstruct the target image and the target surgical video frame to obtain a target surgical scene;

[0183] The projection module 17 is used to project the target surgical image onto the surgeon's smart glasses to assist the surgeon in performing the surgery through intelligent navigation.

[0184] The present application provides a thyroid laparoscopic surgical navigation system based on MR fusion AF technology. First, by obtaining a variety of image types under four-dimensional CT scans, ultrasound scans, and fluorescence laparoscopy, it can provide multi-dimensional images of neck tissue organs and detailed information about the thyroid gland and its surrounding tissues. Based on autofluorescence technology, the full-view development image of the thyroid gland under fluorescence laparoscopy can more clearly identify the thyroid tissue. Secondly, according to the image defect characteristics of the target surgical video frame, the corresponding target image is intelligently screened from a variety of image types, and the image is reconstructed with the target surgical video frame to generate a target surgical screen to comprehensively improve the reconstruction effect of the target surgical screen. At the same time, the fusion of the multimodal images helps the surgeon understand the patient's condition from different angles and different levels, improve the accuracy of diagnosis, and thus improve the accuracy and reliability of thyroid laparoscopic surgical navigation. Finally, the reconstructed target surgical screen is projected onto the surgeon's smart glasses, providing the surgeon with a mixed reality, multi-dimensional real-time surgical field of view. The surgeon can obtain clearer visual information during the operation, thereby performing the surgical operation more accurately, reducing damage to surrounding healthy tissues, and improving the safety and efficiency of the operation. In summary, this application provides an innovative auxiliary means for thyroid laparoscopic surgery by combining multimodal imaging, autofluorescence imaging, real-time video processing, image reconstruction and mixed reality technology, significantly improving the accuracy, safety and efficiency of the surgery.

[0185] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0186] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, a thyroid laparoscopic surgery navigation method based on MR fusion AF technology is implemented.

[0187] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0188] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0189] The above description is only a preferred embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A thyroid laparoscopic surgery navigation method based on MR fusion AF technology, characterized in that: include: Obtain multi-dimensional images of the patient's neck tissues and organs through four-dimensional CT scanning; Obtaining an overall image of the thyroid gland, an image of thyroid lesions, an image of lymph nodes, and an image of venous connections surrounding the lymph nodes of the patient obtained through ultrasound scanning; Based on the autofluorescence technology, a full-view image of the patient's thyroid gland is obtained under a fluorescence laparoscope; receiving a real-time surgical video stream of the patient undergoing laparoscopic surgery, selecting a target surgical video frame from the real-time surgical video stream, and determining image defect features of the target surgical video frame; According to the image defect characteristics of the target surgical video frame, at least one image is selected from the multidimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node perivascular image, and the overall thyroid development image to obtain a target image; After reconstructing the target image and the target surgical video frame, a target surgical scene is obtained; The target surgical image is projected onto the surgeon's smart glasses to assist the surgeon in performing the surgery through intelligent navigation.

2. The method according to claim 1, characterized in that According to the image defect characteristics of the target surgical video frame, at least one image is selected from the multidimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node perivascular image, and the overall thyroid development image to obtain a target image, including: Extracting image features related to the image defect features from the multidimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node perivascular image, and the overall thyroid development image, respectively, to obtain an image feature set, wherein the image feature set includes a plurality of image features; Screening out image features that can optimize the image defect features from the image feature set to obtain target image features; An image corresponding to the target image feature is determined to obtain a target image.

3. The method according to claim 1, characterized in that After reconstructing the target image and the target surgical video frame, a target surgical scene is obtained, including: Adjusting the size of the target image to be consistent with the size of the target surgical video frame to obtain a standard target image; Dividing the standard target image into a plurality of rectangular image blocks of the same size to obtain a plurality of first rectangular image blocks; Dividing the target surgical video frame into a plurality of rectangular image blocks of the same size to obtain a plurality of second rectangular image blocks, wherein the size of the first rectangular image blocks is the same as the size of the second rectangular image blocks; performing similarity calculations on each of the first rectangular image blocks and a plurality of the second rectangular image blocks to obtain similarity calculation results; According to the similarity calculation result, selecting a first target rectangular image block having a similarity greater than a preset similarity from the plurality of first rectangular image blocks; Marking the blocked image area of ​​the target surgical video frame, determining a matching first target rectangular image block based on the adjacent second rectangular image block of the blocked image area and the similarity calculation result, and obtaining a second target rectangular image block; The blocked image area of ​​the target surgical video frame is replaced by the adjacent first rectangular image block of the second target rectangular image block to obtain the target surgical picture.

4. The method according to claim 1, wherein After reconstructing the target image and the target surgical video frame, a target surgical scene is obtained, including: Setting a Gaussian kernel of the target image, wherein the Gaussian kernel is a matrix weighted based on a Gaussian distribution, and the matrix includes a plurality of Gaussian weights; Performing normalized weight processing on the Gaussian kernel of the target image to obtain a target Gaussian kernel, wherein the sum of all Gaussian weights in the target Gaussian kernel is equal to 1; For each pixel in the target image, multiply the pixel values ​​in the neighborhood of each pixel by the corresponding Gaussian weight in the target Gaussian kernel, and sum them up to obtain a Gaussian weight map corresponding to the target image; After weighted multiplication of the target image and the corresponding Gaussian weight map, a weighted map corresponding to the target image is obtained; Inputting the target surgical video frame into a pre-trained convolutional neural network to obtain a convolutional graph corresponding to the target surgical video frame; The target surgical scene is obtained by fusing the weighted graph corresponding to the target image with the convolution graph corresponding to the target surgical video frame.

5. The method according to claim 1, characterized in that After reconstructing the target image and the target surgical video frame to obtain the target surgical scene, the method further includes: Segmenting the real-time surgical video stream into multiple surgical video frames according to a timeline; respectively determining the surgical position of the thyroid lesion tissue in each of the surgical video frames; generating a surgical trajectory of the thyroid lesion tissue based on a time node of each surgical video frame and a surgical position of the thyroid lesion tissue; marking the boundary of the thyroid lesion tissue in the thyroid lesion image; predicting the real-time dynamic position of the thyroid lesion tissue to be removed based on the surgical trajectory and boundaries of the thyroid lesion tissue; The real-time dynamic position is marked in the target surgical screen.

6. The method according to claim 1, characterized in that After obtaining multi-dimensional images of the patient's neck tissues and organs through four-dimensional CT scanning, the following also applies: Acquiring a reference multi-dimensional image having a resolution greater than a preset resolution; Performing dimensionality reduction processing on the multidimensional image to obtain a first image; Performing dimensionality reduction processing on the reference multidimensional image to obtain a second image; extracting first image parameters of the first image and second image parameters of the second image respectively; Calculating a structural similarity index between the first image and the second image based on the first image parameter and the second image parameter; Determining whether the structural similarity index is greater than a preset structural similarity index; If yes, performing the step of selecting at least one image from the multi-dimensional image, the overall thyroid image, the thyroid lesion image, the lymph node image, the lymph node peripheral vein image, and the overall thyroid development image to obtain a target image; If not, the multi-dimensional image is subjected to multi-level filtering and histogram equalization processing, and then input into a convolutional neural network until the structural similarity index corresponding to the output image is greater than a preset structural similarity index.

7. The method according to claim 1, characterized in that After obtaining multi-dimensional images of the patient's neck tissues and organs through four-dimensional CT scanning, the following also applies: Grayscale conversion, normalization, noise removal and edge detection are performed on the multidimensional image.

8. The method according to claim 1, characterized in that The target surgical images include: a simulated surgical field of view reconstructed by four-dimensional CT, a simulated surgical field of view reconstructed by ultrasound images, a surgical field of view reconstructed by thyroid tissue under autofluorescence, and a real-time dynamic surgical field of view.

9. The method according to claim 1, characterized in that The overall thyroid development image marks the thyroid gland, parathyroid glands, fluorescence intensity, range, and boundaries.

10. A thyroid laparoscopic surgery navigation system based on MR fusion AF technology, characterized in that: include: The first acquisition module is used to obtain a multi-dimensional image of the patient's neck tissue and organs obtained through a four-dimensional CT scan; A second acquisition module is used to obtain a thyroid panorama image, a thyroid lesion image, a lymph node image, and an image of the venous system surrounding the lymph nodes of the patient obtained by ultrasound scanning; A third acquisition module is configured to acquire a full-view image of the patient's thyroid gland under a fluorescence laparoscope based on autofluorescence technology; a determination module, configured to receive a real-time surgical video stream of the patient undergoing laparoscopic surgery, select a target surgical video frame from the real-time surgical video stream, and determine image defect features of the target surgical video frame; a screening module, configured to screen at least one image from the multidimensional image, the panorama image of the thyroid gland, the image of the thyroid lesion, the lymph node image, the image of the venous vessels surrounding the lymph node, and the panorama image of the thyroid gland according to the image defect characteristics of the target surgical video frame, to obtain a target image; An image reconstruction module, configured to reconstruct the target image and the target surgical video frame to obtain a target surgical picture; The projection module is used to project the target surgical image onto the surgeon's smart glasses to assist the surgeon in performing the surgery through intelligent navigation.

Images