Multi-modal real-time fusion ai navigation system and method for endoscopic thyroid surgery

Abstract

This invention relates to the fields of medical surgical navigation and artificial intelligence, and discloses a multimodal real-time fusion AI navigation system and method for laparoscopic thyroid surgery. The system includes: a multimodal data acquisition module for simultaneously acquiring near-infrared fluorescence image data and visible light video data of the laparoscopic surgical area and outputting a dual-channel data stream; an AI intelligent processing module for receiving the dual-channel data stream and performing preprocessing; an image fusion and segmentation using a multimodal AI segmentation network based on an edge-guided fusion mechanism, outputting four types of key structure segmentation masks; a visualization fusion module for overlaying and fusing the key structure segmentation masks in a highlighted form into the visible light video data to generate an augmented reality surgical field of view; and a surgical guidance module for generating dynamic guidance markers in the augmented reality surgical field of view based on the key structure segmentation masks and a preset surgical planning path. This invention can accurately identify key anatomical structures such as the parathyroid gland and recurrent laryngeal nerve.

Description

Technical Field

[0001] This invention relates to the fields of medical surgical navigation and artificial intelligence technology, and in particular to a multimodal real-time fusion AI navigation system and method for endoscopic thyroid surgery. Background Technology

[0002] Endoscopic thyroid surgery has become an important surgical procedure for treating thyroid diseases due to its advantages such as minimal trauma, rapid postoperative recovery, and good cosmetic results. However, this surgery faces two major challenges: First, the identification of key anatomical structures is difficult. The parathyroid glands are small, the recurrent laryngeal nerve has many anatomical variations, and the surgical field is obstructed by factors such as similar tissue morphology. Traditional identification methods that rely on the surgeon's experience are prone to damage. Second, the efficiency of multimodal information fusion is low. Near-infrared fluorescence images can capture parathyroid-specific signals, and visible light images can provide clear tissue texture. However, current technologies lack effective fusion mechanisms, making it difficult to simultaneously utilize the complementary information of the two types of images. Furthermore, problems such as modal bias and blurred edges exist, resulting in insufficient navigation accuracy.

[0003] Existing AI navigation systems mostly employ single-modal or simple multimodal network structures, lacking a dedicated fusion architecture designed for the multimodal data characteristics of endoscopic thyroid surgery. This results in shortcomings such as insufficient feature extraction, low segmentation accuracy, and poor real-time performance. For example, some systems use ordinary convolutional neural networks for segmentation, which cannot effectively handle the modal differences between near-infrared and visible light; some fusion networks lack targeted attention mechanisms, making it difficult to focus on the edges and detailed features of key structures, leading to insufficient reliability of surgical guidance and the continued risk of postoperative complications.

[0004] Therefore, there is an urgent need for a surgical navigation system that combines efficient multimodal fusion capabilities, accurate structural recognition performance, and real-time response characteristics to address the aforementioned technical challenges. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the core objective of this invention is to provide a multimodal real-time fusion AI navigation system and method for endoscopic thyroid surgery. Specifically, it provides an integrated navigation system and method that integrates multimodal data acquisition, AI intelligent segmentation, and real-time visualization guidance, which can accurately identify key anatomical structures such as the parathyroid gland and recurrent laryngeal nerve, assisting surgeons in performing precise and safe minimally invasive surgical operations.

[0006] To achieve the above objectives, the following technical solution is adopted:

[0007] In a first aspect, embodiments of the present invention provide a multimodal real-time fusion AI navigation system for laparoscopic thyroid surgery, comprising a multimodal data acquisition module, an AI intelligent processing module, a visualization fusion module, and a surgical guidance module connected in sequence via communication. The multimodal data acquisition module is used to simultaneously acquire near-infrared fluorescence image data and visible light video data of the laparoscopic surgical area, and output a frame-level synchronized dual-channel data stream. The AI ​​intelligent processing module is used to receive the dual-channel data stream and perform preprocessing, including noise reduction, image enhancement, registration, and channel separation processing on the near-infrared fluorescence image data and visible light video data, to obtain preprocessed data. Near-infrared fluorescence images and visible light brightness channel images are used. These preprocessed near-infrared fluorescence images and visible light brightness channel images are fused and segmented using a multimodal AI segmentation network based on an edge-guided fusion mechanism, outputting segmentation masks for key structures such as the parathyroid gland, recurrent laryngeal nerve, superior thyroid vessels, and trachea. A visualization fusion module is used to overlay and fuse the key structure segmentation masks into the visible light video data in a highlighted form, generating an augmented reality surgical field of view. A surgical guidance module is used to generate dynamic guidance markers in the augmented reality surgical field of view based on the key structure segmentation masks and a preset surgical planning path.

[0008] Furthermore, the multimodal data acquisition module includes: a near-infrared fluorescence acquisition unit equipped with a high-sensitivity near-infrared fluorescence camera, with operating wavelengths set to 785nm excitation and 820nm emission, used to capture specific fluorescence signals of the parathyroid glands within the surgical area; a visible light video acquisition unit integrating a high-definition visible light endoscope camera, used to acquire visible light tissue texture images of the surgical area; and a data synchronization unit employing a hardware synchronization triggering mechanism to achieve frame-level synchronization of data acquired by the near-infrared fluorescence acquisition unit and the visible light video acquisition unit, with an inter-frame synchronization deviation ≤10ms.

[0009] Furthermore, the AI ​​intelligent processing module includes a data preprocessing unit and a key structure recognition and segmentation unit. The data preprocessing unit is used to perform noise reduction processing on the near-infrared fluorescence image data to enhance the fluorescence signal, perform image enhancement processing on the visible light video data to improve tissue contrast, perform spatial registration between the noise-reduced and enhanced near-infrared fluorescence image and the visible light image, and perform color space conversion on the registered visible light image to separate the brightness channel image. The key structure recognition and segmentation unit constructs an end-to-end multimodal AI segmentation network based on a residual fusion network-nested structure. It uses the preprocessed near-infrared fluorescence image and the visible light brightness channel image as dual-channel inputs. Through multiple edge-guided residual fusion networks embedded with an edge-guided fusion module, it adaptively fuses multimodal features and outputs a key structure segmentation mask for the parathyroid gland, recurrent laryngeal nerve, upper pole thyroid vessels, and trachea.

[0010] Furthermore, the residual fusion network-nested structure includes: an autoencoder comprising an encoder part and a decoder part for extracting multi-scale image features; multiple edge-guided residual fusion networks, each connected to a feature map of a different scale in the autoencoder; the edge-guided residual fusion network comprising residual blocks and embedding an edge-guided fusion module for generating pixel-level weight mappings based on edge maps extracted from near-infrared fluorescence images and visible light brightness channel images, to weight and fuse multi-modal feature maps of corresponding scales; and a segmentation head connected to the output of the decoder for receiving the fused features output by the decoder part and outputting segmentation masks for multiple key structural categories.

[0011] Furthermore, the following features are specified: the autoencoder comprises three convolutional layers, each followed by a max-pooling layer for downsampling with a downsampling factor of 2; the decoder comprises three convolutional layers, upsampling via transposed convolution to restore image resolution; four edge-guided residual fusion networks are connected to feature maps of four different scales in the autoencoder, including the original size, 1 / 2 downsampled size, 1 / 4 downsampled size, and 1 / 8 downsampled size; each edge-guided residual fusion network comprises two residual blocks, each including a convolutional layer, a batch normalization layer, and a ReLU activation function; the segmentation head comprises a first convolutional layer and a second convolutional layer connected in sequence; the first convolutional layer uses 1×1 convolution for feature dimensionality reduction, and the second convolutional layer outputs segmentation masks for four key structures: parathyroid gland, recurrent laryngeal nerve, superior thyroid vessels, and trachea, using a Sigmoid activation function.

[0012] Furthermore, the edge-guided fusion module generates pixel-level weight mapping by extracting edge maps from the input near-infrared fluorescence image and the visible light brightness channel image, respectively; for the k-th scale, based on the intensity difference between the near-infrared fluorescence edge map and the visible light edge map at that scale, a pixel-level weight map of the near-infrared fluorescence features is generated by calculating the Sigmoid function, wherein the input of the Sigmoid function is the result of scaling the intensity difference by a fixed scaling factor and summing it with a bias term.

[0013] Furthermore, the edge-guided fusion module performs feature fusion in the following manner: for the near-infrared fluorescence feature map and visible light feature map of the k-th scale and c-th channel, the pixel-level weight map is used for weighted fusion to obtain the fused feature map.

[0014] Furthermore, the multimodal AI segmentation network in the AI ​​intelligent processing module is a neural network model optimized by INT8 quantization and / or model pruning. The multimodal AI segmentation network is trained based on the near-infrared fluorescence-visible light paired dataset of endoscopic thyroid surgery. After training, the multimodal AI segmentation network is deployed on the graphics processor hardware to achieve the inference speed required for real-time surgical navigation.

[0015] Furthermore, the construction of the near-infrared fluorescence-visible light paired dataset for endoscopic thyroid surgery includes the following steps: collecting paired data covering different surgical approaches, genders, ages, and anatomical variations; pixel-level fine annotation by multiple senior surgeons, and obtaining annotation consistency with an intra-group correlation coefficient greater than 0.9; and applying at least one data augmentation operation among random cropping, flipping, brightness perturbation, noise addition, elastic deformation, rotation, and scaling transformations to the training dataset.

[0016] Secondly, embodiments of the present invention also provide a multimodal real-time fusion AI navigation method for endoscopic thyroid surgery, applied to the multimodal real-time fusion AI navigation system as described in the first aspect, comprising the following steps: S1, synchronously acquiring near-infrared fluorescence image data and visible light video data of the endoscopic surgical area to obtain a frame-level synchronized dual-channel data stream; S2, preprocessing the dual-channel data stream to obtain a preprocessed near-infrared fluorescence image and a visible light brightness channel image; inputting the preprocessed near-infrared fluorescence image and visible light brightness channel image into a multimodal AI segmentation network based on an edge-guided fusion mechanism for real-time processing to obtain a segmentation mask for key structures of the parathyroid gland, recurrent laryngeal nerve, superior thyroid vessels, and trachea; S3, superimposing and fusing the key structure segmentation mask in a highlighted form into the visible light video data to generate an augmented reality surgical field of view; S4, generating dynamic guidance markers in the augmented reality surgical field of view based on the key structure segmentation mask and a preset surgical planning path.

[0017] Compared with the prior art, the present invention achieves the following beneficial effects:

[0018] 1. Architectural Innovation Advantages: The RFN-Nest nested structure proposed in this invention forms a soft prior attention mechanism through four EG-RFN networks of different scales and an edge-guided fusion module (EGF). This mechanism can adaptively correct the multimodal feature weight allocation and improve the recognition accuracy of fine structures such as the parathyroid gland and recurrent laryngeal nerve.

[0019] 2. Advantages of multimodal fusion: The weighted mapping generation formula based on edge strength and the Hadamard product feature fusion algorithm are adopted to effectively reduce the modal difference between near-infrared and visible light, avoid recognition deviation caused by the lack of single modal information, and the edge clarity and signal fidelity of the fused image are better than those of the existing technology, with a structural edge recognition error of ≤2 pixels.

[0020] 3. Real-time performance and practicality advantages: Through inference optimization strategies such as INT8 quantization and model pruning, combined with high-performance embedded GPU hardware support, the system inference speed is ≥30fps, and the end-to-end processing latency is controlled within 150-200ms in clinical scenarios, fully meeting the real-time navigation requirements of laparoscopic surgery; at the same time, it supports multi-laparoscopic approach adaptation, and the highlighted annotation and dynamic guide arrow design are intuitive and easy to understand, reducing the learning cost for doctors;

[0021] 4. Clinical safety advantages: The model is trained based on data from 1,200 clinical surgeries and optimized with high-intensity data enhancement. It is robust and can be adapted to patients of different genders, ages and anatomical variations. It can assess the blood supply to the parathyroid glands and the integrity of the recurrent laryngeal nerve in real time. It can also provide timely warnings when surgical instruments approach critical structures, effectively reducing the risk of complications such as nerve damage and hypoparathyroidism, and improving surgical safety and prognosis.

[0022] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the invention. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0024] Figure 1 This is a schematic diagram of laparoscopic thyroid surgery provided in an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of a multimodal real-time fusion AI navigation system for endoscopic thyroid surgery provided in Embodiment 1 of the present invention;

[0026] Figure 3 This is a schematic diagram of the architecture of a multimodal real-time fusion AI navigation system for endoscopic thyroid surgery provided in Embodiment 1 of the present invention;

[0027] Figure 4 This is a schematic diagram of the AI ​​intelligent processing module fusion and segmentation process provided in Embodiment 1 of the present invention;

[0028] Figure 5 This is a schematic diagram of AI recognition of important anatomical structures during laparoscopic thyroid surgery according to Embodiment 1 of the present invention;

[0029] Figure 6 This is a flowchart illustrating the multimodal real-time fusion AI navigation method for endoscopic thyroid surgery provided in Embodiment 2 of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0032] Example 1:

[0033] To better understand the clinical scenarios in which the system of this invention is applied and the technical problems to be solved, we will first combine... Figure 1 The key steps of a typical endoscopic thyroid surgery are explained. Figure 1 This is a schematic diagram of an endoscopic thyroid surgery provided in an embodiment of the present invention, wherein: Figure 1 A in the diagram illustrates the patient's surgical position and common incision design locations (such as via the axilla, via the oral vestibule, or via the chest-breast approach), and the navigation system of this invention can be adapted to these different minimally invasive approaches. Figure 1 Figure B illustrates the process of creating a surgical operating cavity through an incision, which provides a field of view and operating space for the endoscope and the acquisition unit of this system. Figure 1 Image C illustrates one of the key challenges in surgery—the identification and protection of the recurrent laryngeal nerve. The recurrent laryngeal nerve is delicate, has a variable course, and is easily damaged. Figure 1 The "D" in the diagram illustrates another key challenge—the identification and protection of the superior parathyroid gland. Parathyroid glands are small, have fragile blood supply, and are difficult to locate. In traditional surgery, surgeons primarily rely on visual inspection and experience to identify them under a microscope. Figure 1 The recurrent laryngeal nerve and parathyroid glands, shown in C and D, present high risks and long learning curves. The multimodal real-time fusion AI navigation system provided by this invention aims to assist doctors in performing tasks accurately and intuitively under such complex laparoscopic conditions. Figure 1The identification and protection of nerves and parathyroid glands, as shown in C and D, can improve surgical safety and efficiency.

[0034] like Figure 2 and Figure 3 The figures show a module diagram and a system architecture diagram of the multimodal real-time fusion AI navigation system for endoscopic thyroid surgery provided in Embodiment 1 of the present invention. The multimodal real-time fusion AI navigation system 100 for endoscopic thyroid surgery in Embodiment 1 of the present invention is an integrated navigation system that integrates multimodal information acquisition, artificial intelligence (AI) intelligent processing, visualization presentation, and surgical guidance. Its core includes a multimodal data acquisition module 110, an AI intelligent processing module 120, a visualization fusion module 130, and a surgical guidance module 140, which are connected in sequence. These modules work together to achieve real-time identification, segmentation, and precise guidance of key anatomical structures. The overall architecture adopts a modular design to ensure that the synchronization deviation between multimodal data frames is ≤10ms; the end-to-end processing latency of ≤100ms is the laboratory optimization target under specific hardware configurations. In actual clinical applications, the typical latency range is 150-200ms, meeting the real-time requirements of endoscopic thyroid surgery. Wherein:

[0035] The multimodal data acquisition module 110 is used to simultaneously acquire near-infrared fluorescence image data and visible light video data of the laparoscopic surgical area, and output a frame-synchronized dual-channel data stream;

[0036] The multimodal data acquisition module 110 includes:

[0037] The near-infrared fluorescence acquisition unit 111 is equipped with a high-sensitivity near-infrared (NIR) fluorescence camera, with an operating wavelength set to 785nm excitation / 820nm emission. It is adapted to the autofluorescence characteristics of the parathyroid gland and can capture the specific fluorescence signal of the parathyroid gland in the surgical area in real time. The acquisition parameters are set as follows: frame rate ≥ 30fps, spatial resolution ≥ 1080P, and signal-to-noise ratio ≥ 50dB, ensuring clear capture of weak fluorescence signals and avoiding background noise interference.

[0038] The visible light video acquisition unit 112 integrates a high-definition visible light (VIS) endoscope camera, supporting 4K ultra-high-definition (3840×2160) video acquisition at a frame rate of 60fps. It features automatic white balance, exposure compensation, and gamma correction, realistically reproducing the morphology, color, and texture characteristics of thyroid glands, blood vessels, nerves, and other tissues within the surgical field of view. This unit seamlessly integrates with laparoscopic surgery systems, compatible with various laparoscopic approaches such as axillary, transoral vestibular, and thoracotomy approaches.

[0039] The data synchronization unit 113 employs a hardware synchronization triggering mechanism (synchronization pulse signal triggering) to achieve frame-level synchronous acquisition of near-infrared fluorescence data and visible light video data, with an inter-frame synchronization deviation of ≤10ms. It synchronously generates a dual-channel data stream with timestamps (H.265 encoded format), providing time-consistent basic data for subsequent fusion processing and ensuring the initial spatial alignment accuracy of multimodal information.

[0040] The AI ​​intelligent processing module 120 is used to receive the dual-channel data stream and perform preprocessing, including noise reduction, image enhancement, registration, and channel separation processing on the near-infrared fluorescence image data and visible light video data to obtain preprocessed near-infrared fluorescence images and visible light brightness channel images; the preprocessed near-infrared fluorescence images and visible light brightness channel images are fused and segmented using a multimodal AI segmentation network based on an edge-guided fusion mechanism, and the output segmentation mask for key structures such as the parathyroid gland, recurrent laryngeal nerve, upper pole thyroid vessels, and trachea.

[0041] Furthermore, the AI ​​intelligent processing module 120 includes a data preprocessing unit 121 and a key structure recognition and segmentation unit 122;

[0042] The noise reduction, registration, and channel separation processes performed by the data preprocessing unit 121 specifically include: the data preprocessing unit performs noise reduction processing on the near-infrared fluorescence image data to enhance the fluorescence signal, performs image enhancement processing on the visible light video data to improve tissue contrast, performs spatial registration between the noise-reduced and enhanced near-infrared fluorescence image and the visible light image, and performs color space conversion on the registered visible light image to separate the luminance channel image. The specific implementation process is as follows:

[0043] (1) Noise reduction enhancement

[0044] Near-infrared fluorescence data were processed using an adaptive threshold noise reduction algorithm (threshold dynamic adjustment formula: Background noise is removed; visible light video image quality is optimized and tissue edge sharpness is improved through contrast-limited adaptive histogram equalization (CLAHE).

[0045] in, The adaptive threshold denoising algorithm dynamically adjusts the threshold to distinguish between signals and background noise in near-infrared fluorescence images. : The average pixel value of a local region in a near-infrared fluorescence image; : Pixel standard deviation of a local region in a near-infrared fluorescence image.

[0046] (2) Image registration

[0047] Based on the physical parameters of the endoscope lens and preoperative calibration data, a rigid transformation algorithm is used to complete the initial spatial alignment of the dual-channel data, reducing registration error. Pixels; subsequent minor local misalignments (such as those caused by tissue deformation or lens micro-movement) are adaptively handled by the internal modules of the AI ​​model. The tip of the surgical instrument is only used as a visual annotation reference point and does not participate in the registration calculation.

[0048] (3) Channel separation

[0049] The visible light image is converted to the YUV color space (luminance, Y), blue-Y, and red-Y, and the luminance-Y, V color space is extracted for subsequent multimodal feature fusion. The chromaticity components (U and V channels) are retained for color restoration of the final visualization.

[0050] Extract the brightness channel (Y channel) for multimodal feature fusion, preserving... The channel is used for subsequent visualization and color restoration to avoid color distortion.

[0051] The key structure identification and segmentation unit 122 constructs an end-to-end multimodal AI segmentation network based on a residual fusion network-nested structure. It uses the preprocessed near-infrared fluorescence image and visible light brightness channel image as dual-channel inputs. Through multiple edge-guided residual fusion networks embedded with edge-guided fusion modules, it adaptively fuses multimodal features and outputs a segmentation mask for key structures such as the parathyroid gland, recurrent laryngeal nerve, upper pole thyroid vessels, and trachea.

[0052] like Figure 4 The diagram shown is a schematic representation of the fusion and segmentation process of the AI ​​intelligent processing module 120 provided in Embodiment 1 of the present invention. The construction of the key structure recognition and segmentation unit 122 includes the following steps:

[0053] 1. Model Input Definition

[0054] A multimodal AI segmentation network uses preprocessed near-infrared fluorescence images ( ) and visible light brightness channel image ( As a dual-channel input, it directly learns the mapping relationship between multimodal features and key structure segmentation masks, without the need for a separate image fusion step.

[0055] The pre-processed (noise reduction, registration, etc.) near-infrared fluorescence image is used as one of the dual-channel inputs of the multimodal AI segmentation network; The visible light brightness channel (Y channel) image after preprocessing (channel separation, noise reduction, etc.) is used as one of the dual-channel inputs of the multimodal AI segmentation network.

[0056] 2. Model Structure Design

[0057] An end-to-end multimodal AI segmentation network is constructed based on a Residual Fusion Network-Nest (RFN-Nest) structure. The Residual Fusion Network-Nest structure includes: an autoencoder, comprising encoder and decoder parts, used to extract multi-scale image features; multiple edge-guided residual fusion networks, each connected to feature maps of different scales in the autoencoder; each edge-guided residual fusion network contains residual blocks and embeds an edge-guided fusion module, used to generate pixel-level weight mappings based on edge maps extracted from near-infrared fluorescence images and visible light brightness channel images, to weight and fuse multimodal feature maps of corresponding scales; and a segmentation head, connected to the decoder output, used to receive the fused features output from the decoder and output segmentation masks for multiple key structural categories.

[0058] 2.1 Network Structure

[0059] Preferably, in one specific embodiment, the residual fusion network-nested structure (RFN-Nest) consists of one autoencoder, four edge-guided residual fusion networks (EG-RFN), and one segmentation head:

[0060] (1) Autoencoder: Contains 6 convolutional layers (Conv1-Conv6), with a kernel size of 3×3, stride of 1, and padding of 1. The activation function after the convolutional layer is the Rectified Linear Unit (ReLU), which introduces non-linear feature transformation to improve the model's expressive power. Among them, Conv1-Conv3 is the encoder (downsampling factor 2, achieved through max pooling). The feature map size compression ratio of the encoder (Conv1-Conv3) is achieved through max pooling to extract high-level semantic features. Max pooling is the downsampling method used in the encoder to retain key information in the feature map and reduce the amount of computation. Conv4-Conv6 is the decoder (upsampling is achieved through transposed convolution). The pre-trained weights are initialized based on transfer learning of public multimodal medical image datasets (such as the brain tumor segmentation dataset BraTS and the medical segmentation Decathlon dataset) to enhance the model's adaptability to medical image segmentation. Decoder (Conv4-Conv6): The module in the autoencoder responsible for upsampling the feature map. It restores the image size through transposed convolution and outputs the segmentation result. Transposed convolution: An upsampling technique used in the decoder to enlarge the feature map size and achieve pixel-level segmentation. The parameters of this autoencoder can be pre-trained and initialized using a large-scale medical image dataset to enable it to have basic medical image feature extraction capabilities. Subsequent fine-tuning is performed using the paired dataset of this invention.

[0061] Wherein, Conv1-Conv6: 6 convolutional layers in the autoencoder, used to extract feature information from multimodal images; kernel size 3×3: the size of the convolutional kernel in the convolutional layer, used to slide and extract local features of the image; stride 1: the stride of the convolutional kernel as it slides across the image, controlling the size change of the feature map; padding 1: the padding process performed on the image edges during the convolution operation to avoid loss of edge features; downsampling factor 2.

[0062] (2) Edge-Guided Fusion Network (EG-RFN): Four networks correspond to four scales (1×, 1 / 2×, 1 / 4×, 1 / 8×), each layer contains two residual blocks (ResBlock), each residual block contains the classic structure of "convolutional layer + batch normalization (BN) layer + ReLU activation", and embeds the Edge-Guided Fusion (EGF) module as a soft prior spatial attention mechanism rather than a hard rule. Subsequent convolutional layers can further modify and optimize the fused features.

[0063] Among them, scales 1×, 1 / 2×, 1 / 4×, and 1 / 8× are the feature map scales corresponding to the four EG-RFNs, which are the original size and downsampled sizes of 1 / 2, 1 / 4, and 1 / 8 times, respectively, to achieve multi-scale feature fusion; ResBlock: Residual blocks, two of which are contained in each EG-RFN, adopting a "convolutional layer + BN layer + ReLU activation" structure to enhance feature extraction and alleviate the gradient vanishing problem; BN (BatchNormalization): Batch normalization layer, used to normalize the output features of the convolutional layer and accelerate model training convergence; EGF (Edge-Guided Fusion): Edge-guided fusion module, embedded in EG-RFN as a soft prior spatial attention mechanism, adaptively allocating multimodal feature weights.

[0064] (3) Segmentation head: It contains two convolutional layers. The first layer uses 1×1 convolution for dimensionality reduction. The second layer outputs segmentation masks for four key structures (parathyroid gland, recurrent laryngeal nerve, upper pole thyroid blood vessel, and trachea). The activation function is the Sigmoid activation function. The loss is calculated for each category to optimize the segmentation accuracy.

[0065] The segmentation head is the module in the network responsible for outputting the segmentation results. It contains two convolutional layers to achieve feature dimensionality reduction and category prediction. The 1×1 convolution is the convolution method used in the first layer of the segmentation head to compress the number of feature channels and reduce computational complexity. The segmentation mask is the pixel-level classification result output by the segmentation head, which clarifies the spatial range of each of the four key structures: parathyroid gland, recurrent laryngeal nerve, upper pole thyroid vessels, and trachea. The sigmoid activation function is used in the output layer of the segmentation head to map the prediction results to the [0,1] interval, adapting to multi-label segmentation scenarios.

[0066] 2.2 Core Integration Mechanism

[0067] (1) Edge map extraction

[0068] Using the Sobel filter, respectively from and Extracting edge maps and The scale is downsampled to four scales using bilinear interpolation to ensure that the scale is adapted to the EG-RFN network.

[0069] Sobel filter: an operator used to extract image edge information, obtaining edge features from near-infrared fluorescence images and visible light Y-channel images respectively; The preprocessed near-infrared fluorescence image is used as one of the inputs for edge map extraction; The preprocessed visible light Y channel (luminance channel) image is used as one of the inputs for edge map extraction; From near-infrared fluorescence images The edge map extracted reflects the structural edge information in the NIR image; From visible light Y channel images The edge map extracted reflects the structural edge information in the VIS image.

[0070] (2) Robust weight mapping generation

[0071] The edge-guided fusion module generates pixel-level weight mappings as follows: For the k-th scale, the edge intensity difference between corresponding pixels in the near-infrared fluorescence edge map and the visible light edge map at that scale is calculated; the edge intensity difference is multiplied by a preset scaling factor α, and an adjustable bias term b is added to obtain an intermediate value; the intermediate value is input into the Sigmoid function to generate a pixel-level weight map for near-infrared fluorescence features. Specifically, the formula for adaptive pixel-level weight allocation using the Sigmoid function is as follows:

[0072]

[0073] This weighting mechanism uses soft priors. For modal dependencies of different key structures (e.g., parathyroid glands depend on NIR fluorescence signals, recurrent laryngeal nerves depend on VIS texture features), the network can correct the weight allocation results through subsequent convolutional layers, avoiding erroneous weighting caused by NIR pseudo-edges. The scaling factor α is an adjustable parameter (preferably 0.8), and the bias term b is also an adjustable parameter. During model training, α and b can be optimized; after model deployment, the bias term b supports runtime fine-tuning based on the actual surgical scenario to flexibly adjust the bias of the weight allocation. Simultaneously, the overall training process of the network enables subsequent convolutional layers to adaptively correct the initial weight map. The runtime is the real-time adjustment phase during model inference, which allows for fine-tuning of the bias term. Optimize weight allocation.

[0074] : Pixel-level weighted map of near-infrared fluorescence features at the k-th scale, used for adaptive allocation of feature contribution of NIR modes; The Sigmoid activation function is used to map the weight calculation results to... The interval is used to achieve normalization; : Scale index (values ​​1-4), corresponding to the four feature scales of the EG-RFN network (1×, 1 / 2×, 1 / 4×, 1 / 8×). : Scaling factor, preferably 0.8, used to adjust the degree of influence of edge strength difference on weight; : Near-infrared fluorescence edge map at the k-th scale; : Visible light Y-channel edge map at the k-th scale; : Bias term, with an optional value of 0.

[0075] (3) Feature fusion

[0076] Furthermore, the working mechanism of the edge-guided fusion module also includes: feature fusion is performed in the following way: for the near-infrared fluorescence feature map of the k-th scale and the c-th channel ( ) and visible light feature map ( ), using pixel-level weight maps Weighted fusion is performed, specifically through Hadamard product (element-by-element multiplication) and addition operations, to obtain the fused feature map. The formula is as follows:

[0077]

[0078] After residual block enhancement, the fused features are input into the decoder for semantic upscaling. Residual block enhancement is used to enhance the fused feature map... The input residual block is used for feature enhancement to improve feature representation capability; the decoder is used to receive the enhanced fused features, perform semantic upgrades through upsampling and other operations, and output pixel-level segmentation results.

[0079] : Near-infrared fluorescence feature map of the k-th scale and c-th channel, including the structural and signal features of the NIR mode; : Visible light feature map of the k-th scale and c-th channel, including the morphological and texture features of the VIS mode; : Channel index of the feature map, corresponding to the feature representation of different dimensions in the network; Hadamard product (element-wise product) is used to perform pixel-wise weighted operations on the feature map and the weight map; The fused feature map of the k-th scale and the c-th channel is obtained by weighted summation of NIR features and VIS features.

[0080] The working mechanism of this edge-guided fusion module can adaptively allocate modal weights based on edge information to achieve accurate fusion.

[0081] 2.3 Model Training and Optimization

[0082] 2.3.1 Dataset Construction

[0083] The construction of the near-infrared fluorescence-visible light paired dataset for endoscopic thyroid surgery includes the following steps: collecting paired data covering different surgical approaches, genders, ages, and anatomical variations; pixel-level fine annotation by multiple senior surgeons, achieving annotation consistency with an intra-group correlation coefficient greater than 0.9; and applying at least one data augmentation operation from random pruning, flipping, brightness perturbation, noise addition, elastic deformation, rotation, and scaling to the training dataset. Specifically:

[0084] (1) Data source: Near-infrared fluorescence-visible light paired data of 1200 cases of endoscopic thyroid surgery (including multiple approaches such as axillary and oral vestibule), covering different genders (420 males and 780 females), ages (18-65 years) and anatomical variations (such as parathyroid ectopic and recurrent laryngeal nerve branch variations); the dataset will be continuously expanded to more than 3000 cases in actual product iteration to further improve the robustness of the model.

[0085] (2) Data annotation: Three senior head and neck surgeons used semi-automated annotation tools to perform fine annotation. The consistency of annotation was verified by the intraclass correlation coefficient (ICC) to ensure that ICC>0.9, that is, to ensure the reliability of annotation. The annotation result is a pixel-level segmentation mask, which clearly defines the spatial range of each type of key structure.

[0086] (3) Data partitioning and augmentation: The data was divided into a training set (840 cases) and a test set (360 cases) in a 7:3 ratio. The training set adopted a high-intensity data augmentation strategy, including random cropping (512×512 patch, adapted to the fine segmentation requirements of 4K images), random flipping (horizontal / vertical), brightness perturbation (±10%), Gaussian noise addition (variance ≤0.01), elastic deformation, random rotation (±15°), and scaling transformation (0.8-1.2 times), which effectively alleviated the risk of overfitting caused by insufficient data.

[0087] Training set (840 cases): A dataset used for model training, divided into 1200 cases in a 7:3 ratio, covering various surgical approaches, genders, ages, and anatomical variations;

[0088] Test set (360 cases): A dataset divided in a 7:3 ratio to validate model performance; it is not used for model training.

[0089] 512×512patch: A data augmentation method with random cropping, resulting in cropped image patch sizes suitable for fine segmentation of 4K images;

[0090] Random flipping (horizontal / vertical): One of the data augmentation strategies, which randomly flips an image horizontally or vertically to increase the diversity of training data;

[0091] Brightness perturbation (±10%): One of the data augmentation strategies, which randomly adjusts the image brightness within a range of ±10% to improve the model's robustness to brightness changes;

[0092] Gaussian noise addition (variance ≤ 0.01): One of the data augmentation strategies, adding Gaussian noise with a variance of no more than 0.01 to the image to enhance the model's resistance to interference;

[0093] Elastic deformation: one of the data augmentation strategies, which involves elastically deforming images to simulate tissue deformation during surgery;

[0094] Random rotation (±15°): One of the data augmentation strategies, which involves randomly rotating the image within a range of ±15° to improve the model's adaptability to images at different angles;

[0095] Scaling transformation (0.8-1.2 times): One of the data augmentation strategies, which involves randomly scaling the image by 0.8-1.2 times to expand the scale range of the training data.

[0096] 2.3.2 Training Configuration and Inference Optimization

[0097] The multimodal AI segmentation network in the AI ​​intelligent processing module 120 is a neural network model optimized through INT8 quantization and / or model pruning. The multimodal AI segmentation network is trained based on a near-infrared fluorescence-visible light paired dataset from laparoscopic thyroid surgery. After training, the multimodal AI segmentation network is deployed on graphics processing unit (GPU) hardware to achieve inference speeds sufficient for real-time surgical navigation. The training of the network model specifically includes the following steps:

[0098] (1) Initialization: The autoencoder parameters were pre-trained using transfer learning on a public multimodal medical dataset, and the EG-RFN and segmentation head parameters were initialized using a He normal distribution.

[0099] Transfer learning pre-training is a method of training autoencoder parameters using publicly available multimodal medical datasets to improve model initialization performance and adaptability to medical images; He normal distribution initialization is an initialization method for EG-RFN and segmentation head parameters, which adapts to the characteristics of the ReLU activation function and alleviates the gradient vanishing problem.

[0100] (2) Training configuration: The Adam optimizer with weight decay (Adam with Weight Decay, AdamW) is used. The initial learning rate is 1e-4, the weight decay is 1e-5, the batch size is 4, and the total number of training rounds is 75. The learning rate decays to 1e-5 and 1e-6 in the 30th and 60th rounds respectively. An early stopping strategy is adopted (if the Dice coefficient of the validation set does not increase in 3 rounds, training is stopped).

[0101] The AdamW optimizer is a weight decay-based Adam optimizer used to optimize parameters during model training, balancing convergence speed and overfitting risk. Initial learning rate 1e-4: The learning rate at the initial stage of model training, controlling the step size for parameter updates. Weight decay 1e-5: The weight decay coefficient in the AdamW optimizer, used to suppress overfitting by penalizing large weight parameters. Batch size 4: The number of samples used for each model parameter update, balancing training efficiency and memory usage. Total training epochs 75: The number of iterations for complete model training, covering all training cycles in the dataset. Learning rate decay (epoch 30 to 1e-5, epoch 60 to 1e-6): A strategy to gradually reduce the learning rate during training, improving model convergence accuracy. Early stopping strategy: Training stops when the Dice coefficient for validation set splitting does not improve for three consecutive epochs to avoid overfitting. Validation set splitting Dice coefficient: An indicator of the model's splitting accuracy on the validation set, used as the basis for determining the early stopping strategy.

[0102] (3) Inference optimization: High-performance graphics processing units (GPUs) (such as NVIDIA Jetson AGX Orin) provide computing power support for model inference. INT8 quantization and model pruning are performed through Tensor Runtime (TensorRT). A tool library for model inference optimization is used to remove redundant parameters and ensure that the inference speed is ≥30fps. The inference input size is fixed at 512×512. The sliding window method is used to adapt to the full field of view segmentation of 4K surgical images to achieve the goal of end-to-end latency ≤100ms in the laboratory environment, which is the upper limit of the total latency from image acquisition to segmentation result output in the laboratory environment.

[0103] INT8 quantization is the model quantization method in TensorRT, which converts model parameters from floating-point to 8-bit integers, reducing computation and improving inference speed. Model pruning is an optimization method that removes redundant parameters from the model, simplifying the model structure to improve inference efficiency; an inference speed of ≥30fps is the minimum number of image frames that the model can process per second, ensuring the real-time requirements of surgery; the inference input size of 512×512 is a fixed input image size for model inference, adapting to the network structure design; the sliding window method is a method of processing 4K surgical images in blocks, adapting to the 512×512 inference input size, and achieving full field-of-view segmentation.

[0104] The visualization fusion module 130 is used to overlay and fuse the key structure segmentation mask into the visible light video data in a highlighted form to generate an augmented reality surgical field of view.

[0105] The visualization fusion module 130 employs specific color annotation technology to overlay the segmentation mask output by the AI ​​onto the visible light surgical field in a highlighted form (e.g., ...). Figure 5(As shown), highlight area transparency is supported. Adjustable and equipped with a one-button on / off function to avoid visual interference.

[0106] The AI ​​intelligent processing module 120 employs an edge-guided fusion mechanism to accurately analyze and fuse the input dual-channel image using a multimodal AI segmentation network, outputting a high-quality segmentation mask. For a visual demonstration of the system's recognition performance, please refer to [link to documentation / reference]. Figure 5 As shown. Figure 5 The right image shows the original visible light field of view during laparoscopic thyroid surgery; the left image shows the AI-recognized image after processing by the system of this invention, in which key anatomical structures have been clearly identified and highlighted (e.g., parathyroid glands, recurrent laryngeal nerve, etc.). This comparison fully demonstrates that the system of this invention can effectively fuse near-infrared fluorescence and visible light information in complex real surgical scenarios, achieving stable and accurate identification of small, vulnerable key structures, providing a reliable foundation for subsequent visualization fusion and surgical guidance.

[0107] Specific color annotation technology is a technique that uses exclusive color annotations to segment and mask different key structures, improving visual recognition; Highlighted area transparency (30%-70%): The adjustable transparency range of the highlighted area balances recognition and surgical field visibility; One-button on / off function: Quickly turn the highlighted area on / off to avoid visual interference.

[0108] The surgical guidance module 140 is used to generate dynamic guidance markers in the augmented reality surgical field of view based on the key structure segmentation mask and the preset surgical planning path.

[0109] The surgical guidance module 140 is based on AI segmentation results and preset surgical planning paths (such as the upper pole thyroid gland treatment and central lymph node dissection paths). These preset surgical planning paths are pre-defined operational routes for surgical steps such as upper pole thyroid gland treatment and central lymph node dissection. Dynamic white arrows (3 pixels wide, 80% transparency) guide the safe operation direction in the visual display. For example, in the upper pole thyroid gland treatment stage, the arrows guide the separation path away from the recurrent laryngeal nerve; in the central lymph node dissection stage, the protection range of the parathyroid glands is dynamically marked (marked with a green dashed box); in the vascular treatment stage, the optimal coagulation position is indicated (marked with a red dot), assisting the surgeon in avoiding damage to critical structures.

[0110] Among them, the dynamic white arrow is a visual surgical guidance symbol with a line width of 3 pixels and a transparency of 80%, indicating the direction of safe operation; the line width of the surgical guidance arrow is 3 pixels, ensuring that the arrow is clearly visible; the transparency of the surgical guidance arrow is 80%, balancing recognizability and visual transparency; the green dashed box is a visual annotation style for the parathyroid gland protection range; and the red dot is a visual annotation style for the optimal vascular closure position.

[0111] Example 2:

[0112] Figure 6 This is a flowchart illustrating the multimodal real-time fusion AI navigation method for endoscopic thyroid surgery provided in Embodiment 2 of the present invention. A multimodal real-time fusion AI navigation method for endoscopic thyroid surgery, applied to the multimodal real-time fusion AI navigation system of Embodiment 1, includes the following steps:

[0113] S1. Simultaneously acquire near-infrared fluorescence image data and visible light video data of the laparoscopic surgical area to obtain a frame-level synchronized dual-channel data stream;

[0114] S2. The dual-channel data stream is preprocessed to obtain a preprocessed near-infrared fluorescence image and a visible light brightness channel image; the preprocessed near-infrared fluorescence image and visible light brightness channel image are input into a multimodal AI segmentation network based on an edge-guided fusion mechanism for real-time processing to obtain a segmentation mask for key structures such as the parathyroid gland, recurrent laryngeal nerve, upper pole thyroid vessels, and trachea.

[0115] S3. The key structure segmentation mask is superimposed and fused into the visible light video data in a highlighted form to generate an augmented reality surgical field of view;

[0116] S4. Based on the key structure segmentation mask and the preset surgical planning path, generate dynamic guidance markers in the augmented reality surgical field of view.

[0117] The multimodal real-time fusion AI navigation method for endoscopic thyroid surgery provided in Embodiment 2 of the present invention can be applied to the multimodal real-time fusion AI navigation system for endoscopic thyroid surgery provided in Embodiment 1 of the present invention. It has the corresponding functions and beneficial effects of the multimodal real-time fusion AI navigation system for endoscopic thyroid surgery. For detailed process, please refer to the relevant operation of the multimodal real-time fusion AI navigation system for endoscopic thyroid surgery in Embodiment 1 above.

[0118] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0119] It should also be noted that, in the embodiments of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in the embodiments of this application may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown in this application, but is to be accorded the widest scope consistent with the principles and novel features disclosed in the embodiments of this application.

Claims

1. A multimodal real-time fusion AI navigation system for endoscopic thyroid surgery, characterized in that, It includes a multimodal data acquisition module, an AI intelligent processing module, a visualization fusion module, and a surgical guidance module, which are connected in sequence. The multimodal data acquisition module is used to simultaneously acquire near-infrared fluorescence image data and visible light video data of the laparoscopic surgical area, and output a frame-synchronized dual-channel data stream; The AI ​​intelligent processing module receives the dual-channel data stream and performs preprocessing. It performs noise reduction, image enhancement, registration, and channel separation on the near-infrared fluorescence image data and visible light video data to obtain preprocessed near-infrared fluorescence images and visible light brightness channel images. The preprocessed near-infrared fluorescence images and visible light brightness channel images are fused and segmented using a multimodal AI segmentation network based on an edge-guided fusion mechanism, and the output segmentation mask for key structures such as the parathyroid gland, recurrent laryngeal nerve, upper pole thyroid vessels, and trachea is then used. The visualization fusion module is used to overlay and fuse the key structure segmentation mask into the visible light video data in a highlighted form to generate an augmented reality surgical field of view. The surgical guidance module is used to generate dynamic guidance markers in the augmented reality surgical field of view based on the key structure segmentation mask and the preset surgical planning path.

2. The system according to claim 1, characterized in that, The multimodal data acquisition module includes: The near-infrared fluorescence acquisition unit is equipped with a high-sensitivity near-infrared fluorescence camera, with the working wavelength set to 785nm excitation and 820nm emission, to capture the specific fluorescence signal of the parathyroid gland in the surgical area; The visible light video acquisition unit integrates a high-definition visible light endoscope camera for acquiring visible light tissue texture images of the surgical area; The data synchronization unit adopts a hardware synchronization triggering mechanism to achieve frame-level synchronization of the data acquired by the near-infrared fluorescence acquisition unit and the visible light video acquisition unit, with an inter-frame synchronization deviation of ≤10ms.

3. The system according to claim 1, characterized in that, The AI ​​intelligent processing module includes a data preprocessing unit and a key structure identification and segmentation unit. The data preprocessing unit is used to perform noise reduction processing on the near-infrared fluorescence image data to enhance the fluorescence signal, perform image enhancement processing on the visible light video data to improve tissue contrast, perform spatial registration between the noise-reduced and enhanced near-infrared fluorescence image and the visible light image, and perform color space conversion on the registered visible light image to separate the brightness channel image. The key structure identification and segmentation unit constructs an end-to-end multimodal AI segmentation network based on a residual fusion network-nested structure. It uses the preprocessed near-infrared fluorescence image and visible light brightness channel image as dual-channel inputs. Through multiple edge-guided residual fusion networks embedded with edge-guided fusion modules, it adaptively fuses multimodal features and outputs a segmentation mask for key structures such as the parathyroid gland, recurrent laryngeal nerve, upper pole thyroid vessels, and trachea.

4. The system according to claim 3, characterized in that, The residual fusion network-nested structure includes: An autoencoder, consisting of an encoder and a decoder, is used to extract multi-scale image features; Multiple edge-guided residual fusion networks are connected to feature maps of different scales in the autoencoder. Each edge-guided residual fusion network contains residual blocks and embeds an edge-guided fusion module, which generates pixel-level weight mappings based on edge maps extracted from near-infrared fluorescence images and visible light brightness channel images, so as to weight and fuse multimodal feature maps of corresponding scales. A segmentation head, connected to the output of the decoder, is used to receive the fused features from a portion of the decoder's output and output segmentation masks for multiple key structural categories.

5. The system according to claim 4, characterized in that, in: The autoencoder comprises three convolutional layers, each followed by a max pooling layer for downsampling, with a downsampling factor of 2; the decoder comprises three convolutional layers, which perform upsampling through transposed convolution to restore image resolution. The edge-guided residual fusion network consists of four networks, which are connected to feature maps of four different scales in the autoencoder. The four scales include the original size, 1 / 2 downsampled size, 1 / 4 downsampled size, and 1 / 8 downsampled size. Each edge-guided residual fusion network contains two residual blocks, and each residual block includes a convolutional layer, a batch normalization layer, and a ReLU activation function in sequence. The segmentation head includes a first convolutional layer and a second convolutional layer connected in sequence. The first convolutional layer uses 1×1 convolution for feature dimensionality reduction, and the second convolutional layer is used to output segmentation masks for four key structures: parathyroid gland, recurrent laryngeal nerve, superior thyroid vessels, and trachea, and uses the Sigmoid activation function.

6. The system according to claim 4 or 5, characterized in that, in, The edge-guided fusion module generates pixel-level weight mappings in the following manner: Edge maps are extracted from the input near-infrared fluorescence image and the visible light brightness channel image, respectively; For the k-th scale, calculate the edge intensity difference between the corresponding pixels in the near-infrared fluorescence edge map and the visible light edge map at that scale; multiply the edge intensity difference by a preset scaling factor α, and add an adjustable bias term b to obtain an intermediate value; input the intermediate value into the Sigmoid function to generate a pixel-level weight map for near-infrared fluorescence features.

7. The system according to claim 6, characterized in that, The edge-guided fusion module performs feature fusion in the following manner: For the near-infrared fluorescence feature map and visible light feature map of the k-th scale and c-th channel, the pixel-level weight map is used to perform weighted fusion to obtain the fused feature map.

8. The system according to claim 1, characterized in that, The multimodal AI segmentation network in the AI ​​intelligent processing module is a neural network model optimized by INT8 quantization and / or model pruning. The multimodal AI segmentation network is trained based on the near-infrared fluorescence-visible light paired dataset of endoscopic thyroid surgery. After training, the multimodal AI segmentation network is deployed on the graphics processor hardware to achieve the inference speed required for real-time surgical navigation.

9. The system according to claim 8, characterized in that, The construction of the near-infrared fluorescence-visible light paired dataset for laparoscopic thyroid surgery includes the following steps: Collect paired data covering different surgical approaches, gender, age, and anatomical variations; The annotations were performed by several senior surgeons at the pixel level, and the annotation consistency was obtained with an intragroup correlation coefficient greater than 0.

9. Apply at least one of the following data augmentation operations to the training dataset: random cropping, flipping, brightness perturbation, noise addition, elastic deformation, rotation, and scaling transformation.

10. A multimodal real-time fusion AI navigation method for endoscopic thyroid surgery, applied to the multimodal real-time fusion AI navigation system as described in any one of claims 1-9, characterized in that, Includes the following steps: S1. Simultaneously acquire near-infrared fluorescence image data and visible light video data of the laparoscopic surgical area to obtain a frame-level synchronized dual-channel data stream; S2. The dual-channel data stream is preprocessed to obtain a preprocessed near-infrared fluorescence image and a visible light brightness channel image; the preprocessed near-infrared fluorescence image and visible light brightness channel image are input into a multimodal AI segmentation network based on an edge-guided fusion mechanism for real-time processing to obtain a segmentation mask for key structures such as the parathyroid gland, recurrent laryngeal nerve, upper pole thyroid vessels, and trachea. S3. The key structure segmentation mask is superimposed and fused into the visible light video data in a highlighted form to generate an augmented reality surgical field of view; S4. Based on the key structure segmentation mask and the preset surgical planning path, generate dynamic guidance markers in the augmented reality surgical field of view.

Images