An ultrasound endoscopic navigation system and method based on deep learning

Abstract

The present invention discloses an ultrasonic endoscopic navigation system and method based on deep learning, which relates to the field of medical image analysis. The system includes: a parallel encoder module for extracting local features of ultrasonic endoscopic images through CNN branches and capturing global context information through Transformer branches; a channel attention fusion module for adaptively weighted fusion of the extracted features; a decoder module for generating anatomical structure segmentation masks based on fused features; a timing processing module for receiving the frame-by-frame segmentation results output by the decoder module and realizing temporal continuity analysis of ultrasonic endoscopic videos through a bidirectional LSTM network; and a multimodal fusion module for aligning and fusing ultrasonic endoscopic images. This application scheme can provide a highly reliable artificial intelligence auxiliary tool for early screening of pancreatic cancer, while reducing the learning threshold and clinical application cost of ultrasonic endoscopic technology, and promoting its popularization in primary medical institutions.

Description

Technical Field

[0001] The present invention relates to the field of medical image analysis, and in particular to an ultrasonic endoscopic navigation system and method based on deep learning. Background Art

[0002] Pancreatic cancer, due to its insidious onset and highly invasive nature, often progresses to the late stage by the time it is diagnosed, with a 5-year survival rate of less than 10%. Early diagnosis is key to improving prognosis, but existing screening methods such as tumor markers and imaging tests have low sensitivity for early lesions. Ultrasound endoscopy, with its high resolution and close-range exploration advantages, can detect abnormal pancreatic echoes ≤5mm in diameter, making it a core technology for early pancreatic cancer screening. However, the interpretation of ultrasound endoscopy images is highly dependent on the operator's experience and spatial imagination. Its scanning sections are varied and the anatomical structure is complex, resulting in a long learning curve and insufficient penetration in primary hospitals. According to statistics, only about 10% of endoscopists worldwide can skillfully operate ultrasound endoscopy, which greatly limits its clinical application.

[0003] At present, although medical image analysis based on deep learning has made progress in the fields of gastroscopy and colonoscopy, it has significant limitations in the analysis of ultrasound endoscopy images: traditional CNN relies on local convolution operations, which makes it difficult to model the long-distance spatial dependency between organs and blood vessels in ultrasound endoscopy images, resulting in limited accuracy in anatomical structure recognition; dynamic ultrasound endoscopy videos require temporal correlation analysis between consecutive frames, while existing single-frame segmentation models lack the coherence modeling of the overall structure of the video, and are prone to misjudgment due to image jitter or sudden changes in perspective; tumor compression or invasion often leads to deformation of anatomical structures, and models that rely solely on a single ultrasound endoscopy modality are prone to performance degradation due to data distribution offset, and need to integrate multimodal information such as CT / MRI to improve robustness. Therefore, based on the above difficulties, the present invention proposes an ultrasound endoscopy navigation system and method based on deep learning. Summary of the Invention

[0004] Technical Purpose

[0005] In order to solve the above problems, the purpose of the present invention is to provide an ultrasound endoscopic navigation system and method based on deep learning, aiming to solve the key bottlenecks in ultrasound endoscopic image analysis, assist endoscopists in quickly identifying key anatomical structures and lesions in ultrasound endoscopic images, lower the operational learning threshold, improve the early diagnosis efficiency of pancreatic cancer and other biliary and pancreatic system tumors, and promote the popularization and application of ultrasound endoscopic technology in primary medical institutions.

[0006] Technical Solution

[0007] In order to achieve the above objectives, the present invention provides an ultrasound endoscopy navigation system and method based on deep learning. The system adopts a parallel encoder architecture, extracts local detail features through CNN branches, and uses the self-attention mechanism of the Transformer branch to capture the global context, and combines the channel attention module to realize feature adaptive fusion; introduces a bidirectional LSTM network and a dynamic weight adjustment mechanism to analyze the spatiotemporal correlation between consecutive frames and optimize the temporal coherence of video segmentation; integrates the MaskScoring R-CNN and elastic registration algorithm to realize cross-modal alignment and feature complementarity of ultrasound endoscopy and CT / MRI images, and uses adversarial training to generate synthetic data to enhance the generalization ability of the model; designs a lightweight decoder and a hybrid loss function, retains detail information through multi-level jump connections, compresses the model parameters, and ensures that the single-frame processing delay is less than 50ms to meet clinical real-time requirements.

[0008] In a first aspect, the present invention provides an ultrasound endoscopic navigation system based on deep learning, comprising:

[0009] A parallel encoder module for extracting local features of the endoscopic ultrasound image through a CNN branch and capturing multi-scale global contextual information through a pyramid-structured Transformer branch; the Transformer branch includes a non-overlapping patch embedding layer and multi-stage downsampling;

[0010] Channel attention fusion module, used to perform adaptive weighted fusion of extracted features;

[0011] The decoder module generates anatomical structure segmentation masks based on the fused features through multi-level skip connections;

[0012] A timing processing module is used to receive the frame-by-frame segmentation results output by the decoder module and implement temporal coherence analysis of the ultrasound endoscopy video through a bidirectional LSTM network combined with an Attention mechanism;

[0013] A multimodal fusion module for registering and fusing endoscopic ultrasound images; the multimodal fusion module includes an adversarial elastic registration submodule that uses a generator to predict a nonlinear deformation field and a discriminator to distinguish between real and synthetic registered images, while incorporating mutual information maximization constraints to ensure anatomical consistency;

[0014] The spatiotemporal attention fusion module is used to dynamically allocate spatial and temporal attention weights according to the segmentation confidence of the current frame.

[0015] Furthermore, the feature map resolution output by the Transformer branch in the 8x downsampling stage is aligned with that of the CNN branch, and the spatial dimension matching of the feature map is achieved through bilinear interpolation.

[0016] Furthermore, the upsampling operation of the decoder module uses a transposed convolutional layer to perform 2x, 2x and 4x upsampling in sequence, and the loss function is jointly optimized using Dice Loss and cross entropy loss.

[0017] Furthermore, the multimodal fusion module also includes the following submodules:

[0018] Mask Scoring R-CNN module for segmenting soft tissue objects from CT / MRI images;

[0019] Elastic transformation module, which realizes cross-modality registration between endoscopic ultrasound and CT / MRI based on B-spline interpolation and mutual information optimization algorithm;

[0020] The multimodal feature enhancement module is used to input the registered fusion image into the parallel encoder module to improve the recognition accuracy of anatomical structure variation cases.

[0021] Furthermore, the spatiotemporal attention fusion module uses the self-attention layer of the Transformer branch to capture the global spatial correlation of organs and blood vessels in a single frame image, extracts the temporal evolution features between adjacent frames through a bidirectional LSTM network, and dynamically allocates spatial and temporal attention weights according to the segmentation confidence of the current frame. The fusion formula is:

[0022]

[0023] Where, It is dynamic fusion output; is the dynamic weight parameter; is the global spatial attention map; is the temporal attention weight; is the slope coefficient, determined by fitting the confidence decay curve of the tumor compression case; is the segmentation confidence of the current frame; is the confidence threshold.

[0024] By jointly modeling the spatial features and temporal dependencies within ultrasound endoscopic video frames, local details and global contextual information are effectively integrated, significantly improving the accuracy of anatomical structure segmentation and the coherence of video parsing. At the same time, it suppresses misidentification caused by image jitter or sudden changes in perspective, and enhances the model's adaptability to dynamic scenes.

[0025] Furthermore, the total loss function of the adversarial elastic registration module is:

[0026] Where, is the output of the total loss function; 、 、 and is the balancing factor for various losses; To combat losses; is the mutual information loss; and is the balance coefficient of the total variation and the L2 regularization term; is the total variation; is the L2 regularization term; is the segmentation confidence of the current frame; is the confidence threshold; is the KL divergence regularization term.

[0027] Through nonlinear deformation field prediction and adversarial training optimization, we break through the limitations of traditional rigid registration and achieve high-precision cross-modal alignment of ultrasound endoscopes and CT / MRI images, significantly improve the registration robustness of cases with complex anatomical variations, and enhance the model's generalization ability under data distribution from different medical centers.

[0028] In a second aspect, the present invention provides an anatomical structure annotation method based on ultrasound endoscopy video that does not involve diagnosis and treatment of diseases, comprising:

[0029] Filter invalid frames in ultrasound endoscopy videos based on the pre-trained CNN model and standardize valid frames;

[0030] Perform anatomical structure segmentation on a single frame image using the Transformer-CNN network;

[0031] Combined with a bidirectional LSTM network, the temporal consistency of the continuous frame segmentation results is corrected.

[0032] Furthermore, the standardization process includes grayscale normalization, CLAHE contrast enhancement, fixed-size cropping and metadata anonymization. The metadata anonymization specifically involves cropping the region of interest or replacing the patient privacy information in the image with a black mask.

[0033] In a third aspect, the present invention further provides a deep learning-based ultrasound endoscopic navigation method that does not involve diagnosis and treatment of diseases. The method is based on the system described in the first aspect, comprising:

[0034] Extract local features of ultrasound endoscopic images and capture multi-scale global context information;

[0035] Adaptively weight the extracted features and generate anatomical structure segmentation masks based on the fused features;

[0036] The output frame-by-frame segmentation results are received and the temporal coherence analysis of the ultrasound endoscopy video is realized through a bidirectional LSTM network;

[0037] The endoscopic ultrasound images were registered and fused.

[0038] In a fourth aspect, the present invention further provides a training device for a multimodal ultrasound endoscopic navigation model, comprising:

[0039] Adversarial training module, used to generate synthetic training data for endoscopic ultrasound and CT / MRI through generative adversarial networks;

[0040] A transfer learning module that uses the pre-trained IDEA gastroscopy navigation model parameters to initialize the weights of the Transformer-CNN network;

[0041] A hybrid loss function is used to jointly optimize Dice Loss, cross entropy loss, and time series consistency loss.

[0042] In a fifth aspect, the present invention also provides a computer device, comprising a management platform and a memory, wherein the management platform is connected to the memory, the memory is used to store computer programs, and the management platform is used to execute the computer programs stored in the memory, so that the computer device executes the aforementioned deep learning-based ultrasound endoscopic navigation method.

[0043] In a sixth aspect, the present invention also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a management platform, the aforementioned deep learning-based ultrasound endoscopic navigation method is implemented.

[0044] This invention achieves feature fusion of local details and global context through a parallel encoder architecture, combines a dynamic spatiotemporal attention mechanism to enhance video temporal coherence, and introduces a multimodal adversarial elastic registration network to solve the problem of nonlinear alignment of cross-modal images, supplemented by a lightweight decoder design to compress computational complexity. This system effectively overcomes the bottlenecks of traditional endoscopic ultrasound image analysis, such as local feature fragmentation, insufficient temporal dependence, sensitivity to anatomical variations, and limited real-time performance. It significantly improves the accuracy of anatomical structure segmentation, the robustness of cross-modal registration, and the adaptability to dynamic scenarios. It provides a highly reliable artificial intelligence-assisted tool for early screening of pancreatic cancer, while reducing the learning threshold and clinical application cost of endoscopic ultrasound technology, and promoting its popularization in primary medical institutions.

[0045] Beneficial effects

[0046] By implementing the deep learning-based ultrasound endoscopic navigation system and method provided by the present invention, the following technical effects are achieved:

[0047] (1) This application uses the collaborative design of CNN and Transformer branches to take into account both local feature extraction and global context modeling, overcome the feature expression limitations of traditional single-branch models in medical image segmentation due to inductive bias, and achieve efficient recognition and precise positioning of multi-scale anatomical structures in ultrasound endoscopic images.

[0048] (2) Based on the design of multi-level skip connections and hybrid loss functions, the model parameters are compressed while retaining high-resolution detail features, significantly reducing the computational complexity, ensuring that the single-frame processing delay meets the clinical real-time requirements, and providing endoscopists with seamless navigation assistance.

[0049] (3) By jointly modeling the spatial features and temporal dependencies within ultrasound endoscopic video frames, local details and global context information are effectively integrated, significantly improving the accuracy of anatomical structure segmentation and the coherence of video parsing. At the same time, it suppresses misidentification caused by image jitter or sudden changes in perspective, and enhances the model's adaptability to dynamic scenes.

[0050] (4) Through nonlinear deformation field prediction and adversarial training optimization, we break through the limitations of traditional rigid registration and achieve high-precision cross-modal alignment of ultrasound endoscopy and CT / MRI images, significantly improve the registration robustness of cases with complex anatomical variations, and enhance the generalization ability of the model under data distribution in different medical centers. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to make the above-mentioned deep learning-based ultrasonic endoscopic navigation system and method of the present invention more obvious and easy to understand, the following is a brief introduction to the drawings required for use in the specific implementation of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 Indicates the technical roadmap of this application;

[0053] Figure 2 Figure 2 shows the schematic diagram of the IDEA-EUS model. DETAILED DESCRIPTION

[0054] To facilitate understanding of the embodiments of the present invention, abbreviations and key terms that may be involved in the embodiments of the present invention are first explained or defined. Undefined abbreviations or key terms are generally understood by those skilled in the art.

[0055] EUS: endoscopic ultrasound;

[0056] IDEA-EUS: Deep learning-based endoscopic ultrasound navigation system model;

[0057] MRI: magnetic resonance imaging;

[0058] MR: magnetic resonance imaging.

[0059] Example 1:

[0060] Provides an ultrasound endoscopic navigation system and method based on deep learning, the technical route is as follows Figure 1 The details are as follows.

[0061] An ultrasound endoscopic navigation image recognition model was constructed based on the Transformer-CNN network. The construction process specifically includes:

[0062] 1. Data collection: The research subjects include clinical data of patients who underwent endoscopic ultrasound examination, including endoscopic ultrasound images and video data. Pathological data were collected for cases diagnosed with tumors.

[0063] 2. Data preprocessing, including data anonymization, data filtering, and data normalization and standardization. The data preprocessing process includes:

[0064] Step 1: Anonymize the image using one of the following two methods:

[0065] ① Remove metadata overlay by cropping the endoscopic ultrasound region of interest;

[0066] ② By editing specific image subregions, potentially confounding image features, patient names, medical record numbers, procedure timestamps, or procedure locations are replaced with black blocks. For example, if images are annotated with unique markers or text to highlight diagnostic features that could potentially bias the model, these images are considered to have confounding image features, and if these features cannot be removed without harming the image, they will not be included in the final database.

[0067] Step 2: Data filtering, including:

[0068] ①Use ordinary white light endoscopy mode images and ultrasound endoscopic ultrasound mode images for CNN1 training and testing to distinguish the ultrasound mode of ultrasound endoscopic endoscopy;

[0069] ② CNN2 was trained and tested using invalid and clear endoscopic ultrasound images, and invalid images were removed from further data collection. Consecutive frames extracted from endoscopic ultrasound videos were filtered using the aforementioned model and then labeled, with images from the same patient grouped in the same dataset.

[0070] Step 3: Data normalization and standardization to improve image contrast and detail for clearer images. Linear normalization is used to map the image's grayscale values ​​to a range of 0 to 255. For endoscopic ultrasound video extraction, images are acquired using an interval sampling method, with one frame every five. The images are then cropped using a fixed pixel width and height (224 × 224) to obtain endoscopic ultrasound images suitable for model training.

[0071] 3. Data annotation: The preprocessed ultrasound endoscopy and CT / MRI images are used for data annotation, so that the model can accurately identify and segment different anatomical structures and lesions in the images.

[0072] Labelme software was used to annotate the abdominal organs, major vascular structures, and space-occupying lesions identified during the endoscopic ultrasound path. The labels were categorized into 23 categories: liver, gallbladder, pancreatic head, pancreatic neck, pancreatic body, pancreatic tail, spleen, kidney, adrenal gland, main pancreatic duct, common bile duct, abdominal aorta, celiac trunk, common hepatic artery, splenic artery, superior mesenteric artery, inferior vena cava, portal vein, splenic vein, superior mesenteric vein, solid tumor, cystic lesion, and enlarged lymph node. The same image could have multiple labels.

[0073] 4. Image recognition model construction. During the image recognition deep learning model construction phase, the model is trained using labeled and segmented endoscopic ultrasound image data to automatically identify anatomical structures and lesions in endoscopic ultrasound images. A Transformer-CNN network is used to build an efficient and accurate image recognition model. By training and optimizing on a large amount of labeled data, the model learns the characteristic representations of different structures and lesions in the image and is able to accurately identify and segment unlabeled images.

[0074] ① Model construction: Nvidia A100 was used to build a Transformer-CNN network under the PyTorch deep learning framework to automatically identify anatomical structures in ultrasound endoscopy images.

[0075] The Transformer-CNN network is a deep learning model for medical image segmentation that effectively leverages multi-scale information to improve segmentation accuracy. This model employs a parallel encoder structure, with a CNN extracting local information and a separate encoder using a Transformer to extract global information. The information extracted by the two encoders is then fused, and a decoder is used to generate the final segmentation result.

[0076] The specific implementation process is as follows: The Transformer-CNN network is a new parallel encoder architecture proposed after considering the unique advantages of CNN and Transformer. Traditional segmentation models rely solely on a single encoder branch to integrate global and local information in the image, while this method combines two independent branches to capture semantic information from the input image respectively. The overall architecture follows a paradigm similar to the U-Net encoder-decoder, consisting of an encoder, a channel attention module, a skip connection, and a decoder. This parallel encoder combines the advantages of ResNet and Transformer. A pyramid structure is introduced in the Transformer component to capture global features at multiple resolutions. In addition, the channel attention module is used to enhance the expressive power of the parallel encoder, enrich the extracted features, and provide guidance for the subsequent decoding process. Skip connections and decoder modules are also used to estimate the final segmentation mask. It mainly consists of three parts:

[0077] The parallel Transformer-CNN encoder introduces a pyramid structure within the Transformer to obtain global feature maps at different scales. To extract more comprehensive and diverse feature representations, the parallel encoder is divided into three stages. In the first stage, a patch embedding layer with a patch size of 4 is applied to the 2D input image of size H × w × 3, ensuring that there are no overlapping patches. The resulting feature map is then processed through the Transformer layers to obtain its global information. In the second stage, to preserve the detailed information in the patch embedding layer, the patch size is reduced to 2. This produces a feature map with a resolution of H8 × w8, which is then further processed through the Transformer layers. In the third stage, the patch size is maintained at 2. Using the pyramid structure of the Transformer encoder, the feature maps are downsampled by factors of 4, 8, and 16, respectively. To eliminate the spatial dimensionality differences between the multi-scale feature maps, the feature maps output by the Transformer branch are bilinearly interpolated during the 8x downsampling stage to ensure that their resolution is strictly aligned with the feature maps output by the CNN branch. This downsampling process is essential for generating feature maps of different scales, allowing for a wider receptive field and capturing a hierarchical representation of the input image. For the other branch, ResNet is used as the backbone network to capture local details of the image, and is also downsampled by factors of 4, 8, and 16, respectively, to ensure that the resolution of the local information extracted by ResNet is consistent with that extracted by Transformer.

[0078] The channel attention module combines local and global information obtained from the CNN and Transformer branches, allowing valuable information to be passed from the parallel encoder to the decoder, resulting in pixel-level segmentation results. Because simple convolution operations are insufficient to effectively fuse local and global features, feature channels are weighted by their information content. By activating channels that contribute to the segmentation result and suppressing irrelevant channels, effective feature fusion is achieved by leveraging the local and global information of the encoder.

[0079] The decoder, the Transformer-CNN network, achieves semantically rich multi-scale feature representation by implementing a parallel encoder and a channel attention module. Similar to TransUNet, the Transformer-CNN network uses skip connections to link low-resolution features to high-resolution features, which are then passed to the decoder to generate the final segmentation mask. The decoder uses convolutional layers to extract combined multi-scale features, including 3×3 convolution, batch normalization, and ReLU layers. Upsampling operations are performed using transposed convolutional layers, obtaining 2x, 2x, and 4x upsampling, respectively. Dice loss and Cross-Entropy are used as loss functions to train the Transformer-CNN network.

[0080] ② Model validation and testing: A dataset of endoscopic ultrasound images was constructed for model validation and testing. The dataset contains endoscopic ultrasound images from different patients, including normal images and images of lesions. The images in the dataset have been segmented and annotated, including segmentation labels for different anatomical structures and lesions. The segmentation performance of the model was evaluated using Dice and JAC coefficients. Higher Dice and JAC coefficients indicate better segmentation performance.

[0081] Construct a real-time navigation model IDEA-EUS based on endoscopic ultrasound video. Figure 2 As shown in the figure, the construction process specifically includes:

[0082] 1. Model Construction. Because the Transformer-CNN network can only process images, using it alone to segment endoscopic images would result in a lack of segmentation consistency across the entire endoscopic video, making it difficult to leverage information between frames. Therefore, we propose incorporating an LSTM network to address this challenge. This network primarily consists of a Transformer-CNN network, a Bi-LSTM layer, and an Attention layer. Samples are taken from the endoscopic video, and the resulting ultrasound endoscopic images are extracted using a Transformer-CNN network. The learned features are then used as input to the Bi-LSTM layer at each time point. The Attention mechanism is then used to learn weights to achieve optimal segmentation results.

[0083] 2. Model details: We plan to use the Transformer-CNN network to extract features from the input ultrasound endoscopy video frames, that is, to process these video frames into feature vectors that can be received and processed by Bi-LSTM.

[0084] The Bi-LSTM consists of a forward-propagating LSTM and a backward-propagating LSTM. In LSTM, a memory controller typically determines which information to retain and forget. Information input and output are implemented through three structures: the input gate, the forget gate, and the output gate. Traditional LSTM networks only consider information during the forward propagation process, thus ignoring future information. In the Bi-LSTM, the current input depends not only on the previous video frame but also on the subsequent video frames, fully considering the temporal information before and after the video frames. The Attention mechanism is a brain signal processing mechanism similar to that unique to human vision. By calculating the weights of the feature vectors output from the Bi-LSTM network at different times, it highlights important features, enabling the overall network model to perform better.

[0085] 3. Model training and optimization: During training, the parameters of the Transformer-CNN network are frozen. Labeled endoscopic ultrasound images and labels serve as input and output for model training. During training, the training data is augmented using methods such as random rotation, image shrinkage and magnification, and blurring. The network is then optimized using optimization methods such as Adam or SGD.

[0086] 4. Model Performance Evaluation: DICE and JAC were used to evaluate the segmentation performance of Transformer-CNN and LSTM network models on anatomical structures and lesions. Model inference time, floating-point operations, and multiply-accumulate operations were used to assess the model's speed and computational complexity. Ultimately, a stable real-time navigation model based on endoscopic ultrasound videos, IDEA-EUS, was developed. IDEA-EUS can identify anatomical structures, such as organs and blood vessels, as well as abnormal structures, in real time within endoscopic ultrasound videos.

[0087] The IDEA-EUS model is optimized by combining multimodal data. The optimization process specifically includes:

[0088] Large space-occupying lesions or tumor invasion can significantly alter the normal abdominal anatomy, a common finding in individuals suitable for endoscopic ultrasound examinations. To address individual anatomical differences, the IDEA-EUS model was optimized by combining multimodal imaging data. A method based on CT / MRI and endoscopic ultrasound image fusion was proposed to construct a multimodal anatomical structure recognition model.

[0089] 1. Data collection: During the collection of ultrasound endoscopic video data, if the patient underwent CT or MRI examination within the previous or subsequent month, the corresponding CT / MRI imaging data will be collected at the same time.

[0090] 2. Data preprocessing and annotation, including:

[0091] For abdominal CT scans, a fixed number of slices was used to identify the most relevant areas of the endoscopic ultrasound video acquisition. Abdominal organs, major vascular structures, and space-occupying lesions were annotated using ITK-SNAP software, using the same labeling as for endoscopic ultrasound. An adaptive method was used to adjust the appropriate window width and window position; image pixel values ​​were normalized to the range of 0–255. Based on convex optimization theory and data probability distribution, image de-meaning and centering preprocessing were performed. Normalization was performed using the Z-score method, and data were normalized based on the mean and standard deviation to approximate a Gaussian distribution. Image enhancement and contrast-constrained adaptive histogram equalization were performed on the grayscale and ground truth masks.

[0092] For abdominal MRI, a fixed number of slices was used to identify the region of the abdominal MRI most relevant to the ultrasound endoscopic video acquisition site. TK-SNAP software was used to annotate abdominal organs, major vascular structures, and space-occupying lesions, using the same labeling as for CT. An adaptive method was used to adjust the appropriate window width and window position; image pixel values ​​were normalized to the range of 0–255. Based on convex optimization theory and data probability distribution, image de-meaning and centering preprocessing were performed, and normalization was performed using the Z-score method. The data were normalized based on the mean and standard deviation to approximate a Gaussian distribution. Image enhancement and contrast-limited adaptive histogram equalization were performed on the grayscale and ground truth masks.

[0093] 3. Model building: Building an anatomical structure recognition model under ultrasound endoscopy video based on multimodal imaging information, including the following steps:

[0094] ① Segment soft tissue targets from ultrasound endoscopy and CT / MRI images based on Mask Scoring R-CNN;

[0095] ② To establish the correspondence between images, feature points are extracted from the segmented soft tissue target area. Feature point extraction methods include corner detection, edge detection, and region centroid detection.

[0096] ③ Due to the different acquisition methods of endoscopic ultrasound and CT / MRI images, there are differences in shape, size, and position. To match the two images, one of them needs to be elastically transformed. Elastic transformation functions include affine transformation using B-spline interpolation, biomechanical model constraints, and finite element simulation.

[0097] ④ To evaluate the effect of image registration, mutual information is defined as the objective function to measure the correlation of information between two images;

[0098] ⑤ Use the gradient descent method to minimize the objective function, solve the optimization problem of the elastic transformation function parameters, and complete the image registration;

[0099] ⑥ Based on the registration results, the endoscopic ultrasound and CT / MRI images are fused together to obtain a fused image;

[0100] ⑦ Based on the fused image, the Transformer-CNN network is used to achieve segmentation of anatomical structures and lesions.

[0101] Example 2:

[0102] Based on the previous embodiment, a dynamic spatiotemporal attention fusion mechanism is added. This mechanism uses the Transformer's self-attention layer to capture the global spatial correlation of organs and blood vessels in a single frame image. At the same time, a bidirectional LSTM is used to extract the temporal evolution features between adjacent frames. The attention weight is adaptively assigned based on the current frame segmentation confidence, suppressing misidentification caused by sudden changes in perspective or image blur.

[0103] Pre-generate a global spatial attention map for each frame of ultrasound endoscopic image using the Transformer encoder , the formula is

[0104]

[0105] Where, is the global spatial attention map; is a function; and are the query and key matrices; is the dimension scaling factor; is the Sigmoid function; is the learnable weight matrix; Local features output by the CNN branch;

[0106] Input the Transformer features of 5 consecutive frames into the bidirectional LSTM and output the temporal attention weights , the formula is:

[0107]

[0108] Where, is the temporal attention weight; is the activation function; is the hidden state of the forward LSTM of the t-1th frame; is the hidden state of the forward LSTM of the t-th frame; is the hidden state of the forward LSTM of the t+1th frame; is the hidden state of the backward LSTM of the t-1th frame; is the hidden state of the backward LSTM of the t-th frame; is the hidden state of the backward LSTM of the t+1th frame; and is a learnable parameter;

[0109] Dynamically fuse spatial and temporal attention based on the segmentation confidence of the current frame:

[0110]

[0111] Where, It is dynamic fusion output; is a dynamic weight parameter used to dynamically adjust the contribution ratio of spatial and temporal attention according to the confidence level; is the slope coefficient, which is used to control the transition smoothness of the fusion weight and is determined by fitting the confidence attenuation curve of the tumor compression case; is the segmentation confidence of the current frame, reflecting the reliability of spatial features; is the confidence threshold, which is used to determine whether to prioritize spatial or temporal features.

[0112] Verification showed that, while achieving a similar average error to the above-mentioned embodiment, the dynamic spatiotemporal attention fusion mechanism improved the Dice coefficient from 89.1% of the baseline model to 92.3% on a test set of 200 pancreatic tumor cases, reduced the anatomical structure misidentification rate of video clips from 12.5% ​​to 6.8%, and reduced the single-frame processing latency to 38ms, meeting real-time clinical requirements. The results show that this mechanism significantly improves the limitations of traditional single-frame segmentation models in dynamic scenes by jointly modeling the spatial features and temporal dependencies within ultrasound endoscopic video frames. It can effectively enhance the accuracy and continuity of anatomical structure segmentation, reduce misidentification caused by sudden changes in viewpoint or image blur, while maintaining real-time processing capabilities and providing endoscopists with smooth and coherent navigation feedback. In addition, its dynamic weight allocation strategy improves the model's adaptability to complex anatomical variations, ensuring robustness in highly dynamic examination scenarios.

[0113] Example 3:

[0114] Building on the previous examples, a multimodal adversarial elastic registration network is added. This network uses a generator to predict the nonlinear deformation field, a discriminator to distinguish between real and synthetic registered images, and a mutual information maximization constraint to ensure the consistency of anatomical structures after deformation, thereby improving the accuracy and generalization of multimodal fusion.

[0115] The generator takes ultrasound endoscopic images and CT images as input and outputs a deformation field, which is formulated as follows:

[0116]

[0117] Where, is the deformation field; For the generator; The image is an endoscopic ultrasound image; is a CT image; are the generator parameters, and the network structure is U-Net+residual block;

[0118] The discriminator distinguishes between real registration pairs and synthetic registration pairs, and the adversarial loss function is:

[0119]

[0120] Where, To combat the loss, a discriminator is used to distinguish between real and synthetic registration pairs. It is the mathematical expectation, which indicates the loss calculation under the real data distribution.

[0121] Mutual information loss uses probability density estimation based on neural network to avoid the dimensionality disaster problem of traditional histogram method and maximize the deformed ultrasound endoscopic image Mutual information with CT images:

[0122]

[0123] Where, For mutual information loss, the mutual information between the deformed endoscopic ultrasound image and the CT image is maximized to maintain anatomical consistency; is the number of training samples; Joint and marginal probability estimators for parameterizing neural networks; For the counter; It is a spatial transformation operation based on the displacement field; is the regularization term weight, which is used to control the smoothness of the displacement field; is the total variation regularization term of the displacement field, which is used to constrain the local smoothness of the deformation field.

[0124] The total loss function integrates adversarial loss, mutual information constraint and deformation field regularization:

[0125] Where, is the output of the total loss function; 、 、 and It is the balance factor of each loss, which is used to adjust the weights among adversarial loss, mutual information loss, and displacement field regularization term; and is the balance coefficient of the total variation and the L2 regularization term; is the total variation; is the L2 regularization term; is the segmentation confidence of the current frame, reflecting the reliability of the registration result; is the confidence threshold, which is used to dynamically adjust the regularization strength; is the KL divergence regularization term, which is used to constrain the displacement field distribution output by the generator.

[0126] Suppose multimodal imaging data from 50 pancreatic cancer patients is collected, each consisting of dynamic endoscopic ultrasound video, abdominal CT, and MRI images. Of these, 30 patients have normal local anatomy, while 20 have significant pancreatic morphology distortion due to tumor invasion. A multimodal adversarial elastic registration network is used to perform cross-modal registration between endoscopic ultrasound and CT, and the performance of the traditional B-spline registration algorithm is compared.

[0127] The effect of the multimodal adversarial elastic registration network is shown in Table 1.

[0128] Table 1. Summary of the effects of multimodal adversarial elastic registration network

[0129]

[0130] As shown in the experimental table, by manually annotating 10 anatomical landmarks in EUS and CT images and calculating the Euclidean distance of corresponding points after registration, the multimodal adversarial elastic registration network reduced the TRE to 1.2 mm in cases with anatomical variation, demonstrating that its elastic deformation field can effectively compensate for nonlinear displacement. Measuring the overlap of pancreatic regions after EUS and CT segmentation, the multimodal adversarial elastic registration network achieved a mean of 0.91 with a standard deviation of 4.3%, demonstrating significant improvement in generalization across different cases and medical centers. Adversarial training accelerated the convergence of the multimodal adversarial elastic registration network, reducing the number of iterations by 40% and the single-case registration time to 1.5 seconds, meeting clinical real-time requirements. Experimental validation demonstrated that it significantly improved the registration accuracy of EUS and CT / MRI images, especially in cases with severe anatomical variation. It effectively compensated for nonlinear displacement and ensured the reliability of multimodal feature fusion. Furthermore, the adversarial training strategy enhanced the model's generalization to data distributions across different medical centers, reducing the reliance on annotated data, and providing a technical foundation for accurate navigation in complex cases.

[0131] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable non-transitory storage media containing computer-usable program code.

[0132] The present invention can provide computer program instructions to a management platform of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the management platform of the computer or other programmable data processing device produce a device for implementing the system.

[0133] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device that implements the functions of the system.

[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions of the described system.

Claims

1. An ultrasound endoscopic navigation system based on deep learning, characterized in that: include: A parallel encoder module for extracting local features of the endoscopic ultrasound image through a CNN branch and capturing multi-scale global contextual information through a pyramid-structured Transformer branch; the Transformer branch includes a non-overlapping patch embedding layer and multi-stage downsampling; Channel attention fusion module, used to perform adaptive weighted fusion of extracted features; The decoder module generates anatomical structure segmentation masks based on the fused features through multi-level skip connections; A timing processing module is used to receive the frame-by-frame segmentation results output by the decoder module and implement temporal coherence analysis of the ultrasound endoscopy video through a bidirectional LSTM network combined with an Attention mechanism; A multimodal fusion module for registering and fusing endoscopic ultrasound images; the multimodal fusion module includes an adversarial elastic registration submodule that uses a generator to predict a nonlinear deformation field and a discriminator to distinguish between real and synthetic registered images, while incorporating mutual information maximization constraints to ensure anatomical consistency; The spatiotemporal attention fusion module is used to dynamically allocate spatial and temporal attention weights according to the segmentation confidence of the current frame.

2. The system according to claim 1, wherein: The resolution of the feature map output by the Transformer branch in the 8x downsampling stage is aligned with that of the CNN branch, and the spatial dimension matching of the feature map is achieved through bilinear interpolation.

3. The system according to claim 1, wherein: The upsampling operation of the decoder module uses a transposed convolutional layer to perform 2x, 2x, and 4x upsampling in sequence, and the loss function is jointly optimized using Dice Loss and cross entropy loss.

4. The system according to claim 1, wherein: The spatiotemporal attention fusion module uses the self-attention layer of the Transformer branch to capture the global spatial correlation between organs and blood vessels in a single frame image, extracts the temporal evolution features between adjacent frames through a bidirectional LSTM network, and dynamically allocates spatial and temporal attention weights based on the segmentation confidence of the current frame. The fusion formula is: Where, It is dynamic fusion output; is the dynamic weight parameter; is the global spatial attention map; is the temporal attention weight; is the slope coefficient, determined by fitting the confidence decay curve of the tumor compression case; is the segmentation confidence of the current frame; is the confidence threshold.

5. The system according to claim 1, wherein: The total loss function of the adversarial elastic registration submodule is: Where, is the output of the total loss function; 、 、 and is the balancing factor for various losses; To combat losses; is the mutual information loss; and is the balance coefficient of the total variation and the L2 regularization term; is the total variation; is the L2 regularization term; is the segmentation confidence of the current frame; is the confidence threshold; is the KL divergence regularization term.

6. A method for annotating anatomical structures based on ultrasound endoscopy video that does not involve diagnosis and treatment of diseases, characterized in that: include: Filter invalid frames in ultrasound endoscopy videos based on the CNN model and standardize valid frames; Performing anatomical segmentation on a single frame using a parallel encoder; the parallel encoder comprises a CNN branch and a pyramid-structured Transformer branch that captures multi-scale global contextual information through non-overlapping patch embedding layers and multi-stage downsampling; The bidirectional LSTM network combined with the Attention mechanism is used to correct the temporal consistency of the segmentation results of consecutive frames, and the spatial attention weight and temporal attention weight are dynamically allocated according to the segmentation confidence of the current frame.

7. The method according to claim 6, characterized in that: The normalization process further includes metadata anonymization, specifically cropping the region of interest or replacing the patient privacy information in the image with a black mask.

8. A deep learning-based ultrasound endoscopic navigation method not involving disease diagnosis and treatment, characterized by: The method is implemented based on the system according to any one of claims 1 to 5: The method comprises: Extract local features of ultrasound endoscopic images and capture multi-scale global context information; Adaptively weight the extracted features and generate anatomical structure segmentation masks based on the fused features; The output frame-by-frame segmentation results are received and the temporal coherence analysis of the ultrasound endoscopy video is realized through a bidirectional LSTM network; The endoscopic ultrasound images were registered and fused.

9. A computer device comprising a management platform and a memory, wherein the management platform is connected to the memory, and the memory is used to store a computer program, wherein: The management platform is used to execute the computer program stored in the memory, so that the computer device executes the method according to claim 8.

10. A computer-readable storage medium storing a computer program, wherein: The computer program executes the method of claim 8 when executed.

Images