Part identification method based on upper digestive tract time sequence information

Abstract

A multi-modal MLKNet recognition method based on upper digestive tract time sequence information comprises the steps that a time sequence feature vector is generated for a gastroscope part image in an image-text alignment mode, the time sequence information is fused into a single-frame image through a multi-scale hierarchical fusion strategy, a feature fusion network adopts a three-way scanning Mamba and a large convolution kernel module, and the time sequence feature vector and the time sequence information are fused into the single-frame image; the time sequence information is expanded to the full image and the receptive field is increased. A data set is divided into a training set and a verification set after being preprocessed and added with time sequence description, a semantic encoder is pre-trained, an image coding network is constructed, image-text feature alignment is carried out, output loss containing various losses is calculated, and parameters are updated through back propagation. According to the method, the distinguishing degree of similar feature images is enhanced, and the robustness of deformation feature classification is improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically the field of medical upper gastrointestinal optical endoscopy, and is a site identification method based on temporal information of the upper gastrointestinal tract. Background Technology

[0002] With the development of medical imaging technology, upper gastrointestinal endoscopy has become a core method for early screening and lesion localization of gastric cancer. However, traditional endoscopic image analysis techniques have the following key shortcomings:

[0003] The temporal fragmentation of single-frame static image analysis: Existing mainstream algorithms (such as YOLOv5 and Faster R-CNN) typically perform site identification based on independent single frames, neglecting the temporal correlation of anatomical structures during continuous endoscopic advancement. The anatomical features of sites such as the stomach body, pylorus, and duodenum exhibit dynamic and progressive characteristics. Single-frame analysis is prone to the following misclassifications: misclassification of similar mucosal textures (e.g., lower stomach body versus duodenal bulb); feature loss due to local occlusion (e.g., pyloric identification under the interference of secretions); and lack of spatial continuity constraints (e.g., incorrect determination of the direction across the gastric angle).

[0004] The limitations of spatiotemporal feature fusion architecture; traditional CNN architecture has inherent defects in long-range temporal modeling: conventional 3×3 convolution kernels have limited receptive fields, making it difficult to capture the spatial correlation of large-scale anatomical landmarks; although Transformer can model global dependencies, its quadratic complexity computational overhead is difficult to meet the needs of real-time endoscopic diagnosis (>30fps video stream processing); existing temporal fusion methods (such as 3D-CNN, LSTM) suffer from feature dimension explosion, leading to a massive increase in the number of model parameters (e.g., the I3D model has more than 50M parameters).

[0005] Insufficient adaptability to dynamic environments; clinical endoscopic videos face the following challenges: motion blur caused by lens movement (average blur radius of 5-15 pixels); specular highlights caused by intracavitary fluid reflection (covering 8-15% of the image area); anatomical deformation caused by peristaltic waves (local deformation rate exceeding 20%); existing methods such as Mask R-CNN show a significant decrease in mAP in dynamic scenes (measured decrease of 12.7%), and cannot utilize temporal redundancy information for self-correction.

[0006] Domain-specific processing is lacking; upper gastrointestinal endoscopic images have unique data characteristics: subtle inter-class differences (e.g., the esophageal-cardia junction is distinguished only by the distribution pattern of blood vessels); non-linear shift in color space (differences in RGB response curves of endoscopes from different manufacturers); anatomical topological constraints (e.g., the duodenum must be located distal to the pylorus); traditional data augmentation methods (rotation / cropping) can destroy inherent anatomical constraints, causing the model to learn non-physiological feature associations.

[0007] In the field of medical gastroscopy site recognition, deep learning has become the mainstream research method due to its powerful feature extraction capabilities. A neural network is built and trained on a certain number of labeled image samples of gastroscopy sites to obtain a model for site classification. This model is then connected to the endoscopy system, where it receives video streams in real time and outputs the current site category on the monitor.

[0008] Current part recognition models address issues such as feature extraction, multi-scale feature interaction, and dynamic blurring. For gastroscopy parts with subtle background differences and indistinct foreground features, it is necessary to incorporate temporal information to improve recognition accuracy. Summary of the Invention

[0009] This invention addresses the shortcomings of gastroscopy site recognition technology by proposing a temporal information-based method for gastroscopy site recognition. This method overcomes the limitations of conventional target recognition models in complex surgical environments and the difficulty of combining sequential temporal information from a single image. The invention injects temporal information into the input image through a semantic coding network and integrates a Mamba structure into the detection network to enhance the fusion of temporal and image feature information. Large convolutional kernels are used to expand the feature receptive field. This solves the problems of highly similar features among multiple sites in the upper gastrointestinal tract in a single image and the difficulty in identifying deformities caused by intraoperative stimulation in different patients.

[0010] The idea behind this invention is as follows: This invention utilizes image-text alignment to generate matching temporal feature vectors for gastroscopy site images; through a multi-scale hierarchical fusion strategy, temporal information is fully integrated into a single frame image; during feature fusion, a three-way Mamba and large convolutional kernel module is used to extend the temporal information to the entire image and increase the receptive field of local features, thereby improving the accuracy of site recognition based on gastroscopy video stream input.

[0011] The specific steps of the multimodal MLKNet (Mamba network with large convolutional kernels) recognition method based on upper gastrointestinal time-series information of the present invention are as follows:

[0012] Step 1: Preprocess the gastroscopy data by segmenting and classifying the gastroscopy images;

[0013] Step 2: Extract temporal information from the classified images of each part, and describe the category information and temporal information with text to form image-text pairs;

[0014] Step 3: Based on the pre-trained language model, convert the text description into a vector representation to obtain temporal text features; and construct positive and negative sample pairs;

[0015] Step 4: Use an image encoder to convert the image into multi-scale image features;

[0016] Step 5: Image-text feature alignment;

[0017] Step 6: Construct a feature fusion network, including a low-level feature aggregation module, a high-level feature aggregation module, and a feature injection module; temporal features and multi-scale image features are input to the low-level feature aggregation module to obtain first LI local features and second LI local features. The first LI local features, along with low-scale features and temporal features from the multi-scale image features, are input to the feature injection module to obtain the first injected feature; the second LI local features, along with mid-scale features and temporal features from the multi-scale image features, are input to the feature injection module to obtain the second injected feature.

[0018] The first injected feature, the second injected feature, and the high-scale feature from the multi-scale image features are input into the high-level feature aggregation module to obtain the first HI local feature and the second HI local feature.

[0019] The first HI local feature, the second injected feature, and the temporal feature are input into the feature injection module to obtain the first output feature; the second HI local feature, the high-scale feature, and the temporal feature are input into the feature injection module to obtain the second output feature; the first injected feature is used as the third output feature.

[0020] Step 7: Calculate the output loss function and perform backpropagation to update the parameters, and train the model;

[0021] Step 8: Decouple the three output features to obtain the text output and image category output.

[0022] The low-level feature aggregation module specifically includes multi-scale feature alignment, image-text feature interaction, Mamba blocks, and feature segmentation.

[0023] The multi-scale feature alignment aligns the multi-scale features according to the mesoscale, and then performs channel concatenation to obtain fused image features; the image-text feature interaction uses a cross-attention mechanism to fuse the fused image features and text features; then, after passing through a Mamba block, the obtained features are segmented by channel to obtain the final first LI local features and second LI local features.

[0024] The high-level feature aggregation module includes feature alignment, a large convolutional kernel module, and feature segmentation;

[0025] After the first and second injected features are aligned with the high-scale features in the multi-scale image features, a large convolutional kernel module is used to expand the receptive field of the fused features. The large convolutional kernel module specifically includes a large kernel convolutional layer, a batch normalization layer, and a fully connected layer. The three features after feature alignment are input to the large kernel convolutional layer, and then average pooling is used to obtain channel-level features. Channel weight calibration is achieved through two fully connected layers. The first layer reduces the dimensionality, and the second layer increases the dimensionality back to the original number of channels. An activation function is used to generate a weight vector in between. Then, it goes through two linear transformations, with a non-linear feedforward module introduced through an activation function in between. Finally, after passing through the batch normalization layer, it is residually connected with the input of the large convolutional kernel module to obtain the final fused features. The fused features are then segmented by channel to obtain the final first HI local features and second HI local features.

[0026] The loss function includes classification loss and image-text comparison loss:

[0027] L all =L cls +L itc

[0028] Among them, L all L represents the total loss. cls L represents the classification loss. itc This indicates the loss in image-text contrast.

[0029] The classification loss function comprises a weighted sum of dynamic focus loss and binary cross-entropy loss. Let C be the number of anatomical site categories and p be the input prediction probability. i Dynamic focus loss L DFL Specifically as follows:

[0030]

[0031] Among them, f i ω represents the frequency of class i in the training set, ∈ represents the smoothing factor, λ represents the scaling factor (default is 1.0), γ represents the focus modulation factor, used to adjust the weights of easy and difficult samples, and ω represents the focus modulation factor. i Let i be the frequency weight of category i.

[0032] Furthermore, the image-text feature alignment specifically includes the following steps:

[0033] Step 5.1: Align the single-modal representation using image-text alignment loss;

[0034] Step 5.2: Maximize the similarity of positive samples by storing historical features in a queue and expanding the negative sample pool.

[0035] Furthermore, step 8 is detailed as follows:

[0036] Step 8.1: Concatenate the first output feature, the second output feature, and the third output feature together and input them into the decoder;

[0037] Step 8.2: Input the decoder output into a projection layer and a classification layer respectively. The projection layer projects the decoder output features to obtain the text output, and the classification layer converts the encoder output features into 1-dimensional features and obtains the image category output through the softmax activation function.

[0038] Furthermore, when performing feature alignment, bilinear interpolation is used when the feature is smaller than the alignment scale, and average pooling is used when the feature is larger than the alignment scale.

[0039] Furthermore, the categories of the gastroscopy images include: external, blurred, esophagus, cardia, gastric body, gastric antrum, pylorus, duodenal bulb, descending duodenum, gastric angle, and gastric fundus.

[0040] Furthermore, the specific structure of the Mamba block is as follows: after normalization, the features are input into the three-directional scanning Mamba block, and other pixels in the horizontal, vertical and 45° directions are scanned. Then, the features are processed by normalization and multilayer perceptron (MLP) and residual connections are performed to obtain the final feature output.

[0041] The beneficial effects of the present invention are as follows:

[0042] First, this invention utilizes a multimodal strategy of image-text pairs and constructs an MLKNet model using a dual encoder approach. By encoding the temporal features between images obtained through text descriptions, the discriminability between images with similar features but different parts is enhanced, overcoming the problem that existing models have difficulty distinguishing highly similar images, thus enabling this invention to achieve higher classification accuracy.

[0043] Second, the feature fusion part uses the Mamba module and large convolution kernel to enhance the global information of image features and expand the receptive field of the image, so that temporal features can be fully fused with image features. This overcomes the problem that existing feature fusion modules cannot enable features to interact fully, and improves the robustness of the present invention in classifying different deformation features. Attached Figure Description

[0044] Figure 1 This is a block diagram of the MLKNet model architecture based on time-series information according to the present invention.

[0045] Figure 2 This is an internal flowchart of the feature fusion block used in this invention.

[0046] Figure 3 This is an internal flowchart of the Low-IT block of the feature fusion block used in this invention.

[0047] Figure 4 This is an internal flowchart of the High-IT block of the feature fusion block used in this invention.

[0048] Figure 5 This is an internal flowchart of the Mamba block of the Low-IT block used in this invention.

[0049] Figure 6 This is an internal flowchart of the LK-Conv block of the High-IT block used in this invention. Detailed Implementation

[0050] To better understand the purpose, structure, and function of this invention, the following detailed description of a multimodal MLKNet recognition method based on upper gastrointestinal time-series information is provided in conjunction with the accompanying drawings.

[0051] like Figure 1 As shown, the multimodal MLKNet recognition method based on upper gastrointestinal time-series information provided in this embodiment specifically includes the following steps:

[0052] Step 1: Preprocess the gastroscopy data.

[0053] Step 1.1: Train a simple binary classification network to segment the acquired gastroscopy videos, and save each video segment as a complete endoscopic procedure for one patient;

[0054] Step 1.2: Extract images of various parts from the gastroscopy video, including 10 endoscopic parts and other label categories: external, blurred, esophagus, cardia, gastric body, gastric antrum, pylorus, duodenal bulb, descending duodenum, gastric angle, and gastric fundus;

[0055] Step 1.3: Crop the image, remove black borders, and retain the actual gastroscopy portion. The actual cropping area is x-axis: 30-1045, y-axis: 700-1862. Scale the image to 256x256 for easier model input.

[0056] Step 1.4: Save the images with similar features in a folder named "Label Name," which will be the category label for each image. Set up a dictionary data structure to map the category names to the numbers 0-9.

[0057] Step 2: Add a description of the timing information to the part image.

[0058] Step 2.1: Add corresponding temporal information to each image using a description formatted as follows: (Current image part is..., Previous part is..., Time interval is..., Next part is..., Time interval is...);

[0059] Step 2.2: Design a program that extracts parts from a video and outputs descriptive information, and create a queue to store context information;

[0060] Step 2.3: Verify the output description to ensure that the description information corresponds one-to-one with the image of the corresponding body part;

[0061] Step 2.4: Finally, obtain the image-text pair data structure and save it in JSON format.

[0062] Step 3: Divide the dataset into training and validation sets in an 8:2 ratio. The partitioning is randomized and uses 10-fold cross-validation.

[0063] Step 4: Pre-train a semantic encoder.

[0064] Step 4.1: Based on a pre-trained language model (such as ModernBERT), construct a text encoder to convert the category information and sequential information in the text description into vector representations;

[0065] Step 4.2: Anatomy-Temporal Comparison Learning: ① Construct positive sample pairs: match the correct anatomical location with the corresponding temporal description (e.g., "gastric antrum, preceding location is gastric body"); ② Construct negative sample pairs: randomly replace anatomical locations or disrupt temporal logic (e.g., incorrectly associate "duodenum" with "preceding cardiac cardia");

[0066] Step 5: Construct an image coding network.

[0067] Step 5.1: Based on the commonly used image encoder vit (vision transformer), convert the image features into vector representations;

[0068] Step 5.2: Output image features B2, B3, B4, and B5 at different scales for subsequent feature fusion.

[0069] Step 6: Image-text feature alignment.

[0070] Step 6.1: Align single-modal representations using image-text alignment loss to simplify subsequent feature fusion;

[0071] Step 6.2: Maximize the similarity of positive samples by storing historical features in a queue to expand the negative sample pool.

[0072] Step 7: Construct a feature fusion network. For example... Figure 2As shown, the feature fusion network consists of a low-level feature aggregation module (Low-IT), a high-level feature aggregation module (High-IT), and a feature injection module (Inject). Temporal features (seq) and features at four scales are input into Low-IT, resulting in two LI local feature outputs. The first LI local feature, along with B3 and seq, is used as input to Inject, resulting in the injected feature P3. The second LI local feature, along with B4 and seq, is used as input to Inject, resulting in the injected feature P4. P5 is consistent with B5. P3, P4, P5, and seq are used as input to High-IT, resulting in two HI local feature outputs. The first HI local feature, along with P4 and seq, is used as input to Inject, resulting in the injected feature N4. The second HI local feature, along with P5 and seq, is used as input to Inject, resulting in the injected feature N5. N3 is consistent with P3. Through low-level feature aggregation, high-level feature aggregation, and feature injection, temporal and image features can be fully fused, and local and global features can interact effectively, allowing local features to contain more image texture and temporal information.

[0073] Step 7.1: Design the Low-IT module. For example... Figure 3 As shown, this module includes multi-scale feature alignment, image-text feature interaction, Mamba amplification of temporal information effect, and feature segmentation.

[0074] Step 7.1.1: Multi-scale feature alignment. The features extracted by the image encoder are aligned according to the scale of B4. The feature size of B5 is smaller than that of B4, so bilinear interpolation is used. The feature sizes of B2 and B3 are larger than that of B4, so average pooling is used. After unifying the features of different sizes to the size of B4, channel stitching is performed.

[0075] Step 7.1.2: Use cross attention (CA) to initially fuse the image features and text features. Since the text sequence and image features have different dimensions, in order to fuse them, it is necessary to first project (Proj) the text sequence and the concatenated image features to ensure that their dimensions are consistent;

[0076] Step 7.1.3: Use Mamba blocks to fully integrate the temporal information from the text features into the image features. For example... Figure 5As shown, the Mamba block mainly consists of two LayerNorm modules, a three-way scanning Mamba (TOM) module, and a multilayer perceptron (MLP). The TOM module scans other pixels in the horizontal, vertical, and 45° directions of each pixel, enabling each pixel to obtain long-range information about its surrounding pixels. The entire Mamba block inputs the initially fused features into the TOM module after LayerNorm normalization for global information interaction, and performs residual connections between the output and input features. The features are then further processed by LayerNorm and MLP, and residual connections are performed to facilitate gradient flow. While the Mamba block is popular for its ability to handle longer sequences, its processing of image information is still insufficient. The proposed bidirectional Mamba can effectively handle information between preceding and following sequences, but in two-dimensional images, information between preceding and following sequences is not enough; information between vertical and horizontal sequences is also needed. The three-way scanning Mamba can acquire local information from vertical, horizontal, and preceding and following sequences. Combined with the long-range advantage of Mamba, it can effectively fuse temporal information with image features.

[0077] Step 7.1.4: Segment the fused features in the channel dimension into two LI local features. The first LI local feature has 192 channels and the second LI local feature has 96 channels, which will be used for subsequent feature injection.

[0078] Step 7.2: Design the High-IT module. For example... Figure 4 As shown, this module includes feature alignment, a large convolutional kernel module, and a feature segmentation part.

[0079] Step 7.2.1: The feature alignment part aligns the features processed by the low-level feature aggregation module and the feature injection module according to the scale of P5. Since the size of P3 and P4 is larger than P5, average pooling is used to unify the size to P5, and then channel splicing is performed.

[0080] Step 7.2.2: Expand the receptive field of the fused features using the large convolutional kernel module LK-Conv. The large convolutional kernel module is as follows: Figure 6As shown, the architecture consists of a large-kernel convolutional layer (DW Conv), batch normalization (BN), downsampling, and upsampling layers, employing ReLU and GeLU as activation functions. Specifically, the input features are first fed into the DW Conv module, which is a large-kernel dilated convolutional block, yielding global features. These features are then compressed using global average pooling to obtain channel-level features. Channel weight calibration is achieved through two fully connected layers (FC): the first layer reduces dimensionality (e.g., channel number × 1 / 4), and the second layer increases dimensionality back to the original channel number, with ReLU activation used in between to generate weight vectors. These weight vectors are multiplied by the input feature map to enhance channel-level features. The output undergoes two linear transformations (1×1 convolutional or fully connected layers), with a non-linear feedforward module (FFN) introduced through activation functions such as GeLU to enhance the non-linear expressive power of the features. Finally, after passing through a batch normalization layer, the output is residually connected to the input of the large convolutional kernel module, with the batch normalization layer used to accelerate convergence.

[0081] Step 7.2.3: Segment the fused features in the channel dimension into two HI local features. The first HI local feature has 192 channels and the second HI local feature has 384 channels, which will be used for subsequent feature injection.

[0082] Step 7.3: The feature injection module fuses the text features (seq) output by the text encoder, the image features output by the image encoder, and the features output by the aggregation module, so that the features at different scales have richer semantic information.

[0083] Step 7.4: Output the features N3, N4, and N5 at three scales.

[0084] Step 8: Calculate the output loss and perform backpropagation to update the parameters.

[0085] Step 8.1: The basic loss function is the Binary Cross Entropy WidthLogits Loss (BCEWithLogitsLoss).

[0086]

[0087] Where N represents the number of samples, and the true label is y. i ∈{0,1}, the model predicts the probability as p(y i ), (probability after Sigmoid activation), log represents taking the logarithm.

[0088] Step 8.2: Improve the detection loss by adding a new loss method – Dynamic Focal Loss – to the basic loss. This automatically adjusts the loss weights based on the frequency of occurrence of each anatomical location (antrum weight > fundus weight). Specifically, based on the basic binary cross-entropy loss (BCEWithLogitsLoss), Dynamic Focal Loss (DFL) is designed to address the class imbalance problem in anatomical location detection. This loss optimizes the model through the following mechanism:

[0089] Category frequency weights: The loss weights are dynamically adjusted based on the frequency of occurrence of each anatomical site in the training data, with higher weights assigned to low-frequency categories (such as the gastric antrum).

[0090] Modulation of difficult and easy samples: The focal factor is introduced to reduce the loss contribution of easy-to-classify samples and strengthen the model's focus on difficult samples.

[0091] Let C be the number of anatomical site categories and p be the input prediction probability. i (Probability after Sigmoid activation), the true label is y i The dynamic focus loss is defined as follows: ∈{0,1}.

[0092]

[0093] Among them, f i ω represents the frequency of class i in the training set, ∈ represents the smoothing factor, λ represents the scaling factor (default is 1.0), γ represents the focus modulation factor, used to adjust the weights of easy and difficult samples, and ω represents the focus modulation factor. i Let i be the frequency weight of category i.

[0094] The final classification loss is a weighted average of the basic loss and the dynamic focus loss:

[0095] L cls =αL BCE +(1-α)L DFL

[0096] Step 8.3: The image-text contrastive learning (ITC) loss is adopted. The standard image-text contrastive learning loss function, ITC loss, is defined as follows:

[0097]

[0098] The image-to-text similarity distribution is as follows:

[0099]

[0100] The text-to-image similarity distribution is as follows:

[0101]

[0102] E (I,T)～D H represents the expected value of the loss calculated from sampling image-text pairs (I,T) from data distribution D, where H represents the cross-entropy calculation, I(i) represents the input image, T(t) represents the input text description, and y represents the input text description. i2t (I) represents the text description that the image should match, y t2i (T) indicates the image that the text description should match.

[0103] Step 8.4: The final loss function consists of classification loss and image-text comparison loss:

[0104] L all =L cls +L itc

[0105] Step 9: Decouple the output features to obtain text output and image category output.

[0106] Step 9.1: Concatenate the output features N3, N4, and N5 together and input them into the decoder. The decoder is constructed from four upsampling layers and convolutional layers, as in a conventional model.

[0107] Step 9.2: Input the decoder output into a projection layer and a classification layer respectively. The projection layer projects the decoder output features into 512 dimensions to obtain the text output. The classification layer converts the encoder output features into 1-dimensional features and obtains the image category output through the softmax activation function.

[0108] Step 10: Train the model.

[0109] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for site identification based on temporal information of the upper digestive tract, characterized in that, Includes the following steps: Step 1: Preprocess the gastroscopy data by segmenting and classifying the gastroscopy images; Step 2: Extract temporal information from the classified images of each part, and describe the category information and temporal information with text to form image-text pairs; Step 3: Based on the pre-trained language model, convert the text description into a vector representation to obtain temporal text features; and construct positive and negative sample pairs; Step 4: Use an image encoder to convert the image into multi-scale image features; Step 5: Image-text feature alignment; Step 6: Construct a feature fusion network, including a low-level feature aggregation module, a high-level feature aggregation module, and a feature injection module; Temporal features and multi-scale image features are input to a low-level feature aggregation module to obtain first LI local features and second LI local features. The first LI local features, along with low-scale features and temporal features from the multi-scale image features, are input to a feature injection module to obtain first injected features. The second LI local features, along with mid-scale features and temporal features from the multi-scale image features, are input to a feature injection module to obtain second injected features. The first injected feature, the second injected feature, and the high-scale feature from the multi-scale image features are input into the high-level feature aggregation module to obtain the first HI local feature and the second HI local feature. The first HI local feature, the second injected feature, and the temporal feature are input into the feature injection module to obtain the first output feature; the second HI local feature, the high-scale feature, and the temporal feature are input into the feature injection module to obtain the second output feature; the first injected feature is used as the third output feature. Step 7: Calculate the output loss function and perform backpropagation to update the parameters, and train the model; Step 8: Decouple the three output features to obtain the text output and image category output.

2. The method for site identification based on upper gastrointestinal tract temporal information according to claim 1, characterized in that, The low-level feature aggregation module specifically includes multi-scale feature alignment, image-text feature interaction, Mamba blocks, and feature segmentation. The multi-scale feature alignment aligns the multi-scale features according to the mesoscale, and then performs channel concatenation to obtain fused image features; the image-text feature interaction uses a cross-attention mechanism to fuse the fused image features and text features; then, after passing through a Mamba block, the obtained features are segmented by channel to obtain the final first LI local features and second LI local features.

3. The method for site identification based on upper gastrointestinal tract temporal information according to claim 2, characterized in that, The high-level feature aggregation module includes feature alignment, a large convolutional kernel module, and feature segmentation; After the first and second injected features are aligned with the high-scale features in the multi-scale image features, a large convolutional kernel module is used to expand the receptive field of the fused features. The large convolutional kernel module specifically includes a large kernel convolutional layer, a batch normalization layer, and a fully connected layer. The three features after feature alignment are input to the large kernel convolutional layer, and then average pooling is used to obtain channel-level features. Channel weight calibration is achieved through two fully connected layers. The first layer reduces the dimensionality, and the second layer increases the dimensionality back to the original number of channels. An activation function is used to generate a weight vector in between. Then, it goes through two linear transformations, with a non-linear feedforward module introduced through an activation function in between. Finally, after passing through the batch normalization layer, it is residually connected with the input of the large convolutional kernel module to obtain the final fused features. The fused features are then segmented by channel to obtain the final first HI local features and second HI local features.

4. The method for site identification based on upper gastrointestinal tract temporal information according to claim 3, characterized in that, The loss function includes classification loss and image-text comparison loss: L all L cls +L itc Among them, L all L represents the total loss. cls L represents the classification loss. itc This indicates the loss in image-text contrast. The classification loss function comprises a weighted sum of dynamic focus loss and binary cross-entropy loss. Let C be the number of anatomical site categories and p be the input prediction probability. i Dynamic focus loss L DFL Specifically as follows: Among them, f i ω represents the frequency of class i in the training set, ∈ represents the smoothing factor, λ represents the scaling factor (default is 1.0), γ represents the focus modulation factor, used to adjust the weights of easy and difficult samples, and ω represents the focus modulation factor. i Let i be the frequency weight of category i.

5. The method for site identification based on upper gastrointestinal tract temporal information according to claim 4, characterized in that, The image-text feature alignment specifically includes the following steps: Step 5.1: Align the single-modal representation using image-text alignment loss; Step 5.2: Maximize the similarity of positive samples by storing historical features in a queue and expanding the negative sample pool.

6. The method for site identification based on upper gastrointestinal tract temporal information according to claim 5, characterized in that, Step 8 is described in detail below: Step 8.1: Concatenate the first output feature, the second output feature, and the third output feature together and input them into the decoder; Step 8.2: Input the decoder output into a projection layer and a classification layer respectively. The projection layer projects the decoder output features to obtain the text output, and the classification layer converts the encoder output features into 1-dimensional features and obtains the image category output through the softmax activation function.

7. The method for site identification based on upper gastrointestinal tract temporal information according to claim 6, characterized in that, When performing feature alignment, bilinear interpolation is used when the feature is smaller than the alignment scale, and average pooling is used when the feature is larger than the alignment scale.

8. The method for site identification based on upper gastrointestinal tract temporal information according to claim 7, characterized in that, The categories of gastroscopy images include: external, blurred, esophagus, cardia, gastric body, gastric antrum, pylorus, duodenal bulb, descending duodenum, gastric angle, and gastric fundus.

9. A method for site identification based on upper gastrointestinal tract temporal information according to claim 8, characterized in that, The specific structure of the Mamba block is as follows: After normalization, the features are input into the three-directional scanning Mamba block, which scans other pixels in the horizontal, vertical and 45° directions. Then, the features are processed by normalization and multilayer perceptron (MLP) and residual connections are performed to obtain the final feature output.

Images