A spatiotemporal feature processing system for atrophic gastritis endoscopic videos

Abstract

The application discloses a kind of atrophic gastritis endoscope video space-time feature processing systems, it is related to medical image analysis field, the system includes: video stream access module, for obtaining gastroscopy video stream, and is divided into video clip;Space feature extraction network is used to extract the space domain visual feature related to atrophic gastritis in each frame image, and generates the segment space feature vector of video clip;Double-branch space-time feature fusion module is used to cascade the time sequence feature and attention enhanced feature output by double-branch and input classification discriminant layer, and output atrophic gastritis degree classification probability distribution;Gastric cavity anatomical site identification network is used to classify anatomical site to current video frame, and output the probability distribution of gastric cavity anatomical site;Multi-modal semantic mapping engine is used to output endoscope auxiliary evaluation report.The application can eliminate subjective experience difference in classification determination and site description in artificial interpretation, and improve the objectivity of atrophic gastritis endoscope diagnosis.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis, and in particular to a spatiotemporal feature processing system for endoscopic videos of atrophic gastritis. Background Technology

[0002] Chronic atrophic gastritis is a chronic gastric disease characterized by glandular atrophy and thinning of the gastric mucosa due to repeated inflammatory damage. It is widely recognized as a key step in the progression of precancerous lesions of the stomach. Most gastric cancers follow the Correa cascade pattern, undergoing a long-term evolution from chronic inflammation, atrophy, intestinal metaplasia to dysplasia and finally carcinogenesis. Therefore, early identification of atrophic gastritis is of great significance for the early prevention and treatment of gastric cancer. Currently, gastroscopy combined with pathological biopsy is the mainstream method for diagnosing atrophic gastritis. In clinical practice, the Kimura-Takemoto classification is widely used for endoscopic grading and risk assessment of atrophic gastritis. This classification, based on the distribution of the atrophy boundary in the gastric mucosa, divides the degree of atrophy from localized to widespread and from mild to severe into six levels: closed type (C1, C2, C3) and open type (O1, O2, O3). The atrophy boundary begins in the antrum and gradually extends towards the lesser curvature, cardia, and greater curvature of the gastric body. C1 to O3 constitute a continuous spectrum of progressively increasing atrophy severity; the wider the atrophy area, the higher the risk of gastric cancer. The Kimura-Takemoto classification is not only the core basis for endoscopists to assess the degree of atrophy and formulate follow-up strategies, but also an important tool for gastric cancer risk stratification. In recent years, deep learning technology has been gradually applied to the field of endoscopic image analysis. Some studies have constructed static image binary classification recognition models or lesion region segmentation models for atrophic gastritis based on convolutional neural networks; other researchers have attempted to develop real-time monitoring models for endoscopic videos using architectures such as U-Net.

[0003] Despite numerous studies introducing deep learning technology into the endoscopic diagnosis of atrophic gastritis, existing methods still have the following significant limitations. First, most methods analyze single-frame static images, failing to fully utilize the temporal information inherent in the endoscope's movement, rotation, and dynamic mucosal exposure during gastroscopy. While a few studies have attempted video stream modeling, their approaches primarily rely on frame-by-frame segmentation or classification, lacking systematic modeling of temporal dependencies between frames within a video segment. Second, in assessing the degree of atrophy, most existing methods only perform binary classification (atrophic / non-atrophic) or lesion region segmentation, failing to effectively integrate with the widely used Kimura-Takemoto six-level classification standard. Although there are reports of combining AI diagnosis with the Kimura-Takemoto classification, the classification still relies on endoscopic experts as a reference standard, failing to achieve fully automated classification output. Third, existing systems generally lack the ability to automatically identify anatomical locations within the gastric cavity, unable to provide location-based diagnostic evidence for structured reports. Therefore, based on the above-mentioned challenges, this invention proposes a spatiotemporal feature processing system for endoscopic videos of atrophic gastritis. Summary of the Invention

[0004] To address the aforementioned issues, the present invention aims to provide a spatiotemporal feature processing system for endoscopic videos of atrophic gastritis. By constructing a video analysis architecture that integrates spatial feature extraction and dual-branch temporal-attention parallel modeling, the system performs end-to-end intelligent processing of dynamic video data throughout the entire gastroscopy process. This eliminates subjective experience differences in classification and location description caused by manual interpretation, thereby improving the objectivity and standardization of endoscopic diagnosis of atrophic gastritis.

[0005] To achieve the above objectives, this invention provides a spatiotemporal feature processing system for endoscopic videos of atrophic gastritis. The overall processing flow is as follows: The system first receives a real-time dynamic gastroscopy video stream, divides it into continuous video segments according to a preset time window, and performs quality screening and normalization preprocessing on each frame image; then, the preprocessed image sequence is input into a spatial feature extraction network. This network adopts a multi-scale convolutional architecture and is pre-trained on an endoscopic image dataset to extract spatial domain visual features related to atrophic gastritis in each frame image, generating segment spatial feature vectors for the video segments; subsequently, the segment spatial feature vectors are input in parallel into a dual-branch spatiotemporal feature fusion module, wherein the first branch captures the spatiotemporal feature through a temporal dependency extraction network. The first branch obtains the forward and backward dependencies between frames in the temporal domain to output temporal features. The second branch, through an attention weight allocation network, assigns differentiated weights to image frames containing key clinical signs based on the global correlation between features of each frame to output attention-enhanced features. The features output by the two branches are cascaded and processed by a classification discriminant layer to output the probability distribution of atrophic gastritis severity classification. At the same time, the gastric anatomical location recognition network classifies the current video frame into anatomical locations and outputs the probability distribution of whether it belongs to the fundus, body, angle, or antrum of the stomach. Finally, the multimodal semantic mapping engine receives the classification probability distribution and the location probability distribution, converts them into structured semantic data, and outputs an endoscopic-assisted assessment report according to a preset report template.

[0006] In a first aspect, the present invention provides a spatiotemporal feature processing system for endoscopic videos of atrophic gastritis, comprising: The video stream access module is used to acquire dynamic gastroscopy video streams, divide them into continuous video segments according to preset time windows, and perform standardized preprocessing on each frame image; A spatial feature extraction network, which employs a multi-scale convolutional architecture and is pre-trained on an endoscopic image dataset, is used to extract spatial domain visual features related to atrophic gastritis in each frame of the image and generate segment spatial feature vectors for video clips. The dual-branch spatiotemporal feature fusion module has a first branch, which is a temporal dependency extraction network used to capture the forward and backward temporal dependencies between frames within a video segment to output temporal features. The second branch is an attention weight allocation network used to assign differential weights to image frames containing preset clinical key signs based on the global correlation between frames to output attention-enhanced features. The features output by the dual branches are cascaded and input into a classification and discrimination layer to output the probability distribution of atrophic gastritis severity subtype. A gastric anatomical location recognition network, which is independent of the spatial feature extraction network, is used to classify anatomical locations in the current video frame and output the probability distribution of gastric anatomical locations. A multimodal semantic mapping engine is used to receive the probability distribution of the severity of atrophic gastritis and the probability distribution of the anatomical location of the gastric cavity, convert them into structured semantic data, and output an endoscopic-assisted assessment report according to a preset report template.

[0007] Furthermore, the spatial feature extraction network is configured to extract spatial domain visual features that include at least: The distribution ratio of white and red areas of the mucosa, the visibility of submucosal blood vessels, the texture of granular hyperplasia or nodular changes on the mucosal surface, the color shift of the mucosa caused by bile reflux, the color characteristics of mucosal erythema and exudative changes, the changes in smoothness caused by mucosal swelling, the discontinuous edge of the erosion area, the deviation of the regularity of the fold morphology, the coverage area of ​​white turbid mucus, and the clustered distribution of localized punctate redness.

[0008] Furthermore, the temporal dependency extraction network in the dual-branch spatiotemporal feature fusion module is a bidirectional recurrent neural network. It extracts the temporal dependency relationship of the video segment from the start frame to the end frame through the forward propagation path, and at the same time extracts the temporal dependency relationship from the end frame to the start frame through the backward propagation path. The bidirectional hidden states are concatenated and used as the temporal feature output.

[0009] Furthermore, the attention weight allocation network in the dual-branch spatiotemporal feature fusion module is an encoder based on a self-attention mechanism. It retains the order information of each image frame in the video segment through position encoding, and calculates the inter-frame attention weight matrix using the spatial feature vector of each frame as the query vector and key vector. The attention enhancement feature is output after the weight matrix and value vector are weighted and summed.

[0010] Furthermore, the classification and discrimination layer includes at least a fully connected transformation sublayer and a probability normalization sublayer. The fully connected transformation sublayer performs a nonlinear mapping on the cascaded temporal features and attention-enhanced features. The probability normalization sublayer converts the mapping result into the probability distribution of each atrophy level under the non-atrophy and Kimura-Takemoto classification system. During the training phase, the classification and discrimination layer uses a loss function that differentiates and weights each category to suppress the imbalance of sample category distribution.

[0011] Furthermore, the gastric anatomical location identification network is a lightweight convolutional neural network, whose output layer generates probability distributions for the current video frame belonging to four anatomical locations: the fundus, body, angle, and antrum of the stomach after probability normalization.

[0012] Furthermore, the multimodal semantic mapping engine includes: The location-feature association unit is used to perform majority voting aggregation on the location probability distribution of each frame output by the gastric anatomical location identification network to determine the anatomical location to which the current video segment belongs and the representative frame within the segment, and to extract the spatial feature vector of the representative frame from the spatial feature extraction network as the mucosal feature data corresponding to the location. The rule-driven text generation unit has a pre-set mapping rule library that maps different mucosal feature vector combinations to clinical natural language descriptions. It is used to trigger the corresponding mapping rules based on the input mucosal feature data to generate structured mucosal description text for each part. The report template assembly unit is used to obtain the structured mucosal description text of each part and the classification conclusion corresponding to the highest probability in the probability distribution of the atrophic gastritis severity classification. It fills the structured report template according to the preset examination information fields, mucosal description fields of each part, atrophy judgment fields and auxiliary assessment conclusion fields, and outputs the endoscopic auxiliary assessment report.

[0013] It should be noted that since the dual-branch spatiotemporal feature fusion module outputs global features that integrate the temporal information of the entire video segment, it cannot directly perform spatial segmentation according to anatomical parts. Therefore, this invention uses the single-frame spatial features representing the frame as the carrier of the part-specific mucosal features, thereby avoiding the granularity mismatch problem between the global fusion features and the local part labels, and ensuring that the descriptive text of each part can be traced back to the frame image with the best actual imaging quality for that part.

[0014] Furthermore, when performing quality screening on each frame of the video stream access module, it includes at least the following: The image region with specular reflection interference is identified and marked by a specular reflection region detection algorithm. Image frames with motion blur or focus failure are removed by an image sharpness evaluation function, and the pixel values ​​of the retained image frames are mapped to a preset standard value range.

[0015] Furthermore, both the spatial feature extraction network and the dual-branch spatiotemporal feature fusion module are initialized with pre-trained weights and jointly trained using multiple data augmentation strategies, including geometric transformation enhancement, photometric transformation enhancement, and degradation simulation enhancement, to improve the model's generalization ability in real endoscopic scenarios.

[0016] Furthermore, a cross-scale feature alignment unit is provided before the dual-branch spatiotemporal feature fusion module. The cross-scale feature alignment unit is used to receive the spatial feature vectors of each frame output by the spatial feature extraction network, calculate the difference metric between the feature vectors of adjacent frames, construct a cumulative distance matrix based on the difference metric, map the original unequal length feature sequence to a standard time length by solving the optimal normalization path, and output the normalized feature sequence after temporal consistency normalization, so as to eliminate the interference of inter-frame temporal scale non-uniformity caused by the speed of gastroscopy lens movement and gastric peristalsis on temporal modeling.

[0017] Furthermore, the dual-branch spatiotemporal feature fusion module also includes an atrophic boundary space localization and classification correction submodule, which is used to perform boundary response mapping on the fusion features output by the dual branches, extract the boundary space distribution features between the atrophic mucosa and the normal mucosa and encode them as boundary position features, generate a classification correction factor by jointly associating and mapping the boundary position features with the fusion features, and modulate the atrophic gastritis severity classification probability distribution output by the classification discrimination layer with the classification correction factor, and output the classification probability distribution corrected by the boundary space information.

[0018] Furthermore, the multimodal semantic mapping engine also includes an output credibility assessment and labeling submodule, which is used to perform cognitive uncertainty quantification assessment on the typing probability distribution output by the classification and discrimination layer, and at the same time perform decision consistency analysis on the output typing results of multiple consecutive video segments throughout the gastroscopy examination. The cognitive uncertainty quantification result and the decision consistency analysis result are integrated into a comprehensive credibility score, and the corresponding credibility level label is added to the auxiliary assessment conclusion in the endoscopic auxiliary assessment report according to the comprehensive credibility score.

[0019] Secondly, the present invention also provides a method for spatiotemporal feature processing of endoscopic videos of atrophic gastritis, the method being based on the system described in the first aspect above, comprising: The dynamic gastroscopy video stream is acquired, divided into continuous video segments according to a preset time window, and quality screening and normalization preprocessing are performed on each frame of the video segment. The preprocessed image sequence is input into a spatial feature extraction network to extract spatial domain visual features related to atrophic gastritis in each frame of the image, and generate segment spatial feature vectors for the video clips. The segment spatial feature vectors are input into the temporal dependency extraction network and the attention weight allocation network, respectively. The temporal dependency extraction network captures the forward and backward dependencies between frames within the video segment in the temporal domain to output temporal features. The attention weight allocation network assigns differentiated weights to image frames containing preset clinical key signs based on the global correlation between the image features of each frame to output attention enhancement features. The temporal features and the attention enhancement features are concatenated and then processed by a classification and discrimination layer to output the probability distribution of atrophic gastritis severity subtypes; The preprocessed image sequence is input into the gastric anatomical location recognition network, which outputs the probability distribution of the gastric anatomical location to which the current video frame belongs. The probability distribution of the severity of atrophic gastritis and the probability distribution of the anatomical location of the stomach cavity are input into a multimodal semantic mapping engine, converted into structured semantic data, and an endoscopic-assisted evaluation report is output according to a preset report template.

[0020] This invention achieves real-time parsing and automated diagnostic analysis of dynamic gastroscopy video streams by constructing an end-to-end processing architecture that integrates endoscopy-specific spatial feature extraction, dual-branch parallel spatiotemporal modeling, simultaneous gastric cavity location identification, and multimodal semantic mapping. Specifically, the endoscopy-specific spatial feature extraction network encodes targeted features for various clinical signs related to atrophic gastritis, providing a clinically discriminative feature foundation for subsequent spatiotemporal modeling. The dual-branch spatiotemporal fusion module, through parallel collaboration of temporal dependency extraction and attention weight allocation, captures the forward and backward dependencies of mucosal evolution within video segments and the differentiated contributions of key sign frames, thereby enhancing the feature expression capability for atrophy degree discrimination while fully utilizing dynamic temporal information. The gastric cavity location identification network synchronously outputs anatomical location information, providing location-specific spatial localization for auxiliary assessment conclusions. The multimodal semantic mapping engine uniformly converts heterogeneous image features and location labels into structured text conforming to clinical terminology, enabling automatic generation and standardized output of auxiliary assessment conclusions.

[0021] Beneficial effects By implementing the spatiotemporal feature processing system for endoscopic videos of atrophic gastritis provided by the present invention, the following technical effects are achieved: (1) A spatial feature extraction network specifically designed for endoscopic images of atrophic gastritis was constructed. This network adopts a multi-scale convolutional architecture and is pre-trained on an endoscopic image dataset. This network has significant advantages in the specificity and clinical relevance of feature extraction. Its output fragment spatial feature vectors provide a discriminative feature basis directly related to the clinical diagnosis of atrophic gastritis for the subsequent dual-branch spatiotemporal fusion module, ensuring the accuracy of the entire system's classification from the source.

[0022] (2) A dual-branch spatiotemporal fusion architecture was constructed, in which a temporal dependency extraction network and an attention weight allocation network operate in parallel. The first branch captures the forward and backward dependencies between frames within a video segment in the temporal domain through a bidirectional recurrent neural network to output temporal features. The second branch, through an encoder based on a self-attention mechanism, assigns differentiated weights to image frames containing key clinical signs based on the global correlation between image features of each frame to output attention-enhanced features. The features output by the two branches are cascaded and then input into the classification and discriminant layer for processing. This parallel architecture breaks through the limitation of traditional methods that process temporal modeling and attention modeling sequentially, enabling the temporal evolution of mucosal exposure and the spatial salience of key atrophy sign frames during endoscopic advance and retreat to be captured and fused simultaneously within the same framework. This achieves comprehensive utilization of spatiotemporal information in dynamic gastroscopy videos and significantly enhances the ability of the classification and discriminant layer to distinguish subtle differences between the Kimura-Takemoto classification levels.

[0023] (3) By introducing a dynamic time warping layer before the dual-branch spatiotemporal fusion module, a cumulative distance matrix is ​​constructed based on the feature difference metric of adjacent frames, and the optimal warping path is solved to map the original feature sequences with unequal time scales to a standard time length. This mechanism effectively eliminates the non-uniformity of inter-frame time scales caused by differences in lens advance and retreat speeds and gastric peristalsis during gastroscopy, enabling the temporal dependency extraction network to accurately capture the forward and backward dependencies of mucosal evolution on a unified time axis. At the same time, it enables the attention weight allocation network to more effectively identify key sign frames on the warped time axis, thereby significantly improving the model's adaptability to non-uniform inter-frame change rates in dynamic gastroscopy videos.

[0024] (4) By adding a shrinkage boundary spatial localization and classification correction submodule to the dual-branch spatiotemporal fusion module, the fusion features are mapped to the boundary response to extract the boundary spatial distribution features between the atrophic mucosa and the normal mucosa and encode them as boundary position features. After joint association mapping of the boundary position features and the fusion features, a classification correction factor is generated to modulate the original classification probability output by the classification discriminant layer. This mechanism explicitly embeds the clinical logic of Kimura-Takemoto classification, which "determines the classification level based on the spatial range of the shrinkage boundary," into the decision-making process of the deep learning model, so that the model's judgment criteria are consistent with the clinical diagnostic criteria. It shows significant performance advantages on difficult samples where the shrinkage boundary features are fuzzy and conventional global classification methods are difficult to accurately classify.

[0025] (5) By adding an output credibility assessment and labeling submodule to the multimodal semantic mapping engine, the reliability of the output results is comprehensively evaluated from two dimensions: cognitive uncertainty quantification and cross-segment decision consistency analysis. Cognitive uncertainty quantification measures the degree of confidence in the current segment classification result through a prediction variance measurement model with multiple forward propagations. Decision consistency analysis assesses the stability of the same auxiliary assessment conclusion by summarizing the classification result sequence of multiple consecutive video segments during the full examination process. The two are integrated into a comprehensive credibility score, and the corresponding credibility level label is attached to the auxiliary assessment conclusion in the report. This mechanism enables clinicians to quickly identify high-risk auxiliary assessment conclusions that require key review when reviewing reports, effectively improving the usability of the system output results in clinical decision-making. Attached Figure Description

[0026] To make the above-described spatiotemporal feature processing system for endoscopic video of atrophic gastritis of the present invention more obvious and understandable, the accompanying drawings used in the specific embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This diagram illustrates the system architecture of the present invention. Figure 2 This represents the schematic diagram of a dual-branch spatiotemporal fusion module. Detailed Implementation

[0028] Example 1: This embodiment uses the endoscopic examination process of a specific patient with atrophic gastritis as an example to fully illustrate the technical solution of the present invention.

[0029] In this embodiment, the dynamic gastroscopy video stream originates from conventional clinical gastroscopy equipment, covering four anatomical regions: the fundus, body, angle, and antrum of the stomach. During the examination, the endoscopist scans each region sequentially according to standardized operating procedures, and the video stream is transmitted in real time to the video stream access module of the system of this invention. The system segments the continuous video stream using fixed time windows, with each time window containing a preset number of consecutive images, constituting a video segment. In this embodiment, each video segment contains 16 frames, and overlapping areas can be set between adjacent segments to ensure the continuity of temporal information. For each frame, the system first performs quality screening and normalization preprocessing: image areas with high-light interference are identified and marked using a specular reflection area detection algorithm; image frames with motion blur or focus failure are removed using an image sharpness evaluation function; the image size of the retained valid frames is uniformly scaled to 224×224 pixels, and the pixel values ​​are mapped to a preset standard value range, thereby forming a standardized image sequence.

[0030] Figure 1 The overall architecture of the system used in this embodiment is shown. For example... Figure 1 As shown, the preprocessed, standardized image sequences are first fed into the EF-CNN endoscopic spatial feature extraction network. This network uses Inception V3 as its basic framework, retaining its multi-scale convolution and feature extraction capabilities, replacing only the top fully connected layer to adapt to the atrophic gastritis feature classification task, and employing the ReLU activation function in the hidden layers. The network is pre-trained on a large-scale endoscopic image dataset to enhance its ability to perceive the unique visual features of endoscopic images, accelerate convergence, and reduce overfitting. During training, data augmentation strategies such as random rotation, scaling, and blurring are employed, with cross-entropy as the loss function, and parameters updated using either the Adam or SGD optimizer. For each input image frame, the network extracts spatial domain visual features related to atrophic gastritis in that frame. The extracted features include at least the following categories: the distribution ratio of white and red areas of the mucosa, used to characterize the overall tendency of mucosal color; the visibility of submucosal vessels, used to reflect the degree of vessel exposure after mucosal atrophy and thinning; the texture features of granular hyperplasia or nodular changes on the mucosal surface, used to characterize the morphological changes of uneven mucosal surface; and the mucosal color shift caused by bile reflux, used to identify the effect of refluxed material on mucosal color; Color features of erythema and exudative changes are used to assess the extent and intensity of inflammatory activity; changes in smoothness caused by mucosal swelling are used to reflect the impact of mucosal edema on surface smoothness; discontinuous features at the edges of eroded areas are used to detect the boundary morphology of mucosal defects; deviations from the regularity of fold morphology are used to determine whether folds are thickened, blunted, or disappeared; features of the area covered by white turbid mucus are used to assess the extent of abnormal mucus characteristics; and the clustering distribution features of localized punctate redness are used to identify the clustering pattern of punctate erythema. For each frame in a video segment, the network outputs a high-dimensional spatial feature vector; the feature vectors of all frames within the segment are arranged in chronological order and normalized to form the segment spatial feature vector sequence of the video segment.

[0031] Figure 2 This is a schematic diagram illustrating the internal workings of the dual-branch spatiotemporal feature fusion module in this embodiment. Figure 2 As shown, this module contains two parallel branches: the first branch is a temporal dependency extraction network, employing a bidirectional long short-term memory (BiLSTM) network; the second branch is an attention weight allocation network, employing a Transformer encoder based on a self-attention mechanism. The two branches process the same input independently and are then cascaded at the end.

[0032] In the temporal dependency extraction network, the sequence of segment spatial feature vectors is input into a BiLSTM. This network extracts the temporal dependencies of video segments from the start frame to the end frame through the forward propagation path, and simultaneously extracts the temporal dependencies from the end frame to the start frame through the backward propagation path. The bidirectional hidden states are concatenated and output as the temporal features. This network effectively captures long-term inter-frame dependencies through a gating mechanism. Its core cell state update process is as follows:

[0033] In the formula, For the current moment The cellular state. Output the forget gate (controls the degree to which information from the previous time step is retained). This represents the cell state at the previous moment. The input gate output controls the degree to which new information is incorporated at the current moment. This represents the current state of the candidate cells.

[0034] In the attention weight allocation network, the sequence of spatial feature vectors is first augmented with frame order information through positional encoding to preserve the temporal position between frames. Then, using the spatial feature vectors of each frame as query vectors (Q) and key vectors (K), an inter-frame attention weight matrix is ​​calculated. Each element in this matrix represents the attention intensity of one frame on another, i.e., a global correlation measure between the features of two frames. The attention weights are then normalized using Softmax and weighted summed with the value vector (V) to output the attention-enhanced features. The calculation formula for this process is:

[0035] In the formula, The dimension of the key vector is used to scale the dot product result to stabilize the gradient. During this process, image frames containing key clinical signs of atrophy (such as mucosal pallor, visible blood vessels, granular changes, etc.) are automatically assigned higher attention weights because their features have high global correlation with other frames.

[0036] The aforementioned temporal features and attention-enhancing features are concatenated along the feature dimension to obtain fused spatiotemporal features. These fused spatiotemporal features are then fed into a classification and discriminant layer for processing. The classification and discriminant layer comprises a fully connected transformation sublayer and a Softmax probability normalization sublayer: the fully connected transformation sublayer performs a non-linear mapping on the concatenated features, compressing the feature dimension to the number of categories (7 categories in this embodiment, namely non-atrophic gastritis and Kimura-Takemoto subtypes C1, C2, C3, O1, O2, O3); the Softmax sublayer converts the mapping result into probability distributions for each category, outputting a probability vector for the severity of atrophic gastritis. During the training phase, the classification and discriminant layer employs a weighted cross-entropy loss function to suppress imbalanced sample category distribution.

[0037] In the formula, For category index (0 indicates non-shrinkage, 1~6 correspond to C1, C2, C3, O1, O2, O3 respectively). For the first Preset weights for classes are used to increase the loss contribution of classes with few samples. For the first The actual label of the class, with a value of 0 or 1. The model predicts the first The probability of a class.

[0038] Meanwhile, the preprocessed image sequence is independently fed into a gastric anatomical location recognition network. This network is a lightweight convolutional neural network (CNN). As an optional implementation, this embodiment uses MobileNetV2 as the backbone network. It decomposes standard convolution into depthwise convolution and pointwise convolution through depthwise separable convolution, significantly reducing the number of parameters and computational cost while maintaining feature extraction capabilities, making it suitable for real-time processing of endoscopic videos. The input to this network is a preprocessed single-frame image (size 224×224×3). After progressive downsampling, a global average pooling layer outputs a feature vector, which is finally mapped to four-dimensional space by a fully connected layer and normalized by Softmax. This generates the probability distribution of the current video frame belonging to one of the four anatomical locations: the fundus, body, angle, and antrum of the stomach. The calculation formula is as follows:

[0039] In the formula, It is a four-dimensional probability distribution vector. This is the weight matrix of the fully connected layer. This is the bias term for the fully connected layer. This is the location-related visual feature vector extracted by the network from the current frame.

[0040] For each frame within a video segment, the network independently outputs a four-dimensional site probability vector according to the formula described above. The system arranges the site probability vectors of all frames within the segment in chronological order and uses a majority voting strategy (i.e., using the anatomical site with the highest frequency within the segment as the site to which the segment belongs) to determine the anatomical site label corresponding to the current video segment. Simultaneously, to establish the association between the single-frame site label and subsequent mucosal feature descriptions, the system records the maximum confidence score of each frame within the segment for its assigned site and selects the frame with the highest confidence score as the representative frame of the segment. The spatial feature vector of this representative frame will be used for subsequent site-feature association extraction. The outputs of the two modules are jointly fed into the multimodal semantic mapping engine. This engine performs the following steps: The site-feature association unit performs majority voting aggregation on the site probability distribution output by each frame based on the gastric anatomical site identification network to determine the anatomical site to which the current video segment belongs and the representative frame within the segment. It then extracts the spatial feature vector of the representative frame from the spatial feature extraction network as the mucosal feature data corresponding to the site.

[0041] The rule-driven text generation unit receives the aforementioned mucosal feature data. This unit has a pre-set mapping rule library that maps different combinations of mucosal feature vectors to clinical natural language descriptions. For example, when the feature data of a certain location contains features related to pale mucosa and visible blood vessels, the corresponding rule is triggered, generating the description "pale mucosa, obvious visible blood vessels"; when the feature data also contains granular texture features, a supplementary rule is further triggered, adding the description "showing granular changes"; when the atrophy feature is not obvious, "smooth mucosa, normal color" is output. This process traverses the four locations of the stomach: fundus, body, angle, and antrum, generating their respective structured mucosal description texts.

[0042] The system determines the severity of atrophic gastritis based on the category with the highest probability in the probability distribution of atrophic gastritis severity classification: if the highest probability corresponds to the non-atrophic category, the conclusion is "no obvious atrophy is observed"; if it corresponds to a certain level (C1~O3) in the Kimura-Takemoto classification, it is determined to be that atrophic level, and at the same time, it outputs a qualitative description of the atrophic range (such as "the atrophic range is limited to the antrum" or "the atrophy extends to the lesser curvature of the gastric body") based on the atrophic range (closed type C1-C3 or open type O1-O3).

[0043] The report template assembly unit acquires structured mucosal description text for each site, atrophy assessment conclusions, and patient information (ID, examination date, physician name, etc.), and fills it in according to a preset structured report template. The structure of this report template is shown in the table below: Table 1. Report Template Structure

[0044] In this embodiment, assuming the system outputs a probability indicating atrophic gastritis for a patient model and the atrophy boundary reaches the lesser curvature of the gastric body, the Kimura-Takemoto classification result is determined to be closed type grade 3 (C3), and the auxiliary assessment conclusion is filled in as "atrophic gastritis (C3 type)"; if the system determines that there is no atrophy, the auxiliary assessment conclusion is filled in as "no obvious atrophy seen". After assembling the various parts according to the above template structure, the system outputs a complete endoscopic auxiliary assessment report.

[0045] This embodiment also describes the overall training strategy of the model. Both EF-CNN and the dual-branch spatiotemporal fusion module use pre-trained weights for parameter initialization (EF-CNN uses ImageNet pre-trained weights, while BiLSTM and Transformer use random initialization or related pre-trained weights) to accelerate convergence and reduce overfitting. Multiple data augmentation strategies are employed during training, including geometric transformations (random rotation, scaling, translation), photometric transformations (brightness and contrast adjustment), and degradation simulation (random blurring), to improve the model's generalization ability in real endoscopic scenarios. The optimizer uses Adam or SGD, and the learning rate employs a stepped decay or cosine annealing strategy. The class weights in the weighted cross-entropy loss function of the classification and discriminative layers... The distribution of samples in Kimura-Takemoto subtypes is effectively addressed by setting the distribution based on the inverse proportion of the number of samples in each category in the training set or by using the median frequency balancing method.

[0046] It should be clarified that the system of this invention is a computer-aided image analysis system, and its output results are the probability distribution and structured assessment information of atrophic gastritis subtypes for clinicians' reference. The final diagnosis of atrophic gastritis needs to be determined by the physician in combination with pathological biopsy and the patient's clinical manifestations. This invention does not directly provide a disease diagnosis conclusion.

[0047] Example 2: During dynamic gastroscopy, factors such as the speed of the endoscope's advance and retreat, its rotation angle, and gastric peristalsis cause the video sequence to exhibit a non-uniform rate of change between frames over time. The same anatomical site may be observed at different time scales at different stages of the examination, and the same atrophy sign may show drastically different evolution patterns between frames when the lens moves quickly and when it scans slowly. Traditional fixed-window temporal modeling methods ignore this non-uniform time scale problem, forcibly aligning features from different time scales to a uniform temporal grid. This results in the model being unable to accurately establish the temporal correspondence between frames, thus limiting the accuracy of classification.

[0048] This embodiment proposes a cross-scale spatiotemporal feature alignment mechanism. The core idea is to introduce a dynamic time warping layer before the dual-branch spatiotemporal fusion module. This layer adaptively resamples the inter-frame feature sequences within a video segment, enabling the same shrinkage features from different time scales to achieve feature alignment on a unified warped time axis. Specifically, this mechanism uses the spatial feature vectors of each frame within the video segment as input, calculates the feature change rate between adjacent frames, and dynamically adjusts the time warping path accordingly. This maps the original unequal-length feature sequences to a standard time length, thereby eliminating temporal distortions caused by camera movement speed and stomach peristalsis. This provides a standardized feature sequence with good temporal consistency for subsequent temporal dependency extraction and attention weight allocation networks.

[0049] For each frame of a video segment, after processing by a spatial feature extraction network, the spatial feature vector of that frame is obtained, denoted as the i-th frame. The spatial feature vector of the frame is Arrange the spatial feature vectors of all frames within the segment in chronological order to form the original feature sequence.

[0050] The difference metric between feature vectors of adjacent frames is calculated as a quantitative indicator of the inter-frame change rate. The difference metric is calculated using a weighted combination of the cosine and Euclidean distances between feature vectors, with the weighting coefficients dynamically adjusted based on the magnitude of the feature vectors. For each pair of adjacent frames... Calculate its difference measure. .

[0051] Measured by the difference between frames The reciprocal of the product is used as the weight to construct the cumulative distance matrix of the original feature sequence on the time axis. Each element in the cumulative distance matrix represents the minimum cumulative path cost from the start position of the sequence to the current position.

[0052] Based on the cumulative distance matrix, a dynamic programming algorithm is used to find the optimal regularization path from the start point to the end point of the sequence. The constraints of this path include: the monotonicity constraint of the path, that the time order is not invertible during the regularization process; the continuity constraint of the path, that the regularization step size does not exceed the preset maximum step size; and the boundary constraint of the path, that the start and end points of the path must be aligned with the start and end points of the original sequence.

[0053] Based on the optimal normalization path, the original unequal-length feature sequences are mapped to a standard time length. ( (This is a preset fixed value). The mapping rule is as follows: For each standard time position on the regular path, the original frame feature or its interpolation result corresponding to that position is taken as the feature value of that position. For positions where the path passes through multiple original frames, a weighted average method is used to fuse the features of multiple frames; for original frames not covered by the path, feature values ​​are supplemented by linear interpolation.

[0054] The normalized standard-length feature sequence is used as input to the dual-branch spatiotemporal feature fusion module, and then fed into the temporal dependency extraction network and the attention weight allocation network for subsequent processing. The normalized feature sequence has a consistent scale in the temporal dimension, which enables the BiLSTM network to more accurately capture the forward and backward dependencies between frames, and at the same time enables the Transformer's attention mechanism to more effectively identify key feature frames on the normalized time axis.

[0055] The optimization objective of the regularized path adopted by this mechanism can be expressed as:

[0056] In the formula, The optimal regularization path is the path that minimizes the weighted cost function among all feasible paths from the starting point to the ending point of the original feature sequence. This path defines how to map the original feature sequences with unequal time scales to the standard time axis. For any regularized path from the start to the end of the original feature sequence, the path consists of several ordered regularization steps. composition. It is the set of all feasible regularized paths. The paths must satisfy the monotonicity constraint (the time order is not reversible), the continuity constraint (the regularization step size does not exceed the preset maximum step size), and the boundary constraint (the start and end points are aligned with the original sequence). To regulate the path A normalization step in the original feature sequence represents the step of normalizing the first feature sequence. The first frame in the normalized standard sequence Frames are aligned on the timeline. is the weighting coefficient of the Euclidean distance term, used to control the proportion of the absolute magnitude difference of features in the total cost. For the first feature sequence in the original sequence The spatial feature vector of the frame image is output by the spatial feature extraction network. For the first time on the normalized standard timeline The feature values ​​corresponding to each position are obtained by interpolating or weighting the original frame features of the path mapping. is the weighting coefficient of the cosine distance term, used to control the proportion of feature orientation differences in the total cost. The weighting coefficient for the step size penalty term is used to control the proportion of jump penalty in the total cost. This is a penalty term for the normalization step size, and its value is the absolute value of the difference between the start frame index and the end frame index in the normalization step. It is used to suppress excessive inter-frame jumps.

[0057] This mechanism was compared with a conventional fixed-window temporal modeling method (i.e., without any temporal scaling of the original feature sequence, directly inputting equally spaced sampled video segments into a BiLSTM and Transformer dual-branch network). The experiments were conducted on the same atrophic gastritis endoscopic video dataset, derived from multi-center clinical gastroscopy examinations (using high-definition electronic gastroscopes), containing a total of 5000 video segments (16 frames per segment, covering C1 to O3 subtypes and non-atrophic samples). All segments were independently annotated by three senior endoscopists, and the consensus result was used as the reference standard after consistency testing (Fleiss' Kappa > 0.85). The comparison conditions were as follows: both groups used the same EF-CNN spatial feature extraction network, the same BiLSTM and Transformer dual-branch fusion architecture, and the same classification and discriminant layer. The only difference was that the experimental group introduced a dynamic temporal scaling layer before the dual-branch fusion module for cross-scale feature alignment, while the control group directly input the original frame sequence into the dual-branch network. Both sets of experiments used the same training, validation, and test set ratio (7:1.5:1.5), the same optimizer and learning rate settings, and trained until the loss function converged. The classification accuracy was then evaluated on the test set. Experimental results showed that after introducing the dynamic temporal warping alignment mechanism, the model achieved an overall accuracy of 91.7% on the Kimura-Takemoto six-level classification task, a 4.5% improvement over the control group (87.2% accuracy) without this mechanism. For video clips captured by rapid endoscopic lens movement (samples where the inter-frame feature change rate is more than one standard deviation above the overall mean), the experimental group achieved a classification accuracy of 89.4%, while the control group only achieved 82.6%. These results indicate that this mechanism, by eliminating temporal distortion caused by differences in lens movement speed and aligning shrinkage features at different time scales on a uniformly warped time axis, effectively improves the model's adaptability to non-uniform inter-frame change rates in dynamic gastroscopy videos, especially showing significant performance gains in complex examination scenarios with fast lens movement and short mucosal exposure time.

[0058] Example 3: The core criterion for the Kimura-Takemoto classification is the spatial distribution of the atrophy boundary on the gastric mucosa. This atrophy boundary begins in the antrum and gradually extends towards the lesser curvature, cardia, and greater curvature of the gastric body. The essential difference between grades C1 to O3 lies in the anatomical location reached by the atrophy boundary. However, existing deep learning-based atrophy classification methods typically employ a global image classification strategy, directly inputting the entire image frame or video clip into the network to output classification probabilities. This lacks explicit modeling of the atrophy boundary, the core criterion for classification. This global classification strategy ignores the crucial discriminative clue of the boundary's spatial location, upon which the Kimura-Takemoto classification relies, leading to a discriminative feature learned by the model that deviates from clinical classification logic.

[0059] This embodiment proposes an adaptive region-aware dynamic tracking and classification correction mechanism for atrophic boundaries. The core idea is to add an atrophic boundary localization subnetwork to the spatiotemporal feature fusion classification module. This subnetwork is specifically used to detect the boundary position between atrophic mucosa and normal mucosa in each frame of the image and encode the boundary position information into a spatial location vector. Subsequently, a boundary-classification association mapping module jointly encodes the boundary location vector and the spatiotemporal fusion features, outputting a classification correction factor. Finally, the correction factor is applied to the original classification probability output by the classification layer to obtain the corrected classification result. This mechanism ensures that the model's judgment logic is consistent with the clinical Kimura-Takemoto classification logic, that is, determining the classification level based on the spatial location of the atrophic boundary, thereby improving the interpretability and accuracy of the classification results.

[0060] Before the classification layer of the spatiotemporal feature fusion and typing module, a shrinkage boundary localization subnetwork is added. This subnetwork takes the fused features from the cascaded dual-branch spatiotemporal fusion module as input, employs a lightweight fully convolutional structure, and outputs a boundary response map with the same resolution as the input image. The value of each pixel in the boundary response map represents the probability that the location belongs to the shrinkage-normal mucosal boundary.

[0061] Post-processing is performed on the boundary response map to extract the spatial location information of the shrunken boundary. The post-processing workflow includes: thresholding the boundary response map to retain pixels with probability values ​​higher than a preset threshold; performing connected component analysis on the retained pixels to select the connected component with the largest area as the main boundary region; and extracting the geometric center coordinates and boundary curve length of the main boundary region as feature descriptions of the boundary location.

[0062] The extracted boundary location features (geometric center coordinates and boundary curve length) are encoded into a fixed-length boundary location vector. The encoding method is as follows: the geometric center coordinates are normalized to the image size range, the boundary curve length is normalized to a preset length range, and the two are concatenated and mapped to a boundary location vector of preset dimensions through a fully connected layer.

[0063] A boundary-fractal association mapping module is constructed. This module takes the boundary location vector and spatiotemporal fusion features as input, calculates the influence weight of the boundary location on the classification decision of each category through an attention mechanism, and outputs a classification correction vector. Each dimension of the correction vector corresponds to a classification category (seven categories in total: non-shrinkage and C1 to O3), representing the degree of support of the boundary location information for the classification result.

[0064] The modified genotyping vector is fused with the original genotyping probabilities output by the classification and discriminant layer on an element-wise weighted basis to obtain the modified genotyping probability distribution. The fusion method is as follows: the modified vector is used as a weight coefficient to modulate the original probabilities, and the modulated probabilities are then normalized to ensure that the sum of all probability values ​​is 1.

[0065] The corrected typing probability distribution is used as the final output for subsequent structured report generation. This correction mechanism ensures that the final typing result not only depends on global spatiotemporal features but also explicitly incorporates the spatial location information of the shrinkage boundary, making the model's judgment logic closer to the clinical Kimura-Takemoto typing criteria.

[0066] The boundary-fractal association weight calculation used in this mechanism can be expressed as:

[0067] In the formula, For boundary location information for the first The classification correction weights represent the influence of the spatial location of the shrinkage boundary on determining whether the image frame belongs to the first class. The degree of support for the atrophy-like level. The value ranges from 0 to 6, corresponding to the seven categories of non-atrophy and the Kimura-Takemoto classification: C1, C2, C3, O1, O2, and O3. The non-linear activation function is the Sigmoid function, which compresses the original calculated values ​​of the modified weights to a probability range between 0 and 1, making it easier to fuse with the classification probability later. The normalization factor takes a value of (i.e., the product of the boundary position vector dimension and the spatiotemporal fusion feature dimension) is used to normalize the result of the double summation to a numerical range comparable to the contribution of a single component, avoiding numerical overflow or gradient instability caused by the amplification of the feature dimension. This represents the total dimension of the boundary location vector, which is the total number of components of the boundary location features after encoding and mapping by the fully connected layer. Component indices for the boundary position vector. This represents the total dimension of the spatiotemporal fusion features, i.e., the total number of components in the fusion feature vector output by the dual-branch spatiotemporal feature fusion module. Component indexes for spatiotemporal fusion features. This is a location-feature correlation function used to calculate the first feature of the boundary location vector. The first component and spatiotemporal fusion feature The correlation strength between the components; the function takes a bilinear form. Implementation, in which The corresponding elements in the trainable bilinear weight matrix are used to automatically learn the correlation pattern between boundary locations and fused features through model training. Here is the feature-class mapping function used to compute the first feature of the spatiotemporal fusion feature. The component is related to the first... The contribution of the classification results; this function uses a linear mapping form. Implementation, in which These are the corresponding elements in the trainable weight matrix. The importance of each feature component for different classification categories is automatically learned through model training.

[0068] This mechanism was compared with conventional global image classification methods (i.e., without introducing a shrinkage boundary localization subnetwork, directly outputting the subtyping probability based on dual-branch spatiotemporal fusion features via a fully connected layer). The experiments were conducted on the same endoscopic video dataset of atrophic gastritis. In addition to the Kimura-Takemoto subtyping labels for each video segment, the dataset's annotation information included pixel-level annotations of the atrophic-normal mucosal boundary in each frame by three senior endoscopists, serving as a reference standard for training and evaluating the boundary localization subnetwork. The comparison conditions were as follows: both groups used the same EF-CNN spatial feature extraction network, the same dual-branch spatiotemporal fusion architecture, and the same basic classification discriminant layer. The only difference was that the experimental group added a shrinkage boundary localization subnetwork and a boundary-subtyping association mapping module before the classification discriminant layer, encoding the boundary location information as a correction factor to modulate the original subtyping probability; the control group directly output the subtyping probability via a fully connected layer using fusion features, without introducing any boundary location information. Both groups used the same data partitioning, training strategy, and hyperparameter settings. Experimental results show that after introducing the boundary tracking and correction mechanism, the model achieved an overall accuracy of 93.1% on the Kimura-Takemoto six-level classification task, an improvement of 3.7% compared to the control group's 89.4%. On difficult samples with ambiguous atrophy boundary features (samples where the pixel value difference between the boundary region and the surrounding region in the boundary response map is lower than the overall mean), the experimental group achieved an accuracy of 87.6%, while the control group only achieved 79.3%. These results indicate that this mechanism, by explicitly encoding the spatial location information of the atrophy boundary and using it to correct classification decisions, ensures that the model's judgment logic remains consistent with the core concept of clinical Kimura-Takemoto classification, which determines the classification level based on the spatial range of the atrophy boundary. This significantly improves classification accuracy on samples with ambiguous boundaries and high discrimination difficulty.

[0069] Example 4: In clinical auxiliary diagnostic systems, the reliability assessment of model output is as important as the diagnostic results themselves. Physicians need to understand the extent to which the model is confident in its output in order to decide whether to accept or verify it. Existing deep learning-based medical image analysis systems typically only output classification probabilities as proxy indicators of confidence, but classification probabilities themselves cannot fully reflect the reliability of the model output: on the one hand, classification probabilities may be distorted due to poor model calibration (overconfidence or lack of confidence); on the other hand, for video stream analysis tasks, the classification probability of a single frame or segment cannot reflect the consistency of the model in reaching the same conclusion throughout the examination process.

[0070] This embodiment proposes a dynamic report credibility labeling mechanism for multi-frame collaborative confidence assessment. The core idea is to add a confidence assessment branch to the spatiotemporal feature fusion typing module's output typing probability. This branch comprehensively evaluates the reliability of the output results from two dimensions: first, the model's cognitive uncertainty regarding the typing result of the current video segment (i.e., the model's own confidence in its output); and second, the model's decision consistency regarding the same typing conclusion throughout the entire gastroscopy examination (i.e., the consistency of output results across different video segments). By fusing these two confidence indices, a comprehensive credibility score is generated. Based on this score, the auxiliary assessment conclusions in the structured report are credibly labeled, providing decision-making reference for clinicians.

[0071] In the classification and discrimination layer of the spatiotemporal feature fusion and classification module, a confidence evaluation branch is added. This branch shares the fusion features output by the dual-branch spatiotemporal fusion module with the classification and discrimination layer, but is processed using an independent network structure. The output of the confidence evaluation branch is a cognitive uncertainty score, ranging from 0 to 1. A lower value indicates lower cognitive uncertainty about the model's classification results, while a higher value indicates higher cognitive uncertainty.

[0072] The cognitive uncertainty score is calculated using the Monte Carlo dropout method: during the model inference phase, multiple forward propagations are performed on the same input (with some neurons randomly dropped each time), resulting in multiple classification probability outputs; the variance of these outputs on the probability distribution is calculated as a measure of cognitive uncertainty; the variance values ​​are mapped to the range of 0 to 1 to obtain the cognitive uncertainty score.

[0073] During the entire gastroscopy examination, the system performs typing and classification on multiple consecutive video segments, recording the typing result and corresponding classification probability for each segment. At the end of the examination, the typing result sequence of all segments is summarized, and a decision consistency index is calculated. The decision consistency index includes: the consistency ratio of typing results (the proportion of segments that output the same typing result); and the stability of classification probabilities (the ratio of the mean to the standard deviation of the highest classification probability across all segments).

[0074] A weighted fusion of cognitive uncertainty and decision consistency indicators generates a comprehensive credibility score. The weights of this fusion are dynamically adjusted based on the clinical scenario: when the number of processed video segments is less than a preset threshold, the cognitive uncertainty score has a higher weight, while the decision consistency indicator has a lower weight; as the number of processed segments increases, the weight of the decision consistency indicator gradually increases, eventually reaching a balance between the two. The fused comprehensive credibility score also ranges from 0 to 1, with a higher value indicating higher overall credibility of the output.

[0075] Based on the range of the overall credibility score, the credibility of the auxiliary evaluation conclusions in the structured report is marked: when the overall credibility score is higher than the first preset threshold, it is marked as "high confidence" and output directly; when the overall credibility score is between the first preset threshold and the second preset threshold, it is marked as "medium confidence" and a prompt message is added to the report; when the overall credibility score is lower than the second preset threshold, it is marked as "low confidence, review recommended" and a prompt indicating that manual review is required is highlighted.

[0076] Structured reports with credibility markers are output to clinicians. When reviewing the reports, clinicians can quickly determine which auxiliary assessment conclusions are highly reliable and which require special attention and review based on the credibility markers, thereby more effectively utilizing the system's output to support clinical decision-making.

Claims

1. A spatiotemporal feature processing system for endoscopic videos of atrophic gastritis, characterized in that, include: The video stream access module is used to acquire dynamic gastroscopy video streams, divide them into continuous video segments according to preset time windows, and perform standardized preprocessing on each frame image; A spatial feature extraction network, which employs a multi-scale convolutional architecture and is pre-trained on an endoscopic image dataset, is used to extract spatial domain visual features related to atrophic gastritis in each frame of the image and generate segment spatial feature vectors for video clips. The dual-branch spatiotemporal feature fusion module has a first branch, which is a temporal dependency extraction network used to capture the forward and backward temporal dependencies between frames within a video segment to output temporal features. The second branch is an attention weight allocation network used to assign differential weights to image frames containing preset clinical key signs based on the global correlation between frames to output attention-enhanced features. The features output by the dual branches are cascaded and input into a classification and discrimination layer to output the probability distribution of atrophic gastritis severity subtype. A gastric anatomical location recognition network, which is independent of the spatial feature extraction network, is used to classify anatomical locations in the current video frame and output the probability distribution of gastric anatomical locations. A multimodal semantic mapping engine is used to receive the probability distribution of the severity of atrophic gastritis and the probability distribution of the anatomical location of the gastric cavity, convert them into structured semantic data, and output an endoscopic-assisted assessment report according to a preset report template.

2. The system according to claim 1, characterized in that, The spatial feature extraction network is configured to extract at least the following spatial domain visual features: The distribution ratio of white and red areas of the mucosa, the visibility of submucosal blood vessels, the texture of granular hyperplasia or nodular changes on the mucosal surface, the color shift of the mucosa caused by bile reflux, the color characteristics of mucosal erythema and exudative changes, the changes in smoothness caused by mucosal swelling, the discontinuous edge of the erosion area, the deviation of the regularity of the fold morphology, the coverage area of ​​white turbid mucus, and the clustered distribution of localized punctate redness.

3. The system according to claim 1, characterized in that: The temporal dependency extraction network in the dual-branch spatiotemporal feature fusion module is a bidirectional recurrent neural network. It extracts the temporal dependency relationship of the video segment from the start frame to the end frame through the forward propagation path, and at the same time extracts the temporal dependency relationship from the end frame to the start frame through the backward propagation path. The bidirectional hidden states are concatenated and used as the temporal feature output.

4. The system according to claim 1, characterized in that: The attention weight allocation network in the dual-branch spatiotemporal feature fusion module is an encoder based on a self-attention mechanism. It retains the order information of each image frame in the video segment through position encoding, and calculates the inter-frame attention weight matrix using the spatial feature vector of each frame as the query vector and key vector. The attention enhancement feature is output after the weight matrix and the value vector are weighted and summed.

5. The system according to claim 1, characterized in that: The classification and discrimination layer includes at least a fully connected transformation sublayer and a probability normalization sublayer. The fully connected transformation sublayer performs a nonlinear mapping on the cascaded temporal features and attention enhancement features. The probability normalization sublayer converts the mapping results into probability distributions of each atrophy level under the non-atrophy and Kimura-Takemoto classification system. The classification and discrimination layer adopts a loss function that is differentially weighted for each category during the training phase.

6. The system according to claim 1, characterized in that: The gastric anatomical location identification network is a lightweight convolutional neural network. Its output layer generates probability distributions for the current video frame to belong to four anatomical locations: the fundus, body, angle, and antrum of the stomach, after probability normalization.

7. The system according to claim 1, characterized in that, The multimodal semantic mapping engine includes: The location-feature association unit is used to perform majority voting aggregation on the location probability distribution of each frame output by the gastric anatomical location identification network to determine the anatomical location to which the current video segment belongs and the representative frame within the segment, and to extract the spatial feature vector of the representative frame from the spatial feature extraction network as the mucosal feature data corresponding to the location. The rule-driven text generation unit has a pre-set mapping rule library that maps different mucosal feature vector combinations to clinical natural language descriptions. It is used to trigger the corresponding mapping rules based on the input mucosal feature data to generate structured mucosal description text for each part. The report template assembly unit is used to obtain the structured mucosal description text of each part and the classification conclusion corresponding to the highest probability in the probability distribution of the atrophic gastritis severity classification. It fills the structured report template according to the preset examination information fields, mucosal description fields of each part, atrophy judgment fields and auxiliary assessment conclusion fields, and outputs the endoscopic auxiliary assessment report.

8. The system according to claim 1, characterized in that: The dual-branch spatiotemporal feature fusion module is further provided with a cross-scale feature alignment unit. The cross-scale feature alignment unit is used to receive the spatial feature vectors of each frame output by the spatial feature extraction network, calculate the difference metric between the feature vectors of adjacent frames, construct a cumulative distance matrix based on the difference metric, map the original unequal length feature sequence to the standard time length by solving the optimal normalization path, and output the normalized feature sequence after temporal consistency normalization.

9. The system according to claim 1, characterized in that: The dual-branch spatiotemporal feature fusion module further includes an atrophic boundary space localization and classification correction submodule, which is used to perform boundary response mapping on the fusion features output by the dual branches, extract the boundary space distribution features between atrophic mucosa and normal mucosa and encode them as boundary position features, generate a classification correction factor by jointly associating and mapping the boundary position features with the fusion features, and modulate the atrophic gastritis severity classification probability distribution output by the classification discrimination layer with the classification correction factor, and output the classification probability distribution corrected by boundary space information.

10. A method for spatiotemporal feature processing of endoscopic videos of atrophic gastritis, characterized in that, The method is implemented based on the system described in any one of claims 1 to 9, and includes: The dynamic gastroscopy video stream is acquired, divided into continuous video segments according to a preset time window, and quality screening and normalization preprocessing are performed on each frame of the video segment. The preprocessed image sequence is input into a spatial feature extraction network to extract spatial domain visual features related to atrophic gastritis in each frame of the image, and generate segment spatial feature vectors for the video clips. The segment spatial feature vectors are input into the temporal dependency extraction network and the attention weight allocation network, respectively. The temporal dependency extraction network captures the forward and backward dependencies between frames within the video segment in the temporal domain to output temporal features. The attention weight allocation network assigns differentiated weights to image frames containing preset clinical key signs based on the global correlation between the image features of each frame to output attention enhancement features. The temporal features and the attention enhancement features are concatenated and then processed by a classification and discrimination layer to output the probability distribution of atrophic gastritis severity subtypes; The preprocessed image sequence is input into the gastric anatomical location recognition network, which outputs the probability distribution of the gastric anatomical location to which the current video frame belongs. The probability distribution of the severity of atrophic gastritis and the probability distribution of the anatomical location of the stomach cavity are input into a multimodal semantic mapping engine, converted into structured semantic data, and an endoscopic-assisted evaluation report is output according to a preset report template.

Images