An esophageal lesion image recognition method based on a deep neural network

Abstract

The application belongs to the field of medical image recognition and relates to an esophageal lesion image recognition method based on a deep neural network, which comprises the following steps: step 1, processing actual digestive endoscopy video data to construct training data; step 2, constructing a deep neural network model and training the deep neural network model through the training data to obtain an early esophageal cancer recognition model; step 3, inputting digestive endoscopy video data to be detected into the early esophageal cancer recognition model, and the early esophageal cancer recognition model judging whether there is a suspected esophageal cancer lesion and outputting a lesion judgment result; the lesion judgment result comprising a spatial position and a progression stage of the lesion; the method models the time sequence dynamics of lesion characteristics in the endoscopy recognition process, and combines a graph neural network to depict the feature correlation of early esophageal squamous cell carcinoma under multi-view conditions, so as to improve the recognition accuracy, consistency and robustness of the model in a real clinical application scenario.

Description

Technical Field

[0001] This invention relates to the field of medical image recognition, and specifically discloses a method for recognizing esophageal lesion images based on deep neural networks. Background Technology

[0002] In clinical practice, early-stage esophageal cancer patients often lack specific clinical symptoms and signs, and their endoscopic lesions are subtle and diverse, making the diagnosis and treatment highly dependent on the experience and expertise of the endoscopist. At the same time, my country also faces challenges such as a shortage of highly skilled endoscopists, significant differences in identification capabilities between different regions and medical institutions, and a heavy workload for endoscopic screening. These factors collectively hinder the effective implementation of early diagnosis and treatment of esophageal cancer.

[0003] How to improve the efficiency and accuracy of early esophageal cancer identification under existing medical resources and reduce over-reliance on human experience has become a key technical challenge that urgently needs to be addressed in current clinical practice.

[0004] In recent years, with the improvement of computer performance and the accumulation of large-scale medical data, artificial intelligence (AI)-assisted recognition technology based on deep neural networks has developed rapidly, providing a new technical approach for the endoscopic identification of early esophageal cancer and gradually forming an effective research scheme. Especially in the field of digestive endoscopy, researchers have tried to introduce deep learning methods such as target detection and image classification into endoscopic image analysis to help endoscopists better discover lesions and determine their nature.

[0005] Specifically, Ohmori et al. reported an AI model based on the Single Shot MultiBox Detector, which can automatically detect early esophageal squamous cell carcinoma in various endoscopic modes, including white light endoscopy, narrow-band imaging (NBI), iodine staining, and magnifying endoscopy, with detection performance comparable to that of experienced endoscopists. However, this study primarily relied on high-quality static endoscopic images for model training and validation, without incorporating dynamic endoscopic video data containing complex backgrounds and operational interference. Therefore, it was impossible to evaluate the model's real-time detection capabilities during actual endoscopic examinations.

[0006] Subsequently, Yang et al. proposed a multimodal AI-based scheme for early esophageal squamous cell carcinoma identification, consisting of two sub-networks: a YOLO v3-based detection submodule for lesion detection in non-magnifying endoscopic mode, and a ResNet-based classification submodule for lesion characterization in magnifying endoscopic mode. This model achieves detection accuracy comparable to that of senior endoscopists on static images and can detect lesions in both high-quality and low-quality dynamic videos under ex vivo conditions. However, the system's processing speed is approximately 10 frames per second, which is insufficient to keep pace with the frame rate of actual endoscopic video streams and thus cannot meet the needs of real-time clinical assisted diagnosis.

[0007] Furthermore, Tada and her team conducted systematic research on assessing the invasion depth of early esophageal cancer lesions, successively proposing several deep learning-based artificial intelligence models to distinguish between superficial submucosal (within SM1) and deep submucosal (SM2–3) lesions. Their research, to some extent, validated the feasibility of applying artificial intelligence technology to assess the invasion depth of esophageal cancer. However, these models primarily achieve binary classification based on invasion depth, and their ability to further refine the assessment of lesion invasion degree remains limited, making it difficult to meet the actual clinical needs for refined classification and individualized treatment decisions.

[0008] Although existing research has explored a great deal of research and made some progress in the automatic identification of early esophageal cancer, from the perspective of clinical application, the relevant methods still have obvious shortcomings in terms of practicality, stability and comprehensive diagnostic capabilities, and are still difficult to meet the application needs in real endoscopic examination scenarios.

[0009] On the one hand, from the perspective of feature modeling, endoscopy is essentially a continuous and dynamic observation process. The internal environment of the human digestive tract is complex and variable. During actual examinations, cardiac activity, respiration, and the movement of the endoscope (advancing, retracting, rotating) all affect the endoscopic image, easily introducing interference such as dynamic blurring, artifacts, and partial obscuring of lesions. Existing neural network models are mainly trained and modeled based on high-quality static images stored in the endoscopy system, making it difficult to effectively capture the key features of early esophageal cancer lesions under the aforementioned interference factors, thus affecting the stability and reliability of the model's diagnostic results in real clinical settings.

[0010] On the other hand, early esophageal cancer often lacks obvious morphological features under endoscopy, and the differences between some lesions and normal mucosal tissue are quite subtle. The visual characteristics of the same lesion may vary significantly from different viewing angles. In clinical practice, endoscopists often need to repeatedly advance and withdraw the endoscope and adjust the viewing angle to conduct comprehensive observation of suspicious lesions from multiple perspectives and at multiple times, and make a final judgment based on the information from these multiple views. However, most existing neural network methods analyze single-frame images as independent samples, failing to fully model the correlation between lesions in the time dimension and under multiple viewing conditions. This leads to inconsistent diagnostic results for the same early esophageal squamous cell carcinoma lesion from different perspectives, thus limiting its auxiliary value in clinical decision-making.

[0011] To address the aforementioned issues, this invention proposes a deep neural network-based method for esophageal lesion image recognition. By introducing Neural Memory Ordinary Differential Equations (nmODE), the method models the temporal dynamics of lesion features during endoscopic diagnosis. Furthermore, it combines graph neural networks to characterize the feature associations of early esophageal squamous cell carcinoma under multi-view conditions, thereby improving the model's recognition accuracy, consistency, and robustness in real clinical applications. Summary of the Invention

[0012] The purpose of this invention is to provide an image recognition method for esophageal lesions based on deep neural networks, addressing the problem of how to identify early esophageal cancer lesions using deep neural networks to improve the model's recognition accuracy in real clinical applications. The specific solution is as follows:

[0013] A method for esophageal lesion image recognition based on deep neural networks includes: Step 1, processing actual digestive endoscopy video data to construct training data; Step 2, constructing a deep neural network model and training the deep neural network model using the training data to obtain an early esophageal cancer recognition model; Step 3, inputting the digestive endoscopy video data to be detected into the early esophageal cancer recognition model, which determines whether there are suspected esophageal cancer lesions and outputs the lesion judgment result; the lesion judgment result includes the spatial location and progression stage of the lesion.

[0014] Further, step 1 includes: Step 1.1, acquiring multiple actual digestive endoscopy video data and saving the actual digestive endoscopy video data in a unified video format; Step 1.2, performing anonymization processing on the digestive endoscopy video data in the unified video format; the anonymization processing includes hiding patient identity information and doctor identity information; Step 1.3, labeling the esophageal cancer lesions appearing in each video frame of the anonymized digestive endoscopy video data; the labeling content includes the spatial location information and progression stage information of the esophageal cancer lesions; Step 1.4, dividing the annotated digestive endoscopy video data into a training set, a validation set, and a test set to obtain training data.

[0015] Furthermore, the deep neural network model includes a feature extraction skeleton module, a temporal feature aggregation module, a multi-view feature aggregation module, and a lesion detection module; the feature extraction skeleton module uses a convolutional neural network structure; the temporal feature aggregation module uses a neural network with constant differential equations; the multi-view feature aggregation module uses a graph attention network; and the lesion detection module uses a single-stage target detection head structure.

[0016] Further, step 2 includes: Step 2.1, inputting the digestive endoscopy video data into the feature extraction skeleton module for spatial feature extraction to obtain a spatial feature sequence; Step 2.2, inputting the spatial feature sequence into the temporal feature aggregation module to capture spatiotemporal dynamic features to obtain spatiotemporal features, and processing the spatiotemporal features to obtain an existence classification; Step 2.3, when early esophageal cancer lesions are present, inputting the spatiotemporal features into the multi-view feature aggregation module to model the early esophageal cancer lesions to obtain aggregated features, and performing linear classification on the aggregated features to obtain a severity classification; Step 2.4, inputting the spatial features into the lesion detection module to obtain lesion location detection values.

[0017] Further, step 2.1 includes: processing the digestive endoscopy video data in time sequence to obtain a digestive endoscopy video sequence; the feature extraction skeleton module divides the original image frames in the digestive endoscopy video sequence into multiple grid regions and captures the regional visual features of different grid regions to obtain the spatial feature representation of each frame image; and combining the spatial feature representations of multiple frames in time sequence to obtain a spatial feature sequence.

[0018] Further, step 2.2 includes: aggregating the spatial feature sequence into a global feature vector through a global average pooling operation; the temporal feature aggregation module models the temporal dependencies in the digestive endoscopy video data based on the global feature vector to obtain spatiotemporal features; and inputting the spatiotemporal features into a linear classifier to determine whether there are early esophageal cancer lesions in the current video frame to obtain a presence classification.

[0019] Further, step 2.3 includes: acquiring multiple spatiotemporal features of the target early esophageal cancer lesion; multiple spatiotemporal features refer to the lesion representation of the target early esophageal cancer lesion extracted from multiple observation angles; using the spatiotemporal features of the target early esophageal cancer lesion as graph nodes, establishing connections between each graph node and other graph nodes to obtain an undirected graph; using a graph attention mechanism to aggregate and update the target graph nodes in the undirected graph to obtain the attention score between the target graph node and its neighboring graph nodes; normalizing the attention score of the target graph node to obtain the final attention weight between the target graph node and its neighboring graph nodes; weighting and aggregating the spatiotemporal features of the target graph node's neighboring graph nodes based on the final attention weight to obtain the updated spatiotemporal features of the target graph node; aggregating the updated spatiotemporal features of all graph nodes using average pooling to obtain the spatiotemporal features; inputting the spatiotemporal features into a linear classifier to determine the progression of the target early esophageal cancer lesion and obtain a severity classification.

[0020] Furthermore, the attention score is:

[0021] ;

[0022] in, For nodes For nodes Attention score; This is the attention parameter vector; This is the transpose of the matrix; This is a vector concatenation operation; It is a linear transformation matrix; and These are the spatiotemporal characteristics of the target graph node i and the neighboring graph node j, respectively.

[0023] The final attention weights are:

[0024] ;

[0025] in, Let be the final attention weight between the target graph node i and its neighboring graph nodes; exp is an exponential function. For neighbor graph node variables; For target graph nodes The set of neighboring graph nodes; For the node variables of the target graph; For nodes For nodes Attention score;

[0026] The updated spatiotemporal features are as follows:

[0027] ;

[0028] in, The spatiotemporal features of the target graph node i after update; It is a non-linear activation function; For the node variables in the neighbor graph.

[0029] Further, step 2.4 includes: dividing the spatial feature representation in the spatial feature sequence into multiple grid cells; predefining multiple anchor boxes within each grid cell to obtain candidate bounding boxes; predicting a set of parameter vectors for each grid cell; the parameter vectors include the positional offset of the center point of the lesion bounding box relative to the current grid cell, the width and height of the lesion bounding box, and the confidence level of the presence of early esophageal cancer lesions in the candidate bounding boxes; and filtering all candidate bounding boxes based on the confidence level of early esophageal cancer lesions to obtain the spatial location of the early esophageal cancer lesions and their corresponding confidence levels.

[0030] Furthermore, the spatial characteristics are:

[0031] ;

[0032] in, Let t be the spatial feature representation of the image in frame t; This is the feature extraction skeleton module; For the t-th frame of the digestive endoscopy video sequence;

[0033] The spatiotemporal characteristics are:

[0034] ;

[0035] Where S represents the spatiotemporal features; nmODE() represents the neural ordinary differential equation; y(0) represents the initial value of the system; and F represents the global eigenvector.

[0036] The aggregation feature is:

[0037] ;

[0038] in, For aggregation features; N is the total number of graph nodes; k is the graph node variable; The spatiotemporal characteristics of the updated graph node k;

[0039] Existence is classified as follows:

[0040] ;

[0041] in, For existence classification; softmax() is the normalization exponential function; S represents the existence-learnable weight parameters; S represents the spatiotemporal features. This is an existence bias parameter;

[0042] The severity is classified as follows:

[0043] ;

[0044] in, For severity classification; softmax() is the normalization exponential function; The severity can be learned as a weight parameter; This is an aggregation feature; This is a severity bias term.

[0045] The present invention has the following advantages and beneficial effects:

[0046] This invention effectively models long-term temporal dynamic information during endoscopic examinations, improving the stability and accuracy of early esophageal cancer identification. Addressing the limitation of existing methods that rely primarily on static, high-quality images and struggle to depict continuous dynamic changes during endoscopy, this invention uses continuous endoscopic video as input and introduces a temporal feature aggregation module based on neural memory ordinary differential equations (nmODE) to continuously model long-distance temporal dependencies. This approach fully utilizes complementary information between consecutive frames, effectively suppressing interference caused by motion blur, tissue occlusion, and rapid changes in perspective, significantly reducing the risk of missed diagnoses and misdiagnoses.

[0047] This invention enhances the model's ability to retain long-term historical information by separating "learning neurons" and "memory neurons." It employs an improved neural ordinary differential equation (nmODE) structure, decoupling external input from the memory state and introducing a nmODE architecture. This allows the model to maintain a smaller number of parameters while possessing stronger long-term memory capabilities. Compared to traditional recurrent neural networks or ordinary NODE models, this invention can more stably retain important information from early frames during endoscopic examinations, thereby improving the overall ability to detect occult early lesions.

[0048] This invention introduces a multi-view feature aggregation mechanism to effectively solve the problem of inconsistent recognition results under different perspectives: Addressing the issue of existing neural network models providing inconsistent recognition results for the same lesion under different observation perspectives, this invention further constructs a multi-view feature aggregation module after lesion detection. This module uses a graph attention network to uniformly model lesion features from different times and observation perspectives. This module can explicitly characterize the correlation between features from multiple perspectives and adaptively allocate importance weights for different views through an attention mechanism, thereby obtaining more consistent and reliable recognition results. Attached Figure Description

[0049] Figure 1An exemplary flowchart of an esophageal lesion image recognition method based on a deep neural network provided by the present invention;

[0050] Figure 2 The model architecture diagram of the deep neural network model provided by this invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0052] To address the shortcomings of existing technologies, this invention proposes a deep neural network-based image recognition method for esophageal lesions. This method is based on an improved YOLO architecture and addresses the difficulties in effectively modeling complex temporal dynamic information and multi-view visual features during endoscopic examinations, as well as the lack of consistency in recognition results. The network structure is modularly designed. The model uses continuous endoscopic video frames as input and achieves stable identification and accurate localization of early esophageal squamous cell carcinoma lesions through frame-by-frame spatial feature extraction, temporal dynamic feature modeling, and multi-view feature fusion. Figure 1 As shown, the technical solution adopted in this invention includes the following steps:

[0053] Step 1, Data Acquisition and Labeling: The actual digestive endoscopy video data is processed to construct training data. Digestive endoscopy video data obtained from different endoscopists is recorded and collected using endoscopic equipment. After desensitization processing, early esophageal cancer lesions appearing in the videos are further labeled. The labeling includes at least the spatial location of the lesion in the endoscopic view and its corresponding progression stage information. The spatial location information of the lesion is labeled using bounding boxes, and the progression stage information is labeled using one-hot vectors. This ultimately constitutes the training data used to train the neural network model.

[0054] Step 2, Model Building and Training: Build a deep neural network model for early esophageal cancer identification, and train the deep neural network model with training data to obtain an early esophageal cancer identification model.

[0055] Step 3, Identification and Output: Input the digestive endoscopy video data to be detected into the early esophageal cancer identification model. The early esophageal cancer identification model determines whether there are suspected esophageal cancer lesions and outputs the lesion identification result; the lesion identification result includes the spatial location and progression stage of the lesion. Load the trained deep neural network model and input the digestive endoscopy video signal to be detected into the trained deep neural network model. The model will automatically determine whether there are esophageal cancer lesions in the video signal. If lesions are found, it outputs the corresponding spatial location and progression stage of the lesion. The spatial location of the lesion is output in the form of a bounding box, and the progression stage of the lesion is output in the form of a one-hot vector.

[0056] Specifically, in step 1, the digestive endoscopy video data used to train the deep neural network model is collected and labeled, including the following sub-steps:

[0057] Step 1.1, Video Acquisition Steps: Acquire multiple actual digestive endoscopy video data and save them in a unified video format. Record and collect digestive endoscopy video data obtained by different endoscopists, and save the videos in a unified .mp4 format for subsequent processing and model training.

[0058] Step 1.2, Data Desensitization Step: Desensitize the gastrointestinal endoscopy video data in a uniform video format. Desensitize the collected gastrointestinal endoscopy videos and related files by masking or removing patient and physician identity information, forming the original dataset for model training.

[0059] Step 1.3, Lesion Annotation: Esophageal cancer lesions appearing in each video frame of the desensitized digestive endoscopy video data are annotated. Using video frames as the basic unit, each esophageal cancer lesion appearing in the video is annotated. The annotation includes at least the spatial location information of the lesion in the endoscopic image and the corresponding progression stage information. The spatial location information of the lesion is annotated using bounding boxes, including the coordinates of the lesion's center point and the length and width of the lesion's bounding rectangle. The progression stage information of the lesion is annotated using one-hot vectors. The annotation process is determined by at least three senior endoscopists through collaborative discussion.

[0060] Step 1.4, Dataset partitioning: The labeled digestive endoscopy video dataset is divided into training set, validation set and test set according to a preset ratio to obtain training data for model training, parameter tuning and performance evaluation.

[0061] Specifically, in step 2, the deep neural network model constructed in this invention adopts a modular design, which includes the following four functional sub-modules: feature extraction skeleton module. Time-series feature aggregation module Multi-view feature aggregation module and lesion detection module The feature extraction skeleton module uses a convolutional neural network structure; the temporal feature aggregation module uses a neural network with frequent differential equations; the multi-view feature aggregation module uses a graph attention network; and the lesion detection module uses a single-stage target detection head structure.

[0062] Specifically, in step 2, the specific technical implementation methods of each functional module, as well as the connection relationships and collaborative working mechanisms between the modules, are as follows: Figure 2 As shown, it includes the following:

[0063] Step 2.1 involves inputting the digestive endoscopy video data into the feature extraction skeleton module for spatial feature extraction, resulting in a spatial feature sequence. Considering that digestive endoscopy videos are typically long in duration, and that long-distance temporal dependencies have a significant impact on early esophageal cancer lesion identification, inputting the entire examination video into a neural network model for end-to-end modeling would significantly increase computational complexity and hinder the model's application in real-world clinical scenarios. To address these issues, this invention employs a phased spatiotemporal feature modeling strategy to decouple and model the spatial and temporal information in the endoscopy video.

[0064] The digestive endoscopy video data is processed sequentially to obtain a digestive endoscopy video sequence. Then, a skeleton extraction module is used to extract the skeleton through spatial features. The input video sequence is subjected to frame-by-frame spatial feature extraction, followed by temporal feature aggregation. Modeling the temporal dependencies between consecutive frames improves overall modeling efficiency while ensuring recognition accuracy.

[0065] Let the input digestive endoscopy video sequence be... ,in, Indicates the number of frames contained in the video. and These represent the height and width of each frame, respectively, with 3 channels, corresponding to a color endoscope image. The video sequence is input frame by frame in chronological order into the spatial feature extraction skeleton module. This is to extract the spatial visual features of each frame of the image.

[0066] To effectively characterize the spatial distribution features and local tissue structure information of early esophageal cancer lesions in endoscopic images, the feature extraction skeleton module... Using a convolutional neural network structure consistent with the YOLO series models, the original image frames in the digestive endoscopy video sequence are segmented into multiple ( ( ) grid areas, and simultaneously capture the regional visual features of different grid areas. Specifically, for the first Frame endoscopic image Through the feature extraction skeleton module Obtain the spatial feature representation of each frame of the image. :

[0067] ;

[0068] in, Let t be the spatial feature representation of the image in frame t; This is the feature extraction skeleton module; For the t-th frame of the digestive endoscopy video sequence; , and These represent the resolution of the feature map along the horizontal and vertical spatial dimensions, respectively. The feature dimension corresponding to each spatial grid region. At this point, the... The frame image is divided into the feature space as follows: There are 1 grid region, and each grid region corresponds to 1 A 3D local visual feature vector is used to characterize the texture structure, color distribution, and morphological changes of the corresponding image region. The spatial feature representations of multiple frames are combined in temporal order to obtain a spatial feature sequence.

[0069] Through the above frame-by-frame spatial feature extraction process, local visual cues related to early esophageal cancer lesions in endoscopic images can be effectively encoded while maintaining high spatial resolution, providing a basis for subsequent temporal feature aggregation modules. It provides stable and uniform feature inputs for modeling cross-frame temporal dynamics.

[0070] Step 2.2 involves inputting the spatial feature sequence into the temporal feature aggregation module to capture spatiotemporal dynamic features, obtaining spatiotemporal features, and processing these features to obtain an existence classification. To fully utilize the complementary information between consecutive endoscopic video frames and effectively model long-term dependencies, this invention will extract information from discrete video frames... Extracted spatial feature sequences Input to the time series feature aggregation module This invention aims to further capture key spatiotemporal dynamic features associated with early esophageal squamous cell carcinoma and to make a preliminary judgment on the presence of esophageal cancer lesions in the current video frame. Given the continuous evolutionary characteristics of lesion morphology and perspective changes during endoscopic examination, this invention employs Neural Ordinary Differential Equations (NODEs) to model the continuous spatiotemporal dynamics of esophageal cancer endoscopic identification. NODEs are a continuous modeling framework that introduces ordinary differential equations into traditional neural networks. The evolution of the network state can be explicitly described by differential equations, thereby enabling information propagation in continuous time or spatial domains. Compared to discrete layered neural network structures, NODEs typically have fewer parameters and a smoother modeling process, making them more suitable for esophageal cancer identification tasks that require characterizing long-term dynamic evolution. Furthermore, NODEs model the system's memory capacity by introducing global attractors, enabling the network to retain historical information over longer time scales and avoiding the historical information decay problem that occurs in traditional recursive or convolutional structures during long-sequence modeling.

[0071] Specifically, for Extracted spatial feature sequences based on grid regions This invention first employs a global average pooling operation to aggregate spatial feature sequences into a global feature vector. This is used to represent the overall spatial semantic information of the image frame:

[0072] ;

[0073] in, This is the global feature vector; For global average pooling; It is a spatial feature sequence.

[0074] Subsequently, the aggregated global feature vector is input frame by frame into the temporal feature aggregation module. This study models long-range temporal dependencies in digestive endoscopy video data, ultimately obtaining highly abstract spatiotemporal features. To address the homeomorphism problem that may exist in the modeling process of traditional neural network differential equations, the nmODE architecture is further employed. This architecture uses the initial state of the ordinary differential equation as the model's hyperparameters, while explicitly introducing the input feature sequence as external input into the differential equation. This creates two types of neurons with different functions within the network: learning neurons and memory neurons. The learning process occurs only in the external learning neurons, while the memory neurons in the ODE module are primarily responsible for continuously memorizing and dynamically evolving the temporal features in the input sequence. This design not only makes the model's dynamic behavior clearer but also effectively alleviates the homeomorphism limitation of traditional neural ordinary differential equations on input-output mapping, further enhancing the model's nonlinear modeling capabilities. Similar ideas have been widely applied in state-space models (such as the Mamba model) and have achieved superior overall performance compared to the Transformer architecture in fields such as natural language processing.

[0075] Temporal feature aggregation module The dynamic evolution process can be expressed as the following ordinary differential equation:

[0076] ;

[0077] in, Let represent the time variable in the differential equation, and ; Represents the system state vector; Represents the system state vector in time The derivative; This represents the external input, obtained by aggregating spatial feature sequences; and These represent the learnable temporal aggregation weight parameters and temporal aggregation bias term, respectively; sin is the sine function. Given an initial state... The locus of the solutions to the above differential equation is denoted as . The corresponding global attractor is the unique stable solution under this input condition.

[0078] To obtain the final spatiotemporal feature representation, only numerical solutions to the aforementioned ordinary differential equations are needed. Considering that this equation is difficult to solve directly in analytical form during practical applications, the odeint method provided by the torchdiffeq toolkit is used to numerically integrate the neural network ordinary differential equations during model training and inference. This method can complete forward propagation and backward gradient calculation within a finite number of computational steps, thus ensuring the trainability of the model. The system state obtained from the numerical solution serves as the final spatiotemporal feature output by the nmODE module. :

[0079] ;

[0080] Where S represents the spatiotemporal characteristics; y(0) represents the initial value of the system, which is set to 0 in this invention; For God, the constant differential equation. This refers to the spatiotemporal features extracted from the current frame by the nmODE model.

[0081] Finally, the obtained spatiotemporal features The input is fed into a linear classifier to determine whether early esophageal cancer lesions exist in the current video frame, resulting in a presence classification. The classification result can be expressed as:

[0082] ;

[0083] in, For existence classification, it represents the classification probability result of whether there is an esophageal cancer lesion in the current video frame; It is a normalized exponential function; and Let represent the existential learnable weight parameters and the existential bias parameters in a linear classifier, respectively. When the time frame is set to 0, it indicates that no esophageal cancer lesions were detected in the current frame; conversely, when the time frame is set to 0, it indicates that no esophageal cancer lesions were detected in the current frame. When a lesion is detected, it indicates that an esophageal cancer lesion has been detected in the current frame. Upon detection of a lesion, this invention further stores the spatial visual features corresponding to the current frame and its associated spatiotemporal features in a feature cache for subsequent multi-view feature aggregation processing and precise spatial location of the lesion, thereby providing more comprehensive feature information support for subsequent detection modules.

[0084] Step 2.3: When early esophageal cancer lesions are present, the spatiotemporal features are input into the multi-view feature aggregation module to model the early esophageal cancer lesions, obtaining aggregated features. The aggregated features are then linearly classified to obtain a severity classification. After detecting the presence of esophageal cancer lesions in the current video frame, this invention further introduces the multi-view feature aggregation module. This module is used to comprehensively model the complementary visual features of lesions under different viewing angles, thereby obtaining more stable and consistent recognition results. It uses a graph attention network as its core, explicitly modeling the feature relationships of lesions under multi-view conditions by constructing a graph structure, and adaptively assigning importance weights to features from different viewpoints using an attention mechanism to enhance the model's ability to capture global discriminative information.

[0085] Specifically, multiple spatiotemporal features of the target early esophageal cancer lesion are acquired; these multiple spatiotemporal features refer to the lesion characteristics of the target early esophageal cancer lesion extracted from multiple observation angles. Assume the feature cache module stores a total of... Spatiotemporal feature vectors associated with the same lesion , where each feature vector This represents the lesion characterization extracted from a specific observation perspective. It involves identifying each spatiotemporal feature of the target early-stage esophageal cancer lesion. Consider each node as a graph node, and establish connections between each graph node and other graph nodes to construct an undirected graph. Among them, the node set edge set This represents the connection relationships between nodes. Given that in actual endoscopic examinations, doctors do not use fixed or standardized sections to observe suspicious lesions, and that different doctors have significant differences in operating habits, endoscope angles, and observation sequences, the visual features of the same lesion can vary significantly across different frames. To fully exploit the complementary information between multiple perspectives, this invention employs a fully connected graph construction strategy, establishing connections between each lesion feature node in the feature cache and all other nodes. This allows information exchange between any two perspectives, enhancing the sufficiency and robustness of multi-view feature fusion.

[0086] After the graph structure is constructed, a graph attention mechanism is used to aggregate and update the features of the target graph nodes in the undirected graph, obtaining the attention score between the target graph node and its neighboring graph nodes. Specifically, for any node... Its neighboring nodes Attention scores between individuals are calculated as follows:

[0087] ;

[0088] in, Represents a node For nodes Attention score; This is a learnable attention parameter vector; This is the transpose of the matrix; Represents a linear transformation matrix; and These are the spatiotemporal characteristics of the target graph node i and the neighboring graph node j, respectively. This indicates a vector concatenation operation.

[0089] The target graph nodes are evaluated using the Softmax function. The attention scores of all neighboring graph nodes are normalized to obtain the final attention weights between the target graph node and its neighboring graph nodes:

[0090] ;

[0091] in, Let be the final attention weight between the target graph node i and its neighboring graph nodes; exp is an exponential function. For neighbor graph node variables; For target graph nodes The set of neighboring nodes in a fully connected graph. Including nodes in the target graph All nodes except those mentioned above; For nodes For nodes Attention score.

[0092] Based on the final attention weights, the spatiotemporal features of the target graph node's neighboring graph nodes are weighted and aggregated to obtain the updated spatiotemporal features of the target graph node. The feature representation is updated by weighted aggregation of the features of its neighbor graph nodes:

[0093] ;

[0094] in, The spatiotemporal features of the target graph node i after update; This represents a non-linear activation function used to enhance the non-linear expressive power of the model. By stacking multiple layers of graph attention networks, node features can be fused layer by layer with lesion information from different time points and perspectives, thereby obtaining a multi-view lesion feature representation that includes global contextual information.

[0095] Finally, the spatiotemporal features of all updated graph nodes in the undirected graph are aggregated using average pooling to obtain spatiotemporal features that fuse multi-view information:

[0096] ;

[0097] in, This is an aggregation feature; This represents the spatiotemporal characteristics of graph node k after being updated through a multi-layer graph attention network; N is the total number of graph nodes; k is the graph node variable. This is a global spatiotemporal feature representation of lesions that aggregates information from different times and observation perspectives.

[0098] Subsequently, the spatiotemporal characteristics The data is input into a linear classifier to determine the progression of early esophageal cancer lesions, resulting in a severity classification of the progression of early esophageal cancer lesions.

[0099] ;

[0100] in, The severity classification represents the probability output for early esophageal cancer identification based on multi-view fusion features; It is a normalized exponential function; and These are the learnable severity weights and severity bias term in a linear classifier, respectively. Through the above multi-view feature aggregation and classification process, this invention can effectively reduce the risk of misjudgment caused by a single viewpoint or single-frame image, and significantly improve the consistency and reliability of the model in clinical applications.

[0101] Step 2.4: Input the spatial features into the lesion detection module to obtain the lesion location detection value. After completing the comprehensive judgment of the progression of esophageal cancer lesions, this invention further introduces a lesion detection module. It can accurately locate the spatial position of early esophageal cancer lesions in the current endoscopic images, thereby achieving complete identification of early esophageal cancer.

[0102] Specifically, the lesion detection module It adopts a single-stage object detection head structure consistent with the YOLO model, using the gridded spatial features corresponding to the current video frame. As input, the location of the lesion in the image and its confidence level are jointly predicted. This module divides the feature map corresponding to the spatial features into... Each of the given grid regions is considered a grid cell, and multiple anchor boxes are predefined within each grid cell to cover potential lesion regions of different scales and aspect ratios, resulting in candidate bounding boxes. For the _th ... Each grid cell, the lesion detection module predicts a set of parameter vectors:

[0103] ;

[0104] in, and These represent the offsets of the center point of the lesion bounding box relative to the current grid cell along the horizontal and vertical axes, respectively. and These represent the width and height of the lesion's bounding box, respectively. Represents bounding box The confidence level of the presence of early esophageal cancer lesions is high.

[0105] Based on the confidence level obtained from the prediction All candidate bounding boxes are filtered, and non-maximum suppression (NMS) is used to remove highly overlapping redundant detection results. Finally, the spatial location of early esophageal cancer lesions in the current frame and their corresponding confidence scores are output.

[0106] Through the above lesion detection module This invention can not only accurately determine the presence of early esophageal cancer lesions, but also precisely locate the specific spatial position of the lesions in endoscopic images, thereby providing a reliable basis for subsequent clinical diagnosis and treatment decisions and assessment of the extent of lesion resection.

[0107] Finally, step 3 connects the trained neural network model to a digestive endoscopy machine to accurately identify the spatial location and specific progression of early esophageal cancer lesions in clinical practice, assisting doctors in disease diagnosis.

[0108] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for esophageal lesion image recognition based on deep neural networks, characterized in that, include: Step 1: Process the actual digestive endoscopy video data to construct training data; Step 2: Construct a deep neural network model and train it using training data to obtain an early esophageal cancer identification model. The deep neural network model includes a feature extraction skeleton module, a temporal feature aggregation module, a multi-view feature aggregation module, and a lesion detection module. The feature extraction skeleton module uses a convolutional neural network structure. The temporal feature aggregation module uses a neural network with constant differential equations. A neural network with constant differential equations refers to a neural network that uses constant differential equations. Multi-view feature aggregation module selects graph attention network; The lesion detection module selects a single-stage target detection head structure; Step 2 includes: Step 2.1, inputting the digestive endoscopy video data into the feature extraction skeleton module for spatial feature extraction to obtain a spatial feature sequence; Step 2.2: Input the spatial feature sequence into the temporal feature aggregation module to capture spatiotemporal dynamic features, obtain spatiotemporal features, and process the spatiotemporal features to obtain an existence classification; Step 2.3: When early esophageal cancer lesions exist, input the spatiotemporal features into the multi-view feature aggregation module to model the early esophageal cancer lesions, obtain aggregated features, and perform linear classification on the aggregated features to obtain a severity classification; Step 2.4: Input the spatial features into the lesion detection module to obtain the lesion location detection value; Step 3: Input the video data of the digestive endoscopy examination to be tested into the early esophageal cancer identification model. The early esophageal cancer identification model determines whether there are suspected esophageal cancer lesions and outputs the lesion identification results. The lesion identification results include the spatial location and progression stage of the lesion.

2. The method for esophageal lesion image recognition based on deep neural networks according to claim 1, characterized in that, Step 1 includes: Step 1.1: Acquire multiple actual digestive endoscopy video data and save the actual digestive endoscopy video data in a unified video format; Step 1.2: Desensitize the digestive endoscopy video data in a uniform video format; the desensitization process includes hiding patient and doctor identity information. Step 1.3: Annotate the esophageal cancer lesions appearing in each video frame of the desensitized digestive endoscopy video data; the annotation content includes the spatial location information and progression stage information of the esophageal cancer lesions; Step 1.4: Divide the labeled digestive endoscopy video data into training set, validation set and test set to obtain training data.

3. The method for esophageal lesion image recognition based on deep neural networks according to claim 1, characterized in that, Step 2.1 includes: The digestive endoscopy video data is processed sequentially to obtain a digestive endoscopy video sequence. The feature extraction skeleton module segments the original image frames in the digestive endoscopy video sequence into multiple grid regions and captures the regional visual features of different grid regions to obtain the spatial feature representation of each frame image; The spatial feature representations of multiple frames are combined in time sequence to obtain a spatial feature sequence.

4. The method for esophageal lesion image recognition based on deep neural networks according to claim 3, characterized in that, Step 2.2 includes: The spatial feature sequence is aggregated into a global feature vector through global average pooling. The temporal feature aggregation module models the temporal dependencies in digestive endoscopy video data based on global feature vectors to obtain spatiotemporal features; The spatiotemporal features are input into a linear classifier to determine whether there are early esophageal cancer lesions in the current video frame, thus obtaining a presence classification.

5. The method for esophageal lesion image recognition based on deep neural networks according to claim 3, characterized in that, Step 2.3 includes: Acquire multiple spatiotemporal features of the target early esophageal cancer lesion; multiple spatiotemporal features refer to the lesion characteristics of the target early esophageal cancer lesion extracted from multiple observation angles; The spatiotemporal features of the target early esophageal cancer lesions are used as graph nodes, and each graph node is connected to other graph nodes to obtain an undirected graph. A graph attention mechanism is used to aggregate and update the target graph nodes in the undirected graph to obtain the attention score between the target graph node and its neighboring graph nodes. The attention scores of the target graph nodes are normalized to obtain the final attention weights between the target graph node and its neighboring graph nodes. Based on the final attention weight, the spatiotemporal features of the neighboring graph nodes of the target graph node are weighted and aggregated to obtain the updated spatiotemporal features of the target graph node. The spatiotemporal features are obtained by aggregating the updated spatiotemporal features of all graph nodes using average pooling. The spatiotemporal features are input into a linear classifier to determine the progression of early esophageal cancer lesions and obtain a severity classification.

6. The method for esophageal lesion image recognition based on deep neural networks according to claim 5, characterized in that, Attention score: ； in, For nodes For nodes Attention score; This is the attention parameter vector; This is the transpose of the matrix; This is a vector concatenation operation; It is a linear transformation matrix; and These are the spatiotemporal characteristics of the target graph node i and the neighboring graph node j, respectively. The final attention weights are: ； in, Let be the final attention weight between the target graph node i and its neighboring graph nodes; exp is an exponential function. For neighbor graph node variables; For target graph nodes The set of neighboring graph nodes; For the node variables of the target graph; For nodes For nodes Attention score; The updated spatiotemporal features are as follows: ； in, The spatiotemporal features of the target graph node i after update; It is a non-linear activation function; For the node variables in the neighbor graph.

7. The method for esophageal lesion image recognition based on deep neural networks according to claim 3, characterized in that, Step 2.4 includes: The spatial feature representation in the spatial feature sequence is divided into multiple grid cells; Multiple anchor boxes are predefined within each grid cell to obtain candidate bounding boxes; For each grid cell, a set of parameter vectors is predicted; the parameter vectors include the position offset of the center point of the lesion bounding box relative to the current grid cell, the width and height of the lesion bounding box, and the confidence that there is an early esophageal cancer lesion in the candidate bounding box; Based on the confidence level of early esophageal cancer lesions, all candidate bounding boxes are filtered to obtain the spatial location of early esophageal cancer lesions and their corresponding confidence levels.

8. The esophageal lesion image recognition method based on deep neural networks according to claim 3, characterized in that, spatial... The characteristics are: ； in, Let t be the spatial feature representation of the image in frame t; This is the feature extraction skeleton module; For the t-th frame of the digestive endoscopy video sequence; The spatiotemporal characteristics are: ； Where S represents the spatiotemporal features; nmODE() represents the neural ordinary differential equation; y(0) represents the initial value of the system; and F represents the global eigenvector. The aggregation feature is: ； in, For aggregation features; N is the total number of graph nodes; k is the graph node variable; The spatiotemporal characteristics of the updated graph node k; Existence is classified as follows: ； in, For existence classification; softmax() is the normalization exponential function; S represents the existence-learnable weight parameters; S represents the spatiotemporal features. This is an existence bias parameter; The severity is classified as follows: ； in, For severity classification; softmax() is the normalization exponential function; The severity can be learned as a weight parameter; This is an aggregation feature; This is a severity bias term.

Images