Endoscopic ultrasound-based lesion invasion depth evaluation system and method, and storage medium

Abstract

Disclosed in the present invention is an endoscopic ultrasound-based lesion invasion depth evaluation system, comprising: an image acquisition module for acquiring an endoscopic ultrasound image, a model creation module for creating a convolutional neural network model, and a model training module. The convolutional neural network model comprises: a feature extraction module, used for performing layer-by-layer feature extraction across at least three layers on the endoscopic ultrasound image, so as to obtain encoded feature maps; a feature fusion module, used for performing feature fusion on at least three groups of encoded feature maps, so as to obtain a first fused feature map; a decoder module, used for using an attention mechanism to process the encoded feature maps used in the feature fusion module and the feature map obtained by the feature fusion module, so as to obtain multiple decoded feature maps, and used for performing image segmentation on a finally obtained decoded feature map, so as to obtain a lesion region; and a class prediction module, used for performing classification with respect to the lesion invasion level on the basis of the first fused feature map and the decoded feature maps. The present invention can accurately predict lesion regions and lesion invasion depth types.

Description

Endoscopic ultrasound-based systems, methods, and storage media for assessing lesion invasion depth.

[0001] Cross-references

[0002] This application claims priority to Chinese Patent Application No. 202410772696.3, filed with the Chinese Patent Office no earlier than June 21, 2024, entitled "System, Method and Storage Medium for Assessing Lesion Infiltration Depth Based on Endoscopic Ultrasound", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to the field of lesion image data processing technology, specifically to a lesion invasion depth assessment system, method, and storage medium based on endoscopic ultrasound. Background Technology

[0004] Endoscopic ultrasound (EUS) is an advanced medical imaging technique that combines endoscopic and ultrasound scanning technologies. By mounting a miniature ultrasound probe at the tip of an endoscope, it allows doctors to perform high-resolution ultrasound scans deep within the body, particularly imaging the digestive tract and surrounding organs such as the pancreas, gallbladder, liver, and esophagus. EUS not only provides detailed images of anatomical structures but also assesses the pathological state of tissues, making it extremely valuable clinically for early tumor detection, tumor staging, and guiding precise tissue biopsies.

[0005] Despite the significant advantages of endoscopic ultrasound (EUS) in diagnosis, several challenges remain in the automated segmentation and classification of ultrasound images. First, EUS images are often affected by low contrast, echo heterogeneity, motion artifacts, and noise generated by the instrument itself, all of which reduce image clarity and increase the difficulty of lesion identification. Second, lesions exhibit diverse forms in EUS images, ranging from minute structural changes to large-scale abnormal regions. Different types of lesions display significant differences in image characteristics, posing considerable challenges for both physician identification and automated segmentation and classification based on computer technology.

[0006] Early research on computer-aided image segmentation and classification primarily relied on manual feature extraction, such as edge detection, thresholding, and morphological manipulation. These methods depend on manually defined rules, are poorly adaptable to complex lesion structures, have limited segmentation accuracy, and suffer from extremely inaccurate classification. In recent years, deep learning technologies such as Convolutional Neural Networks (CNNs) and Transformers have demonstrated significant advantages in EUS image analysis, enabling them to automatically learn high-level features and improve the accuracy of segmentation and classification. However, due to the presence of non-lesion areas similar to lesion regions in ultrasound images and blurred lesion boundaries, it is difficult to accurately identify lesions and determine the type of invasion depth, resulting in high false recognition and misclassification rates. Therefore, there is an urgent need to develop a high-accuracy lesion identification system, method, and storage medium based on endoscopic ultrasound for assessing lesion invasion depth. Summary of the Invention

[0007] In view of the above problems, the present invention provides a system, method and storage medium for assessing the depth of lesion invasion based on endoscopic ultrasound. Based on the present invention, the lesion outline can be identified more accurately and the invasion depth type can be determined.

[0008] The technical solution adopted by this invention to solve the technical problem is as follows:

[0009] In a first aspect, the present invention provides a lesion invasion depth assessment system based on endoscopic ultrasound, comprising:

[0010] The image acquisition module is used to acquire endoscopic images from videos captured by an endoscopic ultrasound system.

[0011] The model creation module is used to create a convolutional neural network model. The convolutional neural network model includes a feature extraction module, a feature fusion module, a decoder module, and a category prediction module. The feature extraction module performs at least three layers of layer-by-layer feature extraction on the endoscopic ultrasound image to obtain encoded feature maps. The feature fusion module fuses at least three sets of encoded feature maps to obtain a first fused feature map f. h The decoder module is used to process the encoded feature map used in the feature fusion module and the feature map obtained by the feature fusion module using an attention mechanism to obtain several decoded feature maps, which are then used to perform image segmentation on the finally obtained decoded feature maps to obtain lesion regions. The category prediction module is used to predict the category based on the first fused feature map f. h Classify the lesion invasion level using the decoded feature map;

[0012] The model training module trains and tests the convolutional neural network model based on the dataset, which then serves as a model for assessing the lesion invasion depth based on endoscopic ultrasound. The dataset includes several endoscopic ultrasound images labeled with lesion contours and invasion depth types.

[0013] In a preferred embodiment, the feature extraction module includes five convolutional neural network modules arranged sequentially. These five modules extract features from the endoscopic ultrasound image layer by layer, obtaining a first coded feature map f1, a second coded feature map f2, a third coded feature map f3, a fourth coded feature map f4, and a fifth coded feature map f5. The feature fusion module is used to fuse the fifth coded feature map f5, the fourth coded feature map f4, and the third coded feature map f3 to obtain a first fused feature map f5. h .

[0014] In a preferred embodiment, the feature fusion module is specifically used to process the fifth encoded feature map f5.

[0015] Upsampling is performed on the fifth encoded feature map f5, which is then convolved to obtain f′5. f′5 and the fourth encoded feature map f4 are multiplied element-wise, and the result is then upsampled and convolved again to obtain f′4. f′4 and f′5 are concatenated, and then convolved again to obtain the first aggregated feature map. f′4 and the third encoded feature map f3 are then multiplied element-wise, and the result is convolved to obtain the second aggregated feature map. The first and second aggregated feature maps are then concatenated and convolved again to obtain the first fused feature map f. h .

[0016] In a preferred embodiment, the decoder module includes a first local decoder, a second local decoder, a third local decoder, and a segmentation prediction module. The first local decoder is used to extract features from the initial decoded feature map S6 and the fifth encoded feature map f5 using a first attention module thereon, to obtain the first decoded feature map S5. The initial decoded feature map S6 is based on the global mapping feature map S... m The system obtains a second local decoder, which uses a first attention module to extract features from the first decoded feature map S5 and the fourth encoded feature map f4 to obtain a second decoded feature map S4; a third local decoder, which uses a first attention module to extract features from the second decoded feature map S4 and the third encoded feature map f3 to obtain a third decoded feature map S3; and a segmentation prediction module, which processes the third decoded feature map S3 to obtain a first segmentation prediction map of the lesion region, wherein the processing includes image segmentation.

[0017] In a preferred embodiment, the segmentation prediction module is specifically used to upsample the third decoded feature map S3 to restore it to the size of the ultrasound endoscopic image, activate the upsampled third decoded feature map S3 with a Sigmoid activation function, and then perform image segmentation to obtain a first segmentation prediction map of the lesion area.

[0018] In a preferred embodiment, the first local decoder is specifically used to process the initial decoded feature map S6 and the fifth encoded feature map f5 through the first attention module to obtain the first attention feature R5, and add the first attention feature R5 and the global mapping feature map S6 to obtain the first decoded feature map S5.

[0019] The second local decoder is specifically used to upsample the first decoded feature map S5, and the upsampled first decoded feature map S5 and the fourth encoded feature map f4 are processed by the first attention module to obtain the second attention feature R4. The second attention feature R4 is added to the upsampled first decoded feature map S5 to obtain the second decoded feature map S4.

[0020] The third local decoder is specifically used to upsample the second decoded feature map S4. The upsampled second decoded feature map S4 and the third encoded feature map f3 are processed by the first attention module to obtain the third attention feature R3. The third attention feature R3 and the upsampled second decoded feature map S4 are added to obtain the third decoded feature map S3.

[0021] In a preferred embodiment, the category prediction module is specifically used to multiply the third encoded feature map f3 and the third decoded feature map S3 to obtain the feature map. Multiplying the fourth encoded feature map f4 and the second decoded feature map S4 yields the feature map. Multiplying the fifth encoded feature map f5 and the first decoded feature map S5 yields the feature map. Used for the first fused feature map f h Attention weighting is applied to obtain the first fusion weighted feature map f. a ; used to combine the first fused weighted feature map f a Features after convolution Features after convolution Features after convolution The input is fed into a fully connected layer, and the output of the fully connected layer is processed by the SOFTMAX function to obtain the category prediction function.

[0022] Secondly, the present invention provides a method for real-time assessment of lesion invasion depth based on endoscopic ultrasound, including:

[0023] Acquire real-time video captured by an endoscopic ultrasound.

[0024] Acquire endoscopic ultrasound images from the real-time video;

[0025] The endoscopic ultrasound image is used as input to the endoscopic ultrasound-based lesion invasion depth assessment model obtained in the endoscopic ultrasound-based lesion invasion depth assessment system as described in any of the first aspects.

[0026] The output of the lesion invasion depth assessment model based on endoscopic ultrasound is obtained.

[0027] Thirdly, the present invention provides a method for assessing the depth of lesion invasion based on endoscopic ultrasound, including:

[0028] Acquire endoscopic images from video captured by an endoscopic ultrasound system;

[0029] A convolutional neural network model is created; the convolutional neural network model includes a feature extraction module, a feature fusion module, a decoder module, and a category prediction module. The feature extraction module is used to perform at least three layers of layer-by-layer feature extraction on the endoscopic ultrasound image to obtain an encoded feature map. The feature fusion module is used to fuse at least three sets of the encoded feature maps to obtain a first fused feature map f. h The decoder module is used to process the encoded feature map used in the feature fusion module and the feature map obtained by the feature fusion module using an attention mechanism to obtain several decoded feature maps, which are then used to perform image segmentation on the finally obtained decoded feature maps to obtain lesion regions. The category prediction module is used to predict the category based on the first fused feature map f. h Classify the lesion invasion level using the decoded feature map;

[0030] The convolutional neural network model was trained and tested based on the dataset to obtain a lesion invasion depth assessment model based on endoscopic ultrasound; the dataset includes several endoscopic ultrasound images labeled with lesion contours and invasion depth types.

[0031] Fourthly, the present invention provides a storage medium storing a computer program, which, when executed by a processor, implements the lesion invasion depth assessment method based on endoscopic ultrasound as described in the third aspect.

[0032] This invention relates to a system, method, and storage medium for assessing lesion invasion depth using endoscopic ultrasound. It acquires endoscopic images from video captured by an endoscopic ultrasound camera, performs model construction and training, and generates a pyramid-shaped feature map through layer-by-layer feature extraction using a feature extraction module. A feature fusion module aggregates feature maps of different resolutions to form a unified feature map, which can be used as an initial guiding region for the decoder module and further mined for classification using a category prediction module. The decoder module decodes a portion of the encoded feature map and the feature map obtained from the feature fusion module using an attention mechanism to further enhance the features and accurately estimate the lesion edge and region. The category prediction module further calculates based on the results from the decoder and feature fusion modules to classify the lesion invasion level. Based on this design, compared to existing technologies, this invention achieves more accurate assessment and prediction of lesion region and lesion invasion depth type. This invention solves the problems of inaccurate lesion region identification and inaccurate lesion invasion depth classification under AI-assisted endoscopic ultrasound images, thus improving prediction accuracy. Attached Figure Description

[0033] Figure 1 is a framework diagram of a lesion invasion depth assessment system based on endoscopic ultrasound.

[0034] Figure 2 is a flowchart of the method for real-time assessment of lesion infiltration depth.

[0035] Figure 3 is a flowchart of the method for assessing the depth of lesion infiltration.

[0036] In the diagram, 10 is the image acquisition module; 20 is the model creation module; 21 is the feature extraction module; 22 is the feature fusion module; 23 is the decoder module; 24 is the category prediction module; and 30 is the model training module. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0038] Current endoscopic ultrasound-based lesion assessments mostly use convolutional neural networks. However, in current assessments, there are non-lesion areas in ultrasound images that are similar to the lesion area, or the lesion boundaries are blurred, making it difficult to accurately identify the lesion and determine the depth and type of invasion. The misidentification rate and misclassification rate are both high.

[0039] Therefore, the present invention provides a system, method and storage medium for assessing the depth of lesion invasion based on endoscopic ultrasound, which improves the accuracy of assessing the depth of lesion invasion based on endoscopic ultrasound images, so as to better support medical work.

[0040] The various embodiments of the present invention described below can be executed individually or in combination with each other, and the embodiments of the present invention do not impose specific limitations in this regard. In the embodiments of the present invention, the terms "first," "second," etc., are used only for distinction and do not imply any order or magnitude. The technical solutions of the various embodiments below can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0041] The following describes the lesion invasion depth assessment system, method, and storage medium based on endoscopic ultrasound, as well as its technical effects.

[0042] Figure 1 is a modular schematic diagram of an endoscopic ultrasound-based lesion invasion depth assessment system, which includes:

[0043] Image acquisition module 10 is used to acquire endoscopic images from video captured by an endoscopic ultrasound.

[0044] Model creation module 20 is used to create convolutional neural network models;

[0045] The model training module 30 trains and tests the convolutional neural network model based on the dataset to serve as a lesion invasion depth assessment model based on endoscopic ultrasound; the dataset includes several endoscopic ultrasound images labeled with lesion contours and invasion depth types.

[0046] The convolutional neural network model includes a feature extraction module 21, a feature fusion module 22 (FAM), a decoder module 23, and a category prediction module 24. The feature extraction module 21 is used to perform at least three layers of layer-by-layer feature extraction on the ultrasound endoscopic image to obtain an encoded feature map; the feature fusion module 22 is used to fuse at least three sets of the encoded feature maps to obtain a first fused feature map f. h The decoder module 23 is used to process the encoded feature map used in the feature fusion module 22 and the feature map obtained by the feature fusion module 22 using an attention mechanism to obtain several decoded feature maps, which are then used to perform image segmentation on the finally obtained decoded feature maps to obtain the lesion region; the category prediction module 24 is used to predict the category based on the first fused feature map f. h The lesion invasion level is classified using the decoded feature map.

[0047] Understandably, the feature extraction module 21 includes at least three convolutional neural network (CNN) modules, arranged sequentially. Each CNN module serves as a feature extraction layer, and the output of each CNN module is a set of encoded feature maps. That is, each layer of feature extraction yields a set of encoded feature maps. The encoded feature maps used by the feature fusion module 22 are the same as those required by the decoder module 23. For example, if the feature extraction module 21 has five CNN modules, it will obtain five sets of encoded feature maps, referred to as the first encoded feature map f1, the second encoded feature map f2, the third encoded feature map f3, the fourth encoded feature map f4, and the fifth encoded feature map f5. If the feature fusion module 22 does not use the first encoded feature map f1 but instead uses the second encoded feature map f2, the third encoded feature map f3, the fourth encoded feature map f4, and the fifth encoded feature map f5, then the decoder module 23 will also only use the second encoded feature map f2, the third encoded feature map f3, the fourth encoded feature map f4, and the fifth encoded feature map f5. The feature map obtained by the feature fusion module 22 can be the first fused feature map f. h However, it is not limited to the first fused feature map f h .

[0048] In this embodiment, the lesion invasion depth assessment system based on endoscopic ultrasound also includes:

[0049] The video acquisition module is used to acquire videos captured by an endoscopic ultrasound.

[0050] The dataset acquisition module is used to construct a dataset using endoscopic ultrasound images labeled with contour and infiltration depth types;

[0051] The model training module 30 validates the tested convolutional neural network model based on the dataset, and the validated convolutional neural network model serves as a lesion invasion depth assessment model based on endoscopic ultrasound.

[0052] In this embodiment, the feature extraction module 21 includes five convolutional neural network modules arranged sequentially. Res2Net is used in this embodiment, but it should be understood that other convolutional neural network modules, such as DenseNet, GoogleNet, and SENet, can also be used. The first group of convolutional neural network modules extracts features from the ultrasound endoscopic image through convolution operations to obtain a first encoded feature map f1. The second group of convolutional neural network modules extracts features from the first encoded feature map f1 through convolution operations to obtain a second encoded feature map f2. The third group of convolutional neural network modules extracts features from the second encoded feature map f2 through convolution operations to obtain a third encoded feature map f3. The fourth group of convolutional neural network modules extracts features from the third encoded feature map f3 through convolution operations to obtain a fourth encoded feature map f4. The fifth group of convolutional neural network modules extracts features from the fourth encoded feature map f4 through convolution operations to obtain a fifth encoded feature map f5.

[0053] Specifically, feature extraction module 21 can extract features with a resolution of [h / 2] k w / 2 k The 5-level feature map {f] i The first coded feature map f1 to the fifth coded feature map f5 are corresponding to the first coded feature map f1 to the fifth coded feature map f5, where h and w represent the height and width of the ultrasound endoscopic image, respectively. In this embodiment, k = i, but it is not limited to k = i. k can be set according to the dimensionality reduction process of feature extraction.

[0054] In this embodiment, the feature fusion module 22, the decoder module 23 and the category prediction module 24 all utilize the third coding feature map f3, the fourth coding feature map f4 and the fifth coding feature map f5, i.e., i takes the values ​​3, 4 and 5.

[0055] In this embodiment, the feature fusion module 22 is used to perform feature fusion on the fifth coded feature map f5, the fourth coded feature map f4, and the third coded feature map f3 to obtain the first fused feature map f5. h .

[0056] Specifically, the feature fusion module 22 is used to perform upsampling on the fifth encoded feature map f5, convolve the upsampled fifth encoded feature map f5 to obtain map f′5, multiply map f′5 and the fourth encoded feature map f4 element-wise, and then perform upsampling and convolution on the result of the element-wise multiplication to obtain map f′4, concatenate map f′4 and map f′5, and then perform convolution on the concatenated map to obtain the first aggregated feature map; multiply map f′4 and the third encoded feature map f3 element-wise, and then convolve the result to obtain the second aggregated feature map, and then concatenate the first aggregated feature map and the second aggregated feature map to obtain the first fused feature map f. h .

[0057] Here, based on the first fusion feature map f h The first aggregated feature map and / or the second aggregated feature map can yield the global mapping feature map S. m Global mapping feature map S m This is a mapping of lesion features. As an example rather than a limitation, the global mapping feature map S... m Based on the first aggregated feature map, the second aggregated feature map, and / or the first fused feature map f h The method of acquisition is as follows: the mapping position is determined based on the lesion location information and the invasion depth type information, and the feature map (first fused feature map f) is obtained based on the mapping position and the feature map obtained by the feature fusion module 22. h The global mapping feature map S is obtained by taking at least one of the first aggregated feature map and the second aggregated feature map. m .

[0058] In this embodiment, the decoder module 23 includes a first local decoder, a second local decoder, a third local decoder, and a segmentation prediction module.

[0059] The first local decoder, the second local decoder, and the third local decoder all include a first attention module. The first attention module receives two sets of feature maps (Figure f). i And Figure S i+1 Input, using attention weights A i It is obtained sequentially through a parallel-serial structure. Where P(·) denotes the upsampling operation, and σ(·) is the Sigmoid activation function. It is a transpose operation that subtracts the input from a matrix E consisting entirely of 1s, i.e. This represents subtracting σ(P(S) from the all-one matrix E. i+1 The operation of S) i+1 Or the S below i Both represent decoded feature maps, and the S initially used by decoder module 23 i+1 From the global mapping feature map S m S6 is obtained and is called the initial decoded feature map. The initial decoded feature map S6 is based on the global mapped feature map S. m As an example, S6 is the downsampled global mapping feature map S. m The next S i+1 From the previous S i+1 Attention-related calculations are performed, meaning S5 is obtained by performing correlation calculations on S6, S4 is obtained by performing correlation calculations on S5, and S3 is obtained by performing correlation calculations on S4. This is for the partially encoded feature map f. i Multiply by attention weight A respectively iThe attention features corresponding to the output are obtained after passing through the convolutional layer Conv(·) in the first attention module. in, This indicates element-wise multiplication.

[0060] The first local decoder is used to apply the first attention module in the first local decoder to the global mapped feature map S. m Feature extraction is performed on the fifth encoded feature map f5 to obtain the first decoded feature map S5; specifically, the global mapping feature map S... m Downsampling is performed to obtain the initial decoded feature map S6. The initial decoded feature map S6 and the fifth encoded feature map f5 are passed through the first attention module to obtain the first attention feature R5. The first attention feature R5 is added to the initial decoded feature map S6 to obtain the first decoded feature map S5. This indicates element-wise addition, where i = 5. The second local decoder uses the first attention module within it to extract features from the first decoded feature map S5 and the fourth encoded feature map f4, obtaining the second decoded feature map S4. Specifically, the first decoded feature map S5 is upsampled, and the upsampled first decoded feature map S5 and the fourth encoded feature map f4 are processed by the first attention module to obtain the second attention feature R4. The second attention feature R4 is then added to the upsampled first decoded feature map S5. At this point, i = 4), resulting in the second decoded feature map S4. The third local decoder, using the first attention module, extracts features from the second decoded feature map S4 and the third encoded feature map f3 to obtain the third decoded feature map S3. Specifically, the second decoded feature map S4 is upsampled, and the upsampled second decoded feature map S4 and the third encoded feature map f3 are processed by the first attention module to obtain the third attention feature R3. The third attention feature R3 is then added to the upsampled second decoded feature map S4. At this point, i = 3), the third decoded feature map S3 is obtained.

[0061] The segmentation prediction module is used to process the third decoded feature map S3 to obtain a first segmentation prediction map of the lesion region. The processing includes image segmentation. Specifically, the segmentation prediction module is used to upsample the third decoded feature map S3 to restore it to the size of the input image (endoscopic ultrasound image). After passing through the Sigmoid activation function, image segmentation is performed to further clarify the lesion region, thus obtaining the lesion region. The lesion region and the third decoded feature map S3 are segmented to obtain the first segmentation prediction map. The first segmentation prediction map shows the lesion region and is called the first segmentation prediction map of the lesion region. For example, the contrast between the lesion region and other regions is enhanced in the first segmentation prediction map, such as by displaying the outline of the lesion region with special colors and / or lines, without limitation on the method.

[0062] Here, the first attention module obtains attention weights through activation functions and transpose operations, and further enhances features through element-wise multiplication and convolution operations. The attention-driven erasure mechanism can ultimately refine inaccurate and coarse estimates into accurate and complete edge prediction maps, thereby establishing the relationship between regions and boundary cues.

[0063] In this embodiment, the category prediction module 24 uses the third decoded feature map S3, the second decoded feature map S4, and the first decoded feature map S5 obtained by the decoder module 23 as attention weights to filter out features. Filtered features Used for classifying the depth of lesion infiltration.

[0064] Specifically:

[0065] The category prediction module 24 is used to multiply the third encoded feature map f3 and the third decoded feature map S3 to obtain the feature map. Multiply the fourth encoded feature map f4 and the second decoded feature map S4 to obtain the feature map. Multiply the fifth encoded feature map f5 and the first decoded feature map S5 to obtain the feature map.

[0066] The category prediction module 24 is used to process the first fused feature map f h Attention SA(·) weighting is performed to obtain the first fusion weighted feature map f. a =SA(f h Specifically, the first fused feature map f h After processing by the average pooling layer, the first fused feature map f h After processing by the max pooling layer, the outputs of the average pooling layer and the max pooling layer are concatenated. The concatenated result is then sequentially convolved and activated. The resulting image is then combined with the first fused feature map f. h Multiply, the result of the multiplication and the first fused feature map fh Adding them together yields the first fused weighted feature map f. a .

[0067] The category prediction module 24 is used to convert the first fused weighted feature map f a Features after convolution Features after convolution Features after convolution After passing through the fully connected layer, the output of the fully connected layer is processed by the normalized exponential function (SOFTMAX function) to obtain the category prediction function. The category prediction function can be used to perform classification and obtain the category prediction result.

[0068] The category prediction module 24 includes a second attention module, which is based on the first fused feature map f. h The features obtained by max pooling and average pooling are concatenated together, and then an attention map is obtained through convolution and activation functions. The features selected by the second attention module are obtained through element-wise multiplication and addition of residuals, providing the necessary information for the classification task.

[0069] Referring to Figure 2, this invention provides a method for real-time assessment of lesion invasion depth based on endoscopic ultrasound, comprising:

[0070] S11. Acquire real-time video captured by the endoscopic ultrasound.

[0071] S12. Read the real-time video to obtain the ultrasound endoscopic image in the real-time video;

[0072] S13. Use the endoscopic ultrasound image as input to the lesion invasion depth assessment model based on endoscopic ultrasound obtained by the model training module 30 above.

[0073] S14. Obtain the output of the lesion invasion depth assessment model based on endoscopic ultrasound.

[0074] The output of the lesion invasion depth assessment model based on endoscopic ultrasound is the lesion contour and invasion depth type.

[0075] The method for obtaining lesion contour and invasion depth type using the lesion invasion depth assessment model in S13 includes:

[0076] S131. (Through 5 sets of convolutional neural network modules) Perform layer-by-layer feature extraction on the ultrasound endoscopic image to obtain the first coded feature map f1, the second coded feature map f2, the third coded feature map f3, the fourth coded feature map f4, and the fifth coded feature map f5 in sequence;

[0077] S132. Perform feature fusion on the fifth coding feature map f5, the fourth coding feature map f4, and the third coding feature map f3 to obtain the first fused feature map f.h Specifically, the fifth encoded feature map f5 is upsampled, and then convolved to obtain image f′5. Image f′5 is then multiplied element-wise with the fourth encoded feature map f4, and the result is upsampled and convolved again to obtain image f′4. Image f′4 and image f′5 are concatenated, and then convolved again to obtain the first aggregated feature map. Image f′4 and the third encoded feature map f3 are then multiplied element-wise, and the result is convolved to obtain the second aggregated feature map. The first aggregated feature map and the second aggregated feature map are then concatenated and convolved again to obtain the first fused feature map f. h ;

[0078] S133. The initial decoded feature map S6 and the fifth encoded feature map f5 are processed by the first attention module to obtain the first attention feature R5. The first attention feature R5 is added to the global mapping feature map S6 to obtain the first decoded feature map S5. The first decoded feature map S5 is upsampled. The upsampled first decoded feature map S5 and the fourth encoded feature map f4 are processed by the first attention module to obtain the second attention feature R4. The second attention feature R4 is added to the upsampled first decoded feature map S5 to obtain the second decoded feature map S4. The second decoded feature map S4 is upsampled. The upsampled second decoded feature map S4 and the third encoded feature map f3 are processed by the first attention module to obtain the third attention feature R3. The third attention feature R3 is added to the upsampled second decoded feature map S4 to obtain the third decoded feature map S3.

[0079] S134. Upsample the third decoding feature map S3 to restore it to the size of the ultrasound endoscopic image. Activate the upsampled third decoding feature map S3 with the Sigmoid activation function and then perform image segmentation to obtain the first segmentation prediction map of the lesion area. Compared with the ultrasound endoscopic image, the lesion area is easier to identify in the first segmentation prediction map.

[0080] S135. Multiply the third encoded feature map f3 and the third decoded feature map S3 to obtain the feature map. Multiplying the fourth encoded feature map f4 and the second decoded feature map S4 yields the feature map. Multiplying the fifth encoded feature map f5 and the first decoded feature map S5 yields the feature map.

[0081] S136, For the first fused feature map f h We perform attention (second attention module) weighting to obtain the first fusion weighted feature map f. a ;

[0082] S137, The first fused weighted feature map f a Features after convolution Features after convolution Features after convolution The input is fed into a fully connected layer, and the output of the fully connected layer is processed by the SOFTMAX function to obtain the category prediction function.

[0083] Referring to Figure 3, this invention provides a method for assessing lesion invasion depth based on endoscopic ultrasound, comprising:

[0084] S21. Obtain endoscopic images from the video acquired by the endoscopic ultrasound.

[0085] S22. Create a convolutional neural network model; the convolutional neural network model includes a feature extraction module 21, a feature fusion module 22, a decoder module 23, and a category prediction module 24. The feature extraction module 21 is used to perform at least three layers of layer-by-layer feature extraction on the ultrasound endoscopic image to obtain coded feature maps (at least three sets, with each layer of feature extraction corresponding to one set of coded feature maps); the feature fusion module 22 is used to fuse at least three sets of coded feature maps to obtain a first fused feature map f. h The decoder module 23 is used to process the encoded feature maps used in the feature fusion module 22 and the feature maps obtained by the feature fusion module 22 using an attention mechanism to obtain several decoded feature maps. These decoded feature maps are then used to perform image segmentation to obtain the lesion region. The category prediction module 24 is used to predict the lesion region based on the first fused feature map f. h Classify the lesion invasion level using the decoded feature map;

[0086] S23. The convolutional neural network model is trained and tested based on the dataset to obtain a lesion invasion depth assessment model based on endoscopic ultrasound; the dataset includes several endoscopic ultrasound images labeled with lesion contours and invasion depth types.

[0087] Specifically, it also includes the process of obtaining a dataset by annotating lesion contours and invasion depth types in endoscopic ultrasound images.

[0088] Specifically, it also includes the step of validating the lesion invasion depth assessment model based on endoscopic ultrasound obtained from training and testing. After the validation, the lesion invasion depth assessment model based on endoscopic ultrasound is obtained.

[0089] In specific implementation, the lesion invasion depth assessment method based on endoscopic ultrasound can refer to the specific implementation of the lesion invasion depth assessment system based on endoscopic ultrasound in any of the above embodiments, and the specific implementation steps will not be repeated.

[0090] The present invention also provides a storage medium storing a computer program, the storage medium including instructions, which, when executed on a computer, cause the computer to perform each step of the lesion invasion depth assessment method based on endoscopic ultrasound described in the above embodiments, thereby realizing the lesion invasion depth assessment method based on endoscopic ultrasound.

[0091] The present invention also provides a computer program product, including a computer program or instructions, characterized in that, when the computer program or instructions are executed by a processor, the method for assessing the depth of lesion invasion based on endoscopic ultrasound is implemented.

[0092] The advantages of the endoscopic ultrasound-based lesion invasion depth assessment system, method, and storage medium of this invention are as follows:

[0093] This invention acquires endoscopic ultrasound images from video captured by an endoscopic ultrasound system, performs model construction and training, and generates a pyramid-shaped feature map through layer-by-layer feature extraction by a feature extraction module 21. A feature fusion module 22 aggregates feature maps of different resolutions to form a unified feature map, which can be used as the initial guiding region for the decoder module 23 and further mined for classification by a category prediction module 24. The decoder module 23 decodes part of the encoded feature map and the feature map obtained by the feature fusion module 22 using an attention mechanism to further enhance the features and accurately estimate the lesion edge and region. The category prediction module 24 further calculates based on the results of the decoder module 23 and the feature fusion module 22 to classify the lesion invasion level. Based on this design, this invention achieves more accurate assessment and prediction of lesion region and lesion invasion depth type. This invention solves the problems of inaccurate lesion region identification and inaccurate lesion invasion depth classification under artificial intelligence-assisted endoscopic ultrasound images.

[0094] It should be emphasized that the present invention proposes a general solution, and the application scenarios of the present invention include but are not limited to the determination of the depth of lesion invasion. According to the present invention, it can be extended to the determination of tumor nature, tumor imaging classification, etc.

[0095] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. Parts not described in detail in a certain embodiment can be found in the relevant descriptions of other embodiments. All aspects of this invention not described in detail are based on existing technology.

[0096] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A lesion invasion depth assessment system based on endoscopic ultrasound, characterized in that, include: The image acquisition module is used to acquire endoscopic images from videos captured by an endoscopic ultrasound system. The model creation module is used to create a convolutional neural network model. The convolutional neural network model includes a feature extraction module, a feature fusion module, a decoder module, and a category prediction module. The feature extraction module performs at least three layers of layer-by-layer feature extraction on the endoscopic ultrasound image to obtain encoded feature maps. The feature fusion module fuses at least three sets of encoded feature maps to obtain a first fused feature map f. h The decoder module is used to process the encoded feature map used in the feature fusion module and the feature map obtained by the feature fusion module using an attention mechanism to obtain several decoded feature maps, which are then used to perform image segmentation on the finally obtained decoded feature maps to obtain lesion regions. The category prediction module is used to predict the category based on the first fused feature map f. h Classify the lesion invasion level using the decoded feature map; The model training module trains and tests the convolutional neural network model based on the dataset, which then serves as a model for assessing the lesion invasion depth based on endoscopic ultrasound. The dataset includes several endoscopic ultrasound images labeled with lesion contours and invasion depth types.

2. The lesion invasion depth assessment system based on endoscopic ultrasound as described in claim 1, characterized in that, The feature extraction module includes five sequentially arranged convolutional neural network modules. These five modules extract features from the endoscopic ultrasound image layer by layer, sequentially obtaining a first coded feature map f1, a second coded feature map f2, a third coded feature map f3, a fourth coded feature map f4, and a fifth coded feature map f5. The feature fusion module is used to fuse the fifth coded feature map f5, the fourth coded feature map f4, and the third coded feature map f3 to obtain a first fused feature map f5. h .

3. The lesion invasion depth assessment system based on endoscopic ultrasound as described in claim 2, characterized in that, The feature fusion module is specifically used to perform upsampling on the fifth encoded feature map f5, convolve the upsampled fifth encoded feature map f5 to obtain map f′5, multiply map f′5 element-wise with the fourth encoded feature map f4, and then perform upsampling and convolution on the result of the element-wise multiplication to obtain map f′4. Map f′4 and map f′5 are concatenated, and then convolved to obtain the first aggregated feature map. Map f′4 and the third encoded feature map f3 are multiplied element-wise, and then convolved to obtain the second aggregated feature map. The first aggregated feature map and the second aggregated feature map are concatenated and then convolved to obtain the first fused feature map f. h .

4. The lesion invasion depth assessment system based on endoscopic ultrasound as described in claim 3, characterized in that, The decoder module includes a first local decoder, a second local decoder, a third local decoder, and a segmentation prediction module. The first local decoder uses its first attention module to extract features from the initial decoded feature map S6 and the fifth encoded feature map f5 to obtain the first decoded feature map S5. The initial decoded feature map S6 is based on the global mapping feature map S. m The system obtains a second local decoder, which uses a first attention module to extract features from the first decoded feature map S5 and the fourth encoded feature map f4 to obtain a second decoded feature map S4; a third local decoder, which uses a first attention module to extract features from the second decoded feature map S4 and the third encoded feature map f3 to obtain a third decoded feature map S3; and a segmentation prediction module, which processes the third decoded feature map S3 to obtain a first segmentation prediction map of the lesion region, wherein the processing includes image segmentation.

5. The lesion invasion depth assessment system based on endoscopic ultrasound as described in claim 4, characterized in that, The segmentation prediction module is specifically used to upsample the third decoded feature map S3 to restore it to the size of the ultrasound endoscopic image. The upsampled third decoded feature map S3 is activated by the Sigmoid activation function and then the image segmentation is performed to obtain the first segmentation prediction map of the lesion area.

6. The lesion invasion depth assessment system based on endoscopic ultrasound as described in claim 4, characterized in that, The first local decoder is specifically used to process the initial decoded feature map S6 and the fifth encoded feature map f5 through the first attention module to obtain the first attention feature R5, and add the first attention feature R5 and the global mapping feature map S6 to obtain the first decoded feature map S5; The second local decoder is specifically used to upsample the first decoded feature map S5, and the upsampled first decoded feature map S5 and the fourth encoded feature map f4 are processed by the first attention module to obtain the second attention feature R4. The second attention feature R4 is added to the upsampled first decoded feature map S5 to obtain the second decoded feature map S4. The third local decoder is specifically used to upsample the second decoded feature map S4. The upsampled second decoded feature map S4 and the third encoded feature map f3 are processed by the first attention module to obtain the third attention feature R3. The third attention feature R3 and the upsampled second decoded feature map S4 are added to obtain the third decoded feature map S3.

7. The lesion invasion depth assessment system based on endoscopic ultrasound as described in claim 4, characterized in that, The category prediction module is specifically used to multiply the third encoded feature map f3 and the third decoded feature map S3 to obtain the feature map. Multiplying the fourth encoded feature map f4 and the second decoded feature map S4 yields the feature map. Multiplying the fifth encoded feature map f5 and the first decoded feature map S5 yields the feature map. Used for the first fused feature map f h Attention weighting is applied to obtain the first fusion weighted feature map f. a ; used to combine the first fused weighted feature map f a Features after convolution Features after convolution Features after convolution The input is fed into a fully connected layer, and the output of the fully connected layer is processed by the SOFTMAX function to obtain the category prediction function.

8. A method for real-time assessment of lesion invasion depth based on endoscopic ultrasound, characterized in that, include: Acquire real-time video captured by an endoscopic ultrasound. Acquire endoscopic ultrasound images from the real-time video; The endoscopic ultrasound image is used as input to the endoscopic ultrasound-based lesion invasion depth assessment model obtained in the endoscopic ultrasound-based lesion invasion depth assessment system as described in any one of claims 1 to 7. The output of the lesion invasion depth assessment model based on endoscopic ultrasound is obtained.

9. A method for assessing lesion invasion depth based on endoscopic ultrasound, characterized in that, include: Acquire endoscopic images from video captured by an endoscopic ultrasound system; A convolutional neural network model is created; the convolutional neural network model includes a feature extraction module, a feature fusion module, a decoder module, and a category prediction module. The feature extraction module is used to perform at least three layers of layer-by-layer feature extraction on the endoscopic ultrasound image to obtain an encoded feature map. The feature fusion module is used to fuse at least three sets of the encoded feature maps to obtain a first fused feature map f. h The decoder module is used to process the encoded feature map used in the feature fusion module and the feature map obtained by the feature fusion module using an attention mechanism to obtain several decoded feature maps, which are then used to perform image segmentation on the finally obtained decoded feature maps to obtain lesion regions. The category prediction module is used to predict the category based on the first fused feature map f. h Classify the lesion invasion level using the decoded feature map; The convolutional neural network model was trained and tested based on the dataset to obtain a lesion invasion depth assessment model based on endoscopic ultrasound; the dataset includes several endoscopic ultrasound images labeled with lesion contours and invasion depth types.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the lesion invasion depth assessment method based on endoscopic ultrasound as described in claim 9.

Images