Endoscopic ultrasound-based lesion evaluation system and method, and storage medium

Abstract

Disclosed in the present invention are an endoscopic ultrasound-based lesion evaluation system and method, and a storage medium. The system comprises: an endoscopic ultrasound target recognition module used for performing target detection on an endoscopic ultrasound image to obtain a first target image; a feature extraction module used for performing layer‑by‑layer feature extraction on the first target image to obtain backbone feature maps; a feature aggregation module used for fusing the backbone feature maps to obtain aggregated feature maps; a decoding and segmentation module used for performing multiple decoding operations on the basis of the aggregated feature maps and obtaining a segmentation prediction map; and a multi‑group category prediction module used for performing computations on the basis of the aggregated feature maps to obtain originating layer, echogenicity level, and disease classification probability results, and constructing a three‑dimensional aggregated probability matrix and calibrating same on the basis of a preset constant matrix. The present invention achieves high segmentation accuracy, enables simultaneous evaluation of the originating layer, echogenicity level, and disease type, simplifies the system complexity, and further improves the accuracy and reliability of evaluation by means of an aggregated probability matrix and constant matrix-based calibration.

Description

Endoscopic ultrasound lesion evaluation system, method and storage medium

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the Chinese patent application No. 202411230065.5, filed on September 4, 2024, and entitled "Endoscopic ultrasound lesion evaluation system, method and storage medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of image processing, in particular to an endoscopic ultrasound lesion evaluation system, method and storage medium. BACKGROUND

[0004] Endoscopic ultrasound (EUS) is one of the important imaging technologies in modern medicine, and is widely used in the diagnosis and treatment of digestive tract diseases. In particular, in the evaluation of submucosal tumor lesions, EUS images can provide detailed tumor information, such as size, shape and echo characteristics. However, traditional EUS images have certain limitations in resolution and clarity, and rely on the experience of doctors for evaluation, which is prone to subjective bias and diagnostic errors.

[0005] In existing intelligent technologies, multiple algorithms are usually used to perform different tasks to evaluate and analyze EUS images, but there are still shortcomings such as low segmentation accuracy and inability to establish correlation information between multiple models. If multiple model correlation is implemented, it will result in high correlation system complexity and poor stability. Therefore, there is an urgent need to develop an endoscopic ultrasound lesion evaluation system, method and storage medium. SUMMARY

[0006] In view of the problem in the prior art that there is no system with low system complexity, good stability, and the ability to accurately evaluate the range, origin level, echo level and disease type of submucosal tumors, the present application provides an endoscopic ultrasound lesion evaluation system, method and storage medium, which can accurately and comprehensively evaluate lesions while taking into account system stability and low system complexity.

[0007] The technical solution adopted by the present application to solve the technical problems is as follows:

[0008] In a first aspect, the present application provides an endoscopic ultrasound lesion evaluation system, comprising:

[0009] an endoscopic ultrasound target recognition module, configured to perform automatic target detection on an endoscopic ultrasound image using a target detection model to obtain a first target image;

[0010] a feature extraction module configured to perform layer-by-layer feature extraction on the first target image to obtain a plurality of backbone feature maps;

[0011] a feature aggregation module configured to fuse the backbone feature maps in adjacent order to obtain a plurality of aggregated feature maps;

[0012] a decoding segmentation module configured to perform multiple decoding operations based on the aggregated feature maps to obtain a segmentation prediction map;

[0013] a plurality of sets of class prediction modules configured to perform operations on the aggregated feature maps to obtain an origin hierarchical classification probability result, an echo level classification probability result, and a disease classification probability result, calculate a joint probability of the origin hierarchical classification probability result, the echo level classification probability result, and the disease classification probability result, and construct a three-dimensional aggregated probability matrix based on the joint probability; and correct the aggregated probability matrix based on a preset constant matrix to obtain a corrected origin hierarchical classification probability result, a corrected echo level classification probability result, and a corrected disease classification probability result.

[0014] In a preferred embodiment, the ultrasound endoscope target recognition module is specifically configured to: perform automatic target detection on the ultrasound endoscope image by using a target detection model to obtain a first detection result, and perform dilation on the first detection result to obtain the first target image.

[0015] In a preferred embodiment, the first detection result is a tumor region coordinate under the ultrasound endoscope, and the first target image is an image of a tumor and a surrounding region in the ultrasound endoscope image.

[0016] In a preferred embodiment, the feature extraction module includes five groups of Transformer network units arranged in sequence, the first group of Transformer network units is configured to perform feature extraction on the first target image by convolution operation to obtain a first backbone feature map, the second group of Transformer network units is configured to perform feature extraction on the first backbone feature map by convolution operation to obtain a second backbone feature map, the third group of Transformer network units is configured to perform feature extraction on the second backbone feature map by convolution operation to obtain a third backbone feature map, the fourth group of Transformer network units is configured to perform feature extraction on the third backbone feature map by convolution operation to obtain a fourth backbone feature map, and the fifth group of Transformer network units is configured to perform feature extraction on the fourth backbone feature map by convolution operation to obtain a fifth backbone feature map.

[0017] In a preferred embodiment, the feature aggregation module is configured to perform feature fusion on every two adjacent backbone feature maps in sequence to obtain fourth, third, second and first aggregated feature maps; the feature fusion includes sequentially performing upsampling and concatenation operation, self-attention module weighting operation and ReLu activation function activation operation.

[0018] In a preferred embodiment, the decoding segmentation module is configured to: perform fusion and concatenation on the third aggregated feature map and the fourth aggregated feature map after upsampling, and perform feature decoding to obtain a third decoded feature map; perform fusion and concatenation on the second aggregated feature map and the third decoded feature map after upsampling, and perform feature decoding to obtain a second decoded feature map; perform fusion and concatenation on the first aggregated feature map and the second decoded feature map after upsampling, and perform feature decoding to obtain a first decoded feature map; and perform upsampling operation on the first decoded feature map to restore the size of the first target image, and then perform activation function activation to obtain a segmentation prediction map.

[0019] In a preferred embodiment, the decoding segmentation module includes a plurality of decoder units and a segmentation prediction unit, the decoder unit includes: a position encoding layer configured to provide spatial position information for each input feature block; a self-attention mechanism layer configured to capture global and local feature interactions between input feature blocks; a feedforward neural network layer configured to perform nonlinear transformation on features output by the self-attention mechanism layer to output a decoded feature map; and the segmentation prediction unit is configured to perform upsampling operation on the result of N-2 decoding operations to restore the same size as the first target image, and then perform activation function activation to obtain a segmentation prediction map.

[0020] In a preferred embodiment, the plurality of category prediction modules includes:

[0021] a feature fusion unit configured to perform fusion and concatenation on the aggregated feature maps and weighting on the aggregated feature maps to obtain a first fused encoded feature map;

[0022] a first decision network branch configured to perform origin level decision classification based on the first fused encoded feature map to obtain an origin level classification probability result;

[0023] a second decision network branch configured to perform echo level decision classification based on the first fused encoded feature map to obtain an echo level classification probability result;

[0024] a third decision network branch configured to perform disease decision classification based on the first fused encoded feature map to obtain a disease classification probability result;

[0025] a probability matrix aggregation unit configured to construct a three-dimensional aggregation probability matrix using the origin hierarchical classification probability result, the echo level classification probability result, and the disease classification probability result as dimensions; a cell of the aggregation probability matrix representing a joint probability of a classification probability of the origin hierarchical classification result, a classification probability of the echo level classification result, and a classification probability of the disease classification result corresponding to the cell;

[0026] a constant matrix correction unit configured to perform a weighted fusion operation on the aggregation probability matrix and a preset constant matrix to obtain a corrected origin hierarchical classification probability result, a corrected echo level classification probability result, and a corrected disease classification probability result.

[0027] In a preferred embodiment, the feature fusion unit is specifically configured to align a plurality of aggregated feature map dimensions through a bilinear interpolation operation, perform a fusion splicing operation after alignment to obtain an aggregated feature map one, and input the aggregated feature map one to a self-attention mechanism weighting layer to obtain a first fusion encoding feature map.

[0028] The first decision network branch, the second decision network branch, and the third decision network branch each include: three groups of Transformer encoding layers configured to input the first fusion encoding feature map, continuously extract and deepen global context information and key feature representation, and obtain a feature map one; one fully connected layer configured to convert the feature map one into a one-dimensional feature vector; and one probability output layer configured to convert the one-dimensional feature vector into a category probability representation and output a classification probability result.

[0029] In a preferred embodiment, the joint probability is a sum of probabilities; and a cell of the constant matrix is a weighting value determined according to an occurrence probability of a combination of the origin hierarchical classification result, the echo level classification result, and the disease classification result in clinical experience.

[0030] In a second aspect, the present application provides an ultrasound endoscopy lesion evaluation method, which includes:

[0031] performing automatic target detection on the ultrasound endoscopy image by using a target detection model to obtain a first target image;

[0032] performing layer-by-layer feature extraction on the first target image to obtain a plurality of backbone feature maps;

[0033] fusing the backbone feature maps in adjacent order to obtain a plurality of aggregated feature maps;

[0034] performing multiple decoding operations based on the aggregated feature maps to obtain a segmentation prediction map;

[0035] performing operations on the aggregated feature maps to obtain an origin hierarchical classification probability result, an echo level classification probability result, and a disease classification probability result.

[0036] a joint probability of the origin hierarchy classification probability result, the echo level classification probability result and the disease classification probability result is calculated, and a three-dimensional aggregated probability matrix is constructed according to the joint probability;

[0037] The aggregated probability matrix is corrected according to a preset constant matrix, to obtain a corrected origin hierarchy classification probability result, a corrected echo level classification probability result and a corrected disease classification probability result.

[0038] In a third aspect, the present application provides a storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps of the lesion evaluation method under endoscopic ultrasound in the second aspect.

[0039] The present application provides a lesion evaluation system and method under endoscopic ultrasound and a storage medium, which can extract and aggregate features of an EUS image, and solve the problem of low segmentation accuracy through multiple segmentation, accurately segment the contour range of a submucosal lesion (tumor), and evaluate the origin hierarchy, echo level and disease type. Through correction of an aggregated probability matrix and a constant matrix, the system complexity is simplified, the system stability is improved, and the accuracy and reliability of lesion evaluation are further improved. Through the present application, the lesion segmentation result, origin hierarchy, echo level and disease type can be obtained at one time and accurately. BRIEF DESCRIPTION OF DRAWINGS

[0040] Fig. 1 is a schematic diagram of the framework of the lesion evaluation system under endoscopic ultrasound.

[0041] Fig. 2 is an input-output diagram of the endoscopic ultrasound target recognition module.

[0042] Fig. 3 is a structural schematic diagram of the decoder unit.

[0043] Fig. 4 is a diagram of existing clinical data.

[0044] Fig. 5 is a flowchart of the lesion evaluation system under endoscopic ultrasound. DETAILED DESCRIPTION

[0045] The present application will be further described in detail below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0046] EUS is applied to the evaluation of digestive tract diseases, and the existing EUS image processing and evaluation method mainly has the following shortcomings:

[0047] Low segmentation accuracy: the existing segmentation algorithm has low segmentation accuracy when processing complex EUS images, and often cannot accurately segment the contour area of the submucosal mass;

[0048] Direct correlation information cannot be established between multiple models; if correlation is achieved, it will result in high system complexity and poor stability;

[0049] It is difficult to accurately evaluate the origin level, echo level and disease type.

[0050] Therefore, the present application provides an endoscopic ultrasound lesion evaluation system, method and storage medium, which can perform feature extraction and feature aggregation on EUS images, and solve the problem of low segmentation accuracy through multi-task segmentation, accurately segmenting the contour range of the submucosal mass. The present application can evaluate the origin level, echo level and disease type of the lesion (such as a tumor), correct the aggregated probability matrix through a constant matrix, simplify the system complexity, improve the system stability, and further improve the accuracy and reliability of lesion evaluation. Through the system and method, the lesion segmentation result, origin level, echo level and disease type can be obtained at one time.

[0051] The various embodiments described below can be performed independently, and the various embodiments can also be performed in combination with each other, and the present application does not specifically limit this. In the embodiments of the present application, "first", "second", etc. are used to describe various components, but these components should not be limited by these terms. These terms are only used to distinguish one component from another. In the present application, "and / or" means any and all combinations of one or more related listed items.

[0052] Next, the endoscopic ultrasound lesion evaluation system, method and technical effects thereof are described.

[0053] Fig. 1 is a schematic diagram of the framework of an endoscopic ultrasound lesion evaluation system provided by an embodiment, as shown in Fig. 1, the lesion evaluation system provided by the present embodiment is suitable for lesion evaluation of digestive tract diseases, and the system comprises:

[0054] An endoscopic ultrasound target recognition module 10 is configured to perform automatic target detection on the endoscopic ultrasound image by using a target detection model to obtain a first target image;

[0055] A feature extraction module 20 is configured to perform layer-by-layer feature extraction on the first target image to obtain a plurality of backbone feature maps;

[0056] A feature aggregation module 30 is configured to fuse the backbone feature maps in adjacent order to obtain a plurality of aggregated feature maps;

[0057] A decoding segmentation module 40 is configured to perform multiple decoding operations based on the aggregated feature maps to obtain a segmentation prediction map;

[0058] A plurality of groups of category prediction modules 50 are configured to perform operations on the aggregated feature maps to obtain origin level classification probability results, echo level classification probability results, and disease classification probability results, calculate joint probabilities of the origin level classification probability results, the echo level classification probability results, and the disease classification probability results, and construct a three-dimensional aggregated probability matrix according to the joint probabilities; correct the aggregated probability matrix according to a preset constant matrix to obtain corrected origin level classification probability results, corrected echo level classification probability results, and corrected disease classification probability results.

[0059] The following will be described in detail taking tumor lesions as an example, but the target of the lesion evaluation system is not limited to tumors, but can be other lesions, and is also applicable to the evaluation of other lesions under endoscopic ultrasound.

[0060] Specifically, the feature extraction module 20 is configured to perform layer-by-layer feature extraction on the first target image to obtain N backbone feature maps, where N≥5; the feature aggregation module 30 is configured to perform feature fusion on each two adjacent backbone feature maps in sequence to obtain N-1 aggregated feature maps; that is, feature fusion is required for any two adjacent backbone feature maps in sequence, where the sequence is the sequence in which the feature extraction module 20 obtains the backbone feature maps; and the decoding segmentation module 40 is configured to perform N-2 times of fusion splicing and decoding operations based on the aggregated feature maps to obtain a segmentation prediction map.

[0061] In this embodiment, the endoscopic ultrasound target recognition module 10 uses a target detection model to automatically detect targets in the endoscopic ultrasound image to obtain a first detection result, which is the coordinates of the tumor region under endoscopic ultrasound. The image of the tumor and its surrounding region is obtained as the first target image according to the tumor region coordinates. Specifically, the first detection result is dilated to obtain the first target image, and the size of the first target image is h x w, where h represents the height and w represents the width. The first target image is the image of the tumor and its surrounding region in the endoscopic ultrasound image.

[0062] The target detection model can use one of the target detection algorithms such as yolo, ssd, and faster-rcnn. The target detection model receives the input endoscopic ultrasound image, automatically detects targets in the image, and outputs the first detection result.

[0063] The dilation refers to the expansion of the image width and height. Specifically, based on a preset dilation coefficient, the tumor coordinate region is dilated to obtain new dilated coordinates, ensuring that the region within these new dilated coordinates contains image information of the tumor and its surrounding mucosal layers. The dilation coefficient is preset; in this invention, the dilation coefficient is half the width and height of the target image, effectively doubling its original size. The top and bottom of the first detection result are dilated by half in height, and the left and right are dilated by half in width, resulting in the tumor coordinate region and its surrounding area. This region serves as the pre-obtained region. The endoscopic ultrasound image is then cropped to obtain the first target image, which is the image of the tumor and its surrounding area within the endoscopic ultrasound image, as shown in Figure 2.

[0064] In this embodiment, the feature extraction module 20 includes N sets of concatenated convolutional neural network units for feature extraction, where N is an integer. Specifically, it includes 5 sets of convolutional neural network units arranged sequentially. In this embodiment, the convolutional neural network units use Transformer network units, that is, 5 sets of Transformer network units are connected in series to obtain a total of 5 backbone feature maps.

[0065] The first group of convolutional neural network units extracts features from the first target image using convolution operations to obtain a first backbone feature map with a resolution of [h / 2, w / 2]. The second group of convolutional neural network units extracts features from the first backbone feature map using convolution operations to obtain a second backbone feature map with a resolution of [h / 2]. 2 w / 2 2 The third group of convolutional neural network units extracts features from the second backbone feature map through convolution operations, resulting in the third backbone feature map with a resolution of [h / 2]. 3 w / 2 3 The fourth group of convolutional neural network units extracts features from the third backbone feature map through convolution operations, resulting in the fourth backbone feature map with a resolution of [h / 2]. 4 w / 2 4 The fifth group of convolutional neural network units extracts features from the fourth backbone feature map through convolution operations, resulting in the fifth backbone feature map with a resolution of [h / 2]. 5 w / 2 5 ].

[0066] The feature aggregation module 30 performs feature fusion on the fifth, fourth, third, second, and first main feature maps according to the order of every two adjacent main feature maps, to obtain the fourth, third, second, and first aggregated feature maps in sequence.

[0067] The feature aggregation module 30 is specifically configured to perform upsampling operation on the fifth backbone feature map, perform feature map splicing operation on the fourth backbone feature map and the upsampled fifth backbone feature map, sequentially perform self-attention module weighting operation and ReLu activation function activation operation on the spliced feature map, and obtain the fourth aggregated feature map.

[0068] The fourth backbone feature map is upsampled, and the upsampled fourth backbone feature map is spliced with the third backbone feature map, and then sequentially subjected to self-attention module weighting operation and ReLu activation function activation operation to obtain the third aggregated feature map.

[0069] The third backbone feature map is upsampled, and the upsampled third backbone feature map is spliced with the second backbone feature map, and then sequentially subjected to self-attention module weighting operation and ReLu activation function activation operation to obtain the second aggregated feature map.

[0070] The second backbone feature map is upsampled, and the upsampled second backbone feature map is spliced with the first backbone feature map, and then sequentially subjected to self-attention module weighting operation and ReLu activation function activation operation to obtain the first aggregated feature map.

[0071] The decoding segmentation module 40 comprises three groups of decoder units and one group of segmentation prediction units. The three groups of decoder units are respectively referred to as a third decoder unit, a second decoder unit and a first decoder unit. The third decoder unit, the second decoder unit, the first decoder unit and the segmentation prediction unit are sequentially arranged.

[0072] The decoder unit is used for feature decoding and restoring a segmentation region.

[0073] The third decoder unit is configured to upsample the fourth aggregated feature map, fuse and splice the third aggregated feature map and the upsampled fourth aggregated feature map, perform feature decoding on the fused and spliced feature map to restore a segmentation region, and obtain a third decoded feature map.

[0074] The second decoder unit is configured to upsample the third decoded feature map, fuse and splice the second aggregated feature map and the upsampled third decoded feature map, perform feature decoding on the fused and spliced feature map to restore a segmentation region, and obtain a second decoded feature map.

[0075] The first decoder unit is configured to upsample the second decoded feature map, fuse and splice the first aggregated feature map and the upsampled second decoded feature map, perform feature decoding on the fused and spliced feature map to restore a segmentation region, and obtain a first decoded feature map.

[0076] The segmentation prediction unit is configured to perform an up-sampling operation on the first decoded feature map (result of the N-2 times of decoding operation), restore the same size h x w as the first target image, and obtain a segmentation prediction map after Sigmoid activation.

[0077] Referring to FIG. 3, the decoder unit includes a position encoding layer, a self-attention mechanism layer, and a feed-forward neural network layer.

[0078] Specifically, the decoder unit includes: 1 up-sampling layer configured to up-sample the aggregated feature map from a low resolution to a high resolution; 1 feature concatenation layer configured to concatenate the output of the up-sampling layer and the aggregated feature map; 1 position encoding layer configured to provide spatial position information for each input feature block; 1 self-attention mechanism layer configured to capture global and local feature interactions between the input feature blocks, and the input of the self-attention mechanism layer includes the output of the position encoding layer; 1 feed-forward neural network layer configured to perform nonlinear transformation on the features output by the self-attention mechanism layer; 2 layer normalization layers, one of which is configured to normalize the output of the self-attention mechanism layer, and the other of which is configured to normalize the output of the feed-forward neural network layer; and 2 residual connection branches, one of which is configured to retain input feature information in the self-attention mechanism layer, and the other of which is configured to retain input feature information in the feed-forward neural network layer. The feature block is the fused and concatenated feature map described above, and the normalized output of the feed-forward neural network layer is a decoded feature map.

[0079] The multiple sets of category prediction modules 50 include a feature fusion unit, three decision network branches, a probability matrix aggregation unit, and a constant matrix correction unit. The three decision network branches are an origin level decision network branch (first decision network branch), an echo level decision network branch (second decision network branch), and a disease classification decision network branch (third decision network branch).

[0080] The feature fusion unit is configured to receive the first aggregated feature map, the second aggregated feature map, and the third aggregated feature map obtained by the feature aggregation module 30, align the dimensions of the three aggregated feature maps through a bilinear interpolation operation, and then perform a Concat fusion and concatenation operation to obtain an aggregated feature map 1. The aggregated feature map 1 is input into a self-attention mechanism weighting layer to obtain a first fused encoding feature map.

[0081] Here, the feature fusion unit can also fuse all the aggregated feature maps, and the embodiment does not limit the number of fused aggregated feature maps.

[0082] The decision network branch is configured to perform decision classification according to the first fused encoding feature map to obtain a classification probability result.

[0083] The decision network branch, each branch internally comprises: three groups of Transformer encoding layers, the Transformer encoding layers are used for continuously extracting and deepening global context information and key feature representation, and a feature map one is obtained after operation of the three groups of Transformer encoding layers; one full connection layer, used for converting the feature map one obtained by operation of the three groups of Transformer encoding layers into a one-dimensional feature vector; and one probability output layer, used for converting the one-dimensional feature vector into a category probability representation, that is, outputting a corresponding classification probability result.

[0084] The classification probability result comprises a classification result and a classification probability. The classification probability results output by the first, second and third decision network branches are respectively:

[0085] The origin level classification probability result: three preset categories, the second layer, the third layer and the fourth layer, which can be understood as an example, and the embodiment is preset to be only one of the second layer, the third layer and the fourth layer, and the expression is: P A =[p a1 ,p a2 ,p a3 ]

[0086] Wherein, P A represents the origin level classification probability result, the probability of origin level classification as the second layer is p a1 , the probability of origin level classification as the third layer is p a2 , and the probability of origin level classification as the fourth layer is p a3 .

[0087] The echo level classification probability result: three preset categories in the embodiment, high echo, low echo and medium echo, and the expression is: P B =[p b1 ,p b2 ,p b3 ]

[0088] Wherein, P B represents the echo level classification probability result, the probability of echo level classification as high echo is p b1 , the probability of echo level classification as low echo is p b2 , and the probability of echo level classification as medium echo is p b3 .

[0089] The disease classification probability result: seven preset categories, interstitialoma, leiomyoma, schwannoma, lipoma, ectopic pancreas, neuroendocrine tumor and lymphoma, and the expression is: P C =[p c1 ,p c2 ,p c3p c4 p c5 p c6 p c7 ]

[0090] wherein P C represents the disease classification probability result, P c1 p c2 p c3 p c4 p c5 p c6 p c7 respectively represent the probability of the disease classification being mesenchymoma, leiomyoma, schwannoma, lipoma, ectopic pancreas, neuroendocrine tumor, and lymphoma.

[0091] The probability matrix aggregation unit constructs an aggregated probability matrix in three dimensions of the origin hierarchical classification probability result, the echo level classification probability result, and the disease classification probability result. Specifically, the three classification results are converted into a three-dimensional matrix representation to obtain a three-dimensional classification probability matrix, with three dimensions corresponding to the origin hierarchical classification, the echo level classification, and the disease classification. Since the origin hierarchy is preset to have 3 categories, the echo level is preset to have 3 categories, and the disease is preset to have 7 categories, the dimension of the matrix is 3x3x7, and the aggregated probability matrix is obtained by mapping the actual classification probability result into the matrix. In this embodiment, the x-axis of the aggregated probability matrix represents the origin hierarchical classification probability result, the y-axis represents the echo level classification probability result, and the z-axis represents the disease classification probability result. The mapping method is to calculate the joint probability of the origin hierarchical classification probability result, the echo level classification probability result, and the disease classification probability result, and then assign the value to the corresponding cell position of the matrix,

[0092] The aggregated probability matrix is the sum of the origin hierarchical classification probability result, the echo level classification probability result, and the disease classification probability result, and the cell of the aggregated probability matrix is the sum of the classification probability of the origin hierarchical classification result, the classification probability of the echo level classification result, and the classification probability of the disease classification result. Specifically, M(i,j,k) = P A [i] + P B [j] + P C [k]

[0093] wherein M(i,j,k) represents the aggregated probability matrix, P A [i] represents the classification probability of the i-th classification result in the origin hierarchical classification probability result, P B [j] represents the classification probability of the j-th classification result in the echo level classification probability result, and P C[k] represents the classification probability of the kth classification result in the disease classification probability result, i, j, k are positive integers, (i, j, k) represents the coordinate point of the aggregation probability matrix, and the element m(i,j,k) of the aggregation probability matrix can be represented as: the element m(i,j,k) of the aggregation probability matrix = [p ai ,p bj ,p ck ].

[0094] A constant matrix correction unit is configured to correct the aggregation probability matrix according to a preset constant matrix to obtain a corrected origin level classification probability result, a corrected echo level classification probability result and a corrected disease classification probability result.

[0095] The constant matrix correction unit inputs the aggregation probability matrix and performs weighted multiplication on the aggregation probability matrix and the preset constant matrix. It can be understood that the preset constant matrix is consistent with the dimension of the aggregation probability matrix, and the element of the constant matrix is a preset weight. The weight is the appearance probability (or the weighted value determined according to the appearance probability) of the combination of the origin level classification result, the echo level classification and the disease classification result according to clinical experience. FIG. 4 shows part of the existing clinical data that can be used for reference in the present embodiment. For example, according to the clinical experience, the second layer origin, low echo and leiomyoma are often combined, so the coordinate (0, 1, 1) in the constant matrix is set to a higher weighted value. Similarly, if a certain coordinate point is a combination that will not appear in the clinic, the coordinate point is set to 0. After the weighted fusion operation, the final aggregation probability matrix is obtained, which is called the corrected aggregation probability matrix, and the corrected origin level classification probability result, the corrected echo level classification probability result and the corrected disease classification probability result are obtained.

[0096] Referring to FIG. 5, an ultrasound endoscopic lesion evaluation method is shown, which includes:

[0097] S1, using a target detection model to automatically detect the target of the ultrasound endoscopic image to obtain a first target image;

[0098] S2, performing layer-by-layer feature extraction on the first target image to obtain a plurality of (N, N≥5) backbone feature maps;

[0099] S3, fusing (every two) adjacent order backbone feature maps to obtain a plurality of (N-1) aggregation feature maps;

[0100] S4, performing N-2 times of decoding operation based on the aggregation feature map to obtain a segmentation prediction map;

[0101] S5, performing operation on the aggregation feature map to obtain an origin level classification probability result, an echo level classification probability result and a disease classification probability result;

[0102] S6, calculate a joint probability of the origin hierarchical classification probability result, the echo level classification probability result and the disease classification probability result, and construct a three-dimensional aggregated probability matrix according to the joint probability;

[0103] S7, correct the aggregated probability matrix according to a preset constant matrix, to obtain a corrected origin hierarchical classification probability result, a corrected echo level classification probability result and a corrected disease classification probability result.

[0104] It should be understood that the above method can be different according to the different execution processes, for example, the execution order of S4 is not limited, so the above method does not represent or imply that all steps must be executed in this order, and a person skilled in the art can transform or change the execution order of the above steps on the basis of the present application, and some embodiments of the above method will be illustrated below.

[0105] The lesion evaluation method under the endoscopic ultrasound can realize lesion evaluation by referring to the lesion evaluation system under the endoscopic ultrasound in any of the above embodiments, and the specific implementation steps will not be repeated.

[0106] The present application also provides a storage medium storing a computer program, characterized in that the computer program is executed by a processor to implement the steps of the lesion evaluation method under the endoscopic ultrasound.

[0107] The effects of the lesion evaluation system, method and storage medium under the endoscopic ultrasound of the present application are:

[0108] The present application designs a feature extraction module 20 and a feature aggregation module 30, specifically uses multiple groups of Transformer network units for feature extraction, and uses multi-scale and multi-level features through hierarchical feature fusion technology to improve the accuracy of image segmentation and target detection.

[0109] The present application optimizes the design of the decoder and the segmentation prediction unit: it can accurately segment the lesion; by designing a decoder unit and a segmentation prediction unit containing a self-attention mechanism, the accuracy and efficiency of image segmentation are effectively improved.

[0110] The present application designs a multi-group category prediction and probability matrix aggregation technology: through multiple decision network branches and a probability matrix aggregation unit, multi-dimensional classification of origin hierarchy, echo level and disease category is comprehensively realized, providing comprehensive lesion evaluation information and simplifying the system complexity; the correction of the aggregated probability matrix by the constant matrix improves the multi-class classification accuracy of the model, and the model still maintains high accuracy when dealing with different patients and complex lesions, and has good robustness and generalization performance.

[0111] It should be noted that in the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be seen in the relevant description of other embodiments.

[0112] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0113] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0115] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure. Therefore, the appended claims are intended to cover all such additional variations and modifications as fall within the scope of the present application.

Claims

1. An endoscopic ultrasound-based lesion assessment system, characterized by, The method comprises the following steps: An endoscopic ultrasound target recognition module is used to automatically detect targets in an endoscopic ultrasound image by using a target detection model to obtain a first target image; A feature extraction module is used to extract features from the first target image layer by layer to obtain a plurality of backbone feature maps; A feature aggregation module is used to fuse the backbone feature maps in adjacent order to obtain a plurality of aggregated feature maps; A decoding segmentation module is used to perform multiple decoding operations based on the aggregated feature maps to obtain a segmentation prediction map; A multi-group class prediction module is used to calculate the joint probability of the origin level classification probability result, the echo level classification probability result and the disease classification probability result, and construct a three-dimensional aggregated probability matrix according to the joint probability; the aggregated probability matrix is corrected according to a preset constant matrix to obtain the corrected origin level classification probability result, the corrected echo level classification probability result and the corrected disease classification probability result.

2. The endoscopic lesion assessment system of claim 1, wherein, The feature extraction module comprises five groups of Transformer network units arranged in sequence, the first group of Transformer network units is used to extract features from the first target image by convolution operation to obtain a first backbone feature map, the second group of Transformer network units is used to extract features from the first backbone feature map by convolution operation to obtain a second backbone feature map, the third group of Transformer network units is used to extract features from the second backbone feature map by convolution operation to obtain a third backbone feature map, the fourth group of Transformer network units is used to extract features from the third backbone feature map by convolution operation to obtain a fourth backbone feature map, and the fifth group of Transformer network units is used to extract features from the fourth backbone feature map by convolution operation to obtain a fifth backbone feature map.

3. The endoscopic lesion assessment system of claim 2, wherein The feature aggregation module is used to perform feature fusion on the fifth backbone feature map, the fourth backbone feature map, the third backbone feature map, the second backbone feature map and the first backbone feature map in the manner of every two adjacent order backbone feature maps to sequentially obtain a fourth aggregated feature map, a third aggregated feature map, a second aggregated feature map and a first aggregated feature map; the feature fusion includes sequentially performing upsampling and splicing operation, self-attention module weighting operation and ReLu activation function activation operation.

4. The endoscopic lesion assessment system of claim 3, wherein The decoding segmentation module is specifically used for: fusing and splicing the third aggregated feature map and the fourth aggregated feature map after upsampling, and then performing feature decoding to obtain a third decoding feature map; fusing and splicing the second aggregated feature map and the third decoding feature map after upsampling, and then performing feature decoding to obtain a second decoding feature map; fusing and splicing the first aggregated feature map and the second decoding feature map after upsampling, and then performing feature decoding to obtain a first decoding feature map; The first decoding feature map is upsampled to restore the size of the first target image, and then activated by an activation function to obtain a segmentation prediction map.

5. The endomicroscopy lesion assessment system according to any one of claims 1 to 4, wherein The decoding segmentation module comprises a plurality of decoder units and one segmentation prediction unit, the decoder units comprising: one position encoding layer for providing spatial position information for each input feature block; one self-attention mechanism layer for capturing global and local feature interactions between input feature blocks; and one feed-forward neural network layer for performing nonlinear transformation on features output by the self-attention mechanism layer to output a decoding feature map; and the segmentation prediction unit is configured to perform upsampling operation on results of multiple decoding operations to restore the same size as the first target image, and then perform activation through an activation function to obtain a segmentation prediction map.

6. The endoscopic lesion assessment system of claim 1, wherein The plurality of category prediction modules comprise: a feature fusion unit configured to fuse and weight the aggregated feature maps to obtain a first fused encoding feature map; a first decision network branch configured to perform origin level decision classification based on the first fused encoding feature map to obtain an origin level classification probability result; a second decision network branch configured to perform echo level decision classification based on the first fused encoding feature map to obtain an echo level classification probability result; a third decision network branch configured to perform disease decision classification based on the first fused encoding feature map to obtain a disease classification probability result; a probability matrix aggregation unit configured to construct a three-dimensional aggregated probability matrix using the origin level classification probability result, the echo level classification probability result and the disease classification probability result as dimensions; and a unit of the aggregated probability matrix representing a joint probability of a classification probability of the origin level classification result, a classification probability of the echo level classification result and a classification probability of the disease classification result corresponding to the unit. a constant matrix correction unit configured to perform weighted fusion operation on the aggregated probability matrix and a preset constant matrix to obtain a corrected origin level classification probability result, a corrected echo level classification probability result and a corrected disease classification probability result.

7. The endoscopic lesion assessment system of claim 6, wherein The feature fusion unit is specifically configured to align dimensions of a plurality of aggregated feature maps through a bilinear interpolation operation, perform fusion splicing operation on the aligned aggregated feature maps to obtain an aggregated feature map one, and input the aggregated feature map one into a self-attention mechanism weighting layer to obtain the first fused encoding feature map. The first decision network branch, the second decision network branch and the third decision network branch each comprise: three groups of Transformer encoding layers, which are input with the first fused encoding feature map and are configured to continuously extract and deepen global context information and key feature representation to obtain a feature map one; one fully connected layer configured to convert the feature map one into a one-dimensional feature vector; and one probability output layer configured to convert the one-dimensional feature vector into a category probability representation to output a classification probability result. The joint probability is a sum of probabilities; and a unit of the constant matrix is a weighting value determined according to an occurrence probability of a combination of the origin level classification result, the echo level classification result and the disease classification result in clinical experience.

8. The endoscopic lesion assessment system of claim 6, wherein, The method comprises:

9. An ultrasonic endoscopic method for evaluating a lesion, characterized by, performing target automatic detection on the endoscopic ultrasound image by using a target detection model to obtain a first target image; performing layer-by-layer feature extraction on the first target image to obtain a plurality of backbone feature maps; fusing adjacent sequential backbone feature maps to obtain a plurality of aggregated feature maps; and ​ performing multiple decoding operations based on the aggregated feature map to obtain a segmentation prediction map; performing operations on the aggregated feature map to obtain an origin level classification probability result, an echo level classification probability result, and a disease classification probability result; calculating joint probabilities of the origin level classification probability result, the echo level classification probability result, and the disease classification probability result, and constructing a three-dimensional aggregated probability matrix based on the joint probabilities; correcting the aggregated probability matrix according to a preset constant matrix to obtain a corrected origin level classification probability result, a corrected echo level classification probability result, and a corrected disease classification probability result.

10. A storage medium storing a computer program, characterized by The computer program is executed by a processor to implement the steps of the lesion evaluation method under an endoscopic ultrasound as claimed in claim 9.

Images