Ultrasonic endoscopic lesion assessment system, method and storage medium

Abstract

The present invention discloses an endoscopic ultrasound lesion assessment system, method and storage medium, the system comprising: an endoscopic ultrasound target recognition module for performing target detection on an endoscopic ultrasound image to obtain a first target image, a feature extraction module for performing layer-by-layer feature extraction on the first target image to obtain a trunk feature map, a feature aggregation module for fusing the trunk feature map to obtain an aggregate feature map, a decoding and segmentation module for performing multiple decoding operations based on the aggregate feature map to obtain a segmentation prediction map, and a multi-group category prediction module for performing operations based on the aggregate feature map to obtain an origin level, an echo level and a disease classification probability result and construct and correct a three-dimensional aggregate probability matrix according to a preset constant matrix. The segmentation accuracy of the present invention is high, and the origin level, echo level and disease type can be evaluated simultaneously, the system complexity is simplified, and the accuracy and reliability of the evaluation are further improved through the correction of the aggregate probability matrix and the constant matrix.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a system, method and storage medium for evaluating lesions under ultrasound endoscopy. Background Art

[0002] Endoscopic Ultrasound (EUS) is one of the important imaging techniques in modern medicine and is widely used in the diagnosis and treatment of gastrointestinal diseases. Especially in the evaluation of submucosal tumor lesions, EUS images can provide detailed tumor information, such as size, morphology, and echo characteristics. However, traditional EUS images have certain limitations in resolution and clarity, and rely on the doctor's experience for evaluation, which is prone to subjective bias and diagnostic errors.

[0003] In existing intelligent technologies, multiple algorithms are usually used to perform different tasks to evaluate and analyze EUS images, but there are still shortcomings such as low segmentation accuracy and inability to directly establish correlation information between multiple models. If multi-model correlation is achieved, the correlation system will be very complex and have poor stability. Therefore, it is urgent to develop an ultrasound endoscopic lesion evaluation system, method and storage medium. Summary of the invention

[0004] In view of the problem that the prior art lacks a system with low system complexity, good stability, and the ability to accurately assess the range, origin level, echo level, and disease type of submucosal tumors, the present invention provides an endoscopic ultrasound lesion assessment system, method, and storage medium that can accurately and comprehensively assess lesions while taking into account system stability and low system complexity.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] In a first aspect, the present invention provides an endoscopic ultrasound lesion assessment system, comprising:

[0007] An ultrasonic endoscope target recognition module is used to automatically detect targets on the ultrasonic endoscope image using a target detection model to obtain a first target image;

[0008] A feature extraction module, used to extract features of the first target image layer by layer to obtain multiple trunk feature maps;

[0009] A feature aggregation module is used to fuse the adjacent main feature maps to obtain multiple aggregated feature maps;

[0010] A decoding and segmentation module is used to perform multiple decoding operations based on the aggregated feature map to obtain a segmentation prediction map;

[0011] A multi-group category prediction module is used to perform operations based on the aggregated feature graph to obtain the origin level classification probability results, echo level classification probability results, and disease classification probability results, calculate the joint probability of the origin level classification probability results, the echo level classification probability results, and the disease classification probability results, and construct a three-dimensional aggregated probability matrix based on them; correct the aggregated probability matrix according to a preset constant matrix to obtain the corrected origin level classification probability results, the corrected echo level classification probability results, and the corrected disease classification probability results.

[0012] In a preferred embodiment, the ultrasonic endoscope target recognition module is specifically used to: use the target detection model to automatically detect the target on the ultrasonic endoscope image to obtain a first detection result, and expand the first detection result to obtain a first target image.

[0013] In a preferred embodiment, the first detection result is the coordinates of the tumor region under ultrasound endoscopy, and the first target image is an image of the tumor and its surrounding area in the ultrasound endoscopy image.

[0014] In a preferred embodiment, the feature extraction module includes 5 groups of Transformer network units arranged in sequence, the first group of Transformer network units is used to implement feature extraction on the first target image through convolution operation to obtain a first backbone feature map, the second group of Transformer network units is used to implement feature extraction on the first backbone feature map through convolution operation to obtain a second backbone feature map, the third group of Transformer network units is used to implement feature extraction on the second backbone feature map through convolution operation to obtain a third backbone feature map, the fourth group of Transformer network units is used to implement feature extraction on the third backbone feature map through convolution operation to obtain a fourth backbone feature map, and the fifth group of Transformer network units is used to implement feature extraction on the fourth backbone feature map through convolution operation to obtain a fifth backbone feature map.

[0015] In a preferred embodiment, the feature aggregation module is used to operate the fifth trunk feature map, the fourth trunk feature map, the third trunk feature map, the second trunk feature map, and the first trunk feature map in a manner of performing feature fusion on every two adjacent trunk feature maps, and sequentially obtain the fourth aggregated feature map, the third aggregated feature map, the second aggregated feature map, and the first aggregated feature map; performing the feature fusion includes sequentially performing upsampling and splicing operations, self-attention module weighted operation operations, and ReLu activation function activation operations.

[0016] In a preferred embodiment, the decoding and segmentation module is specifically used to: fuse and splice the third aggregate feature map and the upsampled fourth aggregate feature map, and perform feature decoding after fusion and splicing to obtain a third decoding feature map; fuse and splice the second aggregate feature map and the upsampled third decoding feature map, and perform feature decoding after fusion and splicing to obtain a second decoding feature map; fuse and splice the first aggregate feature map and the upsampled second decoding feature map, and perform feature decoding after fusion and splicing to obtain a first decoding feature map; upsample the first decoding feature map to restore it to the size of the first target image, and then activate it through an activation function to obtain a segmentation prediction map.

[0017] In a preferred embodiment, the decoding segmentation module includes multiple groups of decoder units and a group of segmentation prediction units, and the decoder unit includes: a position encoding layer for providing spatial position information for each input feature block; a self-attention mechanism layer for capturing global and local feature interactions between input feature blocks; a feedforward neural network layer for performing nonlinear transformation on the features output by the self-attention mechanism layer and outputting a decoding feature map; the segmentation prediction unit is used to upsample the results of N-2 decoding operations, restore them to the same size as the first target image, and then activate them through an activation function to obtain a segmentation prediction map.

[0018] In a preferred embodiment, the multiple group category prediction modules include:

[0019] A feature fusion unit, used for fusing and concatenating the aggregate feature graphs and weighting the aggregate feature graphs to obtain a first fused coding feature graph;

[0020] The first decision network branch is used to perform decision classification of the origin level according to the first fusion coding feature map to obtain an origin level classification probability result;

[0021] The second decision network branch is used to make a decision classification of the echo level according to the first fusion coding feature map to obtain an echo level classification probability result;

[0022] The third decision network branch is used to make a decision classification of the disease according to the first fusion coding feature map to obtain a disease classification probability result;

[0023] The probability matrix aggregation unit is used to construct a three-dimensional aggregation probability matrix using the origin level classification probability result, the echo level classification probability result and the disease classification probability result as dimensions; the unit of the aggregation probability matrix represents the joint probability of the classification probability of the origin level classification result, the classification probability of the echo level classification result and the classification probability of the disease classification result corresponding to the unit;

[0024] The constant matrix correction unit is used to perform weighted fusion operation on the aggregated probability matrix and the preset constant matrix to obtain the corrected origin level classification probability result, the corrected echo level classification probability result, and the corrected disease classification probability result.

[0025] In a preferred embodiment, the feature fusion unit is specifically used to align several aggregate feature map dimensions through bilinear interpolation operation, and after alignment, perform fusion and splicing operation to obtain aggregate feature Figure 1 , aggregate features Figure 1 Input to the weighted layer of the self-attention mechanism to obtain the first fusion encoding feature map;

[0026] The first decision network branch, the second decision network branch and the third decision network branch all include: three groups of Transformer encoding layers, the input of which is the first fusion encoding feature map, which is used to continuously extract and deepen the global context information and key feature representation, and obtain the feature Figure 1 ; 1 fully connected layer, used to transform the features Figure 1 Convert it into a one-dimensional feature vector; 1 probability output layer, used to convert the one-dimensional feature vector into a category probability representation and output the classification probability result.

[0027] In a preferred embodiment, the joint probability is the sum of probabilities; the units of the constant matrix are weighted values ​​determined according to the probability of occurrence of the combination of origin level classification results, echo level classification and disease classification results in clinical experience.

[0028] In a second aspect, the present invention provides a method for evaluating lesions under ultrasound endoscopy, comprising:

[0029] Automatically detecting targets on the ultrasonic endoscope image using the target detection model to obtain a first target image;

[0030] Extracting features layer by layer on the first target image to obtain a plurality of backbone feature maps;

[0031] The adjacent order backbone feature maps are fused to obtain multiple aggregated feature maps;

[0032] Perform multiple decoding operations based on the aggregated feature map to obtain a segmentation prediction map;

[0033] According to the aggregated feature graph, operations are performed to obtain the probability results of origin level classification, echo level classification, and disease classification;

[0034] Calculate the joint probability of the origin level classification probability results, the echo level classification probability results and the disease classification probability results and construct a three-dimensional aggregate probability matrix based on them;

[0035] The aggregation probability matrix is ​​corrected according to a preset constant matrix to obtain a corrected origin level classification probability result, a corrected echo level classification probability result, and a corrected disease classification probability result.

[0036] In a third aspect, the present invention provides a storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the steps of the method for evaluating lesions under ultrasound endoscopy described in the second aspect are implemented.

[0037] The present invention provides an endoscopic ultrasound lesion assessment system, method and storage medium, which can extract and aggregate features of EUS images, and solve the problem of low segmentation accuracy through multiple segmentations, accurately segment the contour range of submucosal lesions (tumors), and assess their origin level, echo level, and disease type. Through the correction of the aggregation probability matrix and the constant matrix, the system complexity is simplified, the system stability is improved, and the accuracy and reliability of lesion assessment are further improved. Through the present invention, the lesion segmentation results, origin level, echo level and disease type can be obtained once and for all. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the framework of the endoscopic ultrasound lesion assessment system.

[0039] Figure 2 Input and output diagram of the ultrasound endoscope target recognition module.

[0040] Figure 3 A schematic diagram of the structure of a decoder unit.

[0041] Figure 4 This is an existing clinical data chart.

[0042] Figure 5 Flow chart of the endoscopic ultrasound lesion assessment system. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0044] EUS is used to evaluate digestive tract diseases. Existing EUS image processing and evaluation methods have the following main disadvantages:

[0045] Low segmentation accuracy: The existing segmentation algorithms have low segmentation accuracy when processing complex EUS images and often cannot accurately segment the contour area of ​​submucosal tumors;

[0046] It is impossible to directly establish correlation information between multiple models; if correlation is achieved, the system will be very complex and unstable;

[0047] It is difficult to accurately assess the origin level, echo level and disease type.

[0048] To this end, the present invention provides an endoscopic ultrasound lesion assessment system, method and storage medium, which can extract and aggregate features of EUS images, and solve the problem of low segmentation accuracy through multi-task segmentation, accurately segment the contour range of submucosal tumors, and can evaluate the origin level, echo level, and disease type of lesions (such as tumors). By correcting the aggregation probability matrix through a constant matrix, the system complexity is simplified, the system stability is improved, and the accuracy and reliability of lesion assessment are further improved. Through this system and method, the lesion segmentation results, origin level, echo level and disease type can be obtained at one time.

[0049] The following embodiments of the present invention can be performed separately, and the embodiments can also be performed in combination with each other, and the embodiments of the present invention do not specifically limit this. In the embodiments of the present invention, "first", "second", etc. are used to describe various components, but these components should not be limited by these terms. These terms are only used to distinguish one component from another component. If "and / or" is mentioned in the present invention, it means any and all combinations including one or more related listed items.

[0050] Below, the ultrasound endoscopic lesion assessment system, method and technical effects are described.

[0051] Figure 1 A schematic diagram of a framework of an ultrasound endoscopic lesion assessment system provided in an embodiment is shown in FIG. Figure 1 As shown, the lesion assessment system provided in this embodiment is suitable for lesion assessment of digestive tract diseases, and the system includes:

[0052] The ultrasonic endoscope target recognition module 10 is used to automatically detect the target on the ultrasonic endoscope image using the target detection model to obtain a first target image;

[0053] A feature extraction module 20 is used to extract features of the first target image layer by layer to obtain a plurality of main feature maps;

[0054] A feature aggregation module 30 is used to fuse the adjacent trunk feature maps to obtain multiple aggregated feature maps;

[0055] A decoding and segmentation module 40 is used to perform multiple decoding operations based on the aggregated feature map to obtain a segmentation prediction map;

[0056] The multi-group category prediction module 50 is used to perform operations based on the aggregated feature graph to obtain the origin level classification probability results, the echo level classification probability results, and the disease classification probability results, calculate the joint probability of the origin level classification probability results, the echo level classification probability results, and the disease classification probability results, and construct a three-dimensional aggregated probability matrix based on them; correct the aggregated probability matrix according to a preset constant matrix to obtain the corrected origin level classification probability results, the corrected echo level classification probability results, and the corrected disease classification probability results.

[0057] The following is a detailed description using tumor lesions as an example, but the lesion assessment system is not limited to target only tumors, but can be other lesions, and is also applicable to the assessment of other lesions under ultrasound endoscopy.

[0058] Specifically, the feature extraction module 20 is used to perform layer-by-layer feature extraction on the first target image to obtain N backbone feature maps, where N≥5; the feature aggregation module 30 is used to perform feature fusion on the N backbone feature maps in a manner such that every two adjacent backbone feature maps are subjected to feature fusion to obtain N-1 aggregated feature maps; that is, any two adjacent backbone feature maps need to be feature fused, and the order is the order in which the feature extraction module 20 obtains the backbone feature maps; the decoding and segmentation module 40 is used to perform N-2 fusion, splicing and decoding operations based on the aggregated feature maps to obtain a segmentation prediction map.

[0059] In this embodiment, the ultrasonic endoscope target recognition module 10 uses the target detection model to automatically detect the target of the ultrasonic endoscope image to obtain a first detection result, and the first detection result is the coordinates of the tumor region under the ultrasonic endoscope, and the image of the tumor and its surrounding area is obtained as the first target image according to the coordinates of the tumor region. Specifically, the first detection result is expanded to obtain the first target image, and the size of the first target image is h×w, where h represents the height and w represents the width. The first target image is an image of the tumor and its surrounding area in the ultrasonic endoscope image.

[0060] The target detection model can use one of the target detection algorithms such as YOLO, SSD, FASTER-RCNN, etc. The target detection model receives the input ultrasonic endoscopic image, performs automatic target detection on it, and outputs a first detection result.

[0061] The expansion is the enlargement of the image width and height, specifically: according to the preset expansion coefficient, the tumor coordinate area is expanded to obtain new expansion coordinates, so that the area within the new expansion coordinates contains the image information of the tumor and its surrounding mucosal layers. The expansion coefficient adopts the preset method. The expansion coefficient used in the present invention is 1 / 2 of the width and height of the target image, that is, it becomes twice the original size. The first detection result is expanded by 1 / 2 in height on the top and bottom, and 1 / 2 in width on the left and right, respectively, to obtain the tumor coordinate area and its surrounding area. This area is used as the pre-acquisition area, and the ultrasound endoscopic image is cropped to obtain the first target image. The first target image is the image of the tumor and its surrounding area in the ultrasound endoscopic image. See Figure 2 .

[0062] In this embodiment, the feature extraction module 20 includes N groups of convolutional neural network units connected in series for feature extraction, where N is an integer. Specifically, it includes 5 groups of convolutional neural network units arranged in sequence. In this embodiment, the convolutional neural network unit uses a Transformer network unit, that is, 5 groups of Transformer network units are connected in series, and a total of 5 backbone feature maps are obtained.

[0063] The first group of convolutional neural network units extracts features of the first target image through convolution operations to obtain the first backbone feature map with a resolution of ; The second group of convolutional neural network units extracts features from the first backbone feature map through convolution operation to obtain the second backbone feature map with a resolution of ; The third group of convolutional neural network units extracts features from the second backbone feature map through convolution operation to obtain the third backbone feature map with a resolution of ; The fourth group of convolutional neural network units extracts features from the third backbone feature map through convolution operation to obtain the fourth backbone feature map with a resolution of ; The fifth group of convolutional neural network units extracts the fourth trunk feature map through convolution operation to obtain the fifth trunk feature map with a resolution of .

[0064] The feature aggregation module 30 performs feature fusion on the fifth trunk feature map, the fourth trunk feature map, the third trunk feature map, the second trunk feature map, and the first trunk feature map in the manner of every two adjacent trunk feature maps, and sequentially obtains the fourth aggregated feature map, the third aggregated feature map, the second aggregated feature map, and the first aggregated feature map.

[0065] The feature aggregation module 30 is specifically used to: perform an upsampling operation on the fifth trunk feature map, perform a feature map splicing operation on the upsampled fifth trunk feature map and the fourth trunk feature map, and after splicing, sequentially perform a self-attention module weighted operation and a ReLu activation function activation operation to obtain a fourth aggregated feature map;

[0066] An upsampling operation is performed on the fourth trunk feature map, and a feature map splicing operation is performed on the upsampled fourth trunk feature map and the third trunk feature map. After the splicing, a weighted operation operation of a self-attention module and an activation operation of a ReLu activation function are sequentially performed to obtain a third aggregated feature map;

[0067] An upsampling operation is performed on the third backbone feature map, and a feature map splicing operation is performed on the upsampled third backbone feature map and the second backbone feature map. After the splicing, a weighted operation operation of a self-attention module and an activation operation of a ReLu activation function are sequentially performed to obtain a second aggregated feature map;

[0068] An upsampling operation is performed on the second backbone feature map, and a feature map splicing operation is performed on the upsampled second backbone feature map and the first backbone feature map. After splicing, a self-attention module weighted operation and a ReLu activation function activation operation are sequentially performed to obtain a first aggregated feature map.

[0069] The decoding and segmentation module 40 includes three groups of decoder units and one group of segmentation prediction units. The three groups of decoder units are respectively called the third decoder unit, the second decoder unit and the first decoder unit. The third decoder unit, the second decoder unit, the first decoder unit and the segmentation prediction unit are arranged in sequence.

[0070] The decoder unit is used for feature decoding and restoration of segmented regions.

[0071] The third decoder unit is used to upsample the fourth aggregate feature map, fuse and splice the third aggregate feature map and the upsampled fourth aggregate feature map, and perform feature decoding on the fused and spliced ​​feature map to restore the segmented area to obtain a third decoded feature map;

[0072] The second decoder unit is used to upsample the third decoding feature map, fuse and splice the second aggregated feature map and the upsampled third decoding feature map, perform feature decoding on the fused and spliced ​​feature map, restore the segmented area, and obtain a second decoding feature map;

[0073] The first decoder unit is used to upsample the second decoding feature map, fuse and splice the first aggregated feature map and the upsampled second decoding feature map, perform feature decoding on the fused and spliced ​​feature map, and restore the segmented area to obtain the first decoding feature map;

[0074] Among them, the segmentation prediction unit is used to upsample the first decoded feature map (the result of N-2 decoding operations) to restore it to the same size h×w as the first target image, and then obtain the segmentation prediction map after activation by the Sigmoid activation function.

[0075] See also Figure 3 This is a specific structural example of the decoder unit, which includes a position encoding layer, a self-attention mechanism layer and a feedforward neural network layer.

[0076] Specifically, the decoder unit includes: 1 upsampling layer for upsampling the aggregate feature map from low resolution to high resolution; 1 feature concatenation layer for concatenating the output result of the upsampling layer and the aggregate feature map; 1 position encoding layer for providing spatial position information for each input feature block; 1 self-attention mechanism layer for capturing the global and local feature interactions between input feature blocks, and its input includes the output of the position encoding layer; 1 feedforward neural network layer for performing nonlinear transformation on the features output by the self-attention mechanism layer; 2 layer normalization layers, one layer normalization layer for normalizing the output of the self-attention mechanism layer, and the other layer normalization layer for normalizing the output of the feedforward neural network layer; 2 residual connection branches, one for retaining input feature information in the self-attention mechanism layer, and the other for retaining input feature information between feedforward neural network layers. The feature block is the fused and concatenated feature map described above, and the normalized output of the feedforward neural network layer is the decoding feature map.

[0077] The multi-group category prediction module 50 includes a feature fusion unit, three decision network branches, a probability matrix aggregation unit, and a constant matrix correction unit. The three decision network branches are respectively an origin level decision network branch (a first decision network branch), an echo level decision network branch (a second decision network branch), and a disease classification decision network branch (a third decision network branch).

[0078] The feature fusion unit is used to receive the first aggregated feature map, the second aggregated feature map, and the third aggregated feature map obtained by the feature aggregation module 30, align the dimensions of the three aggregated feature maps through a bilinear interpolation operation, and then perform a Concat fusion splicing operation to obtain an aggregated feature map. Figure 1 , aggregate features Figure 1 After a weighted layer of self-attention mechanism, the first fused encoding feature map is obtained.

[0079] Here, the feature fusion unit may also fuse all aggregated feature maps, and this embodiment does not limit the number of fused aggregated feature maps.

[0080] The decision network branch is used to perform decision classification according to the first fusion coding feature map to obtain a classification probability result.

[0081] The decision network branch, each of which contains: three sets of Transformer coding layers, the Transformer coding layer is used to continuously extract and deepen the global context information and key feature representation, and the feature is obtained after the three sets of Transformer coding layers are operated Figure 1 ; 1 fully connected layer, used to convert the features obtained by the above three sets of Transformer encoding layers Figure 1 Converted into a one-dimensional feature vector; 1 probability output layer, used to convert the one-dimensional feature vector into a category probability representation, that is, output the corresponding classification probability result.

[0082] The classification probability result includes the classification result and the classification probability. The classification probability results output by the first, second and third decision network branches are:

[0083] Origin level classification probability result: 3 categories are preset, namely, the 2nd layer, the 3rd layer, and the 4th layer. It can be understood that this is an example. In this embodiment, the origin level is preset to be only one of the 2nd layer, the 3rd layer, and the 4th layer. The expression is:

[0084] P A =[ p a1 , p a2 , p a3 ]

[0085] in, It represents the probability result of origin level classification. The probability of origin level classification being the second level is , the probability of the origin level being classified as level 3 is , the probability of the origin level being classified as level 4 is ;

[0086] Echo level classification probability result: In this embodiment, three categories are preset, namely high echo, low echo and medium echo, and the expression is:

[0087] P B =[ p b1 , p b2 , p b3 ]

[0088] in, represents the probability result of echo level classification. The probability of echo level classification as high echo is , the probability of the echo level being classified as low echo is , the probability of the echo level being classified as medium echo is ;

[0089] Disease classification probability results: 7 categories are preset here, namely stromal tumor, leiomyoma, schwannoma, lipoma, ectopic pancreas, neuroendocrine tumor, lymphoma, and the expression is:

[0090] P C =[ p c1 , p c2 , p c3 , p c4 , p c5 , p c6 , p c7 ]

[0091] in, represents the disease classification probability result, They represent the probability of the disease being classified as stromal tumor, leiomyoma, neurilemmoma, lipoma, ectopic pancreas, neuroendocrine tumor, and lymphoma, respectively.

[0092] The probability matrix aggregation unit constructs an aggregated probability matrix based on three dimensions: the origin level classification probability result, the echo level classification probability result, and the disease classification probability result. Specifically, the three classification results are converted into a three-dimensional matrix representation to obtain a three-dimensional classification probability matrix, and the three dimensions correspond to the origin level classification, the echo level classification, and the disease classification. Since the origin level presets 3 categories, the echo level presets 3 categories, and the disease presets 7 categories, the dimension of the matrix is ​​3×3×7, and according to the actual classification probability results, it is mapped to the matrix to obtain the aggregated probability matrix. In this embodiment, the x-axis of the aggregated probability matrix represents the origin level classification probability result, the y-axis represents the echo level classification probability result, and the z-axis represents the disease classification probability result. The mapping method is to calculate the joint probability of the origin level classification probability result, the echo level classification probability result, and the disease classification probability result, and then assign them to the corresponding unit positions of the matrix.

[0093] The aggregate probability matrix is ​​the sum of the origin level classification probability results, echo level classification probability results, and disease classification probability results. The unit of the aggregate probability matrix is ​​the sum of the classification probability of the origin level classification result, the classification probability of the echo level classification result, and the classification probability of the disease classification result corresponding to the unit. The details are as follows:

[0094] M(i,j,k) = P A [i]+ P B [j]+ P C [k]

[0095] in, represents the aggregation probability matrix, P A [i] represents the classification probability of the i-th classification result in the origin level classification probability result, P B [j] represents the classification probability of the jth classification result in the echo level classification probability result, P C [k] Represents the classification probability of the kth classification result in the disease classification probability result, i, j, k are all positive integers, (i, j, k) represents the coordinate point of the aggregation probability matrix, and the unit m of the aggregation probability matrix It can be expressed as: The unit m of the aggregation probability matrix = [ p a i , p b j , p ck ] .

[0096] The constant matrix correction unit is used to correct the aggregation probability matrix according to a preset constant matrix to obtain a corrected origin level classification probability result, a corrected echo level classification probability result, and a corrected disease classification probability result.

[0097] The constant matrix correction unit inputs the aggregation probability matrix and performs weighted multiplication with the preset constant matrix. It can be understood that the dimensions of the preset constant matrix and the aggregation probability matrix are consistent, and the units of the constant matrix are preset weights, and the weights are the occurrence probability (or weighted value determined according to the occurrence probability) of the combination of the origin level classification results, echo level classification and disease classification results in clinical experience. Figure 4 This is an example of some clinical data that can be used for reference in the existing embodiment. For example, in common clinical experience, the second layer origin, low echo, and leiomyoma are fixed combinations that often appear. In the constant matrix, the coordinate (0, 1, 1) will be set with a higher weight value. Similarly, if a certain coordinate point is a combination that will never appear in the clinic, then the coordinate point is set to 0. After the weighted fusion operation, the final aggregation probability matrix is ​​obtained, which is called the corrected aggregation probability matrix, that is, the corrected origin level classification probability result, the corrected echo level classification probability result, and the corrected disease classification probability result are obtained.

[0098] See also Figure 5 , exemplifies methods for endoscopic ultrasound lesion assessment, including:

[0099] S1. Automatically detect targets on the ultrasonic endoscope image using a target detection model to obtain a first target image;

[0100] S2, extracting features layer by layer on the first target image to obtain multiple (N, N ≥ 5) trunk feature maps;

[0101] S3, fuse (every two) adjacent order backbone feature maps to obtain multiple (N-1) aggregate feature maps;

[0102] S4, performing multiple (N-2) decoding operations based on the aggregated feature map to obtain a segmentation prediction map;

[0103] S5. Calculate the probability of origin classification, the probability of echo level classification, and the probability of disease classification according to the aggregated feature graph;

[0104] S6, calculating the joint probability of the origin level classification probability results, the echo level classification probability results and the disease classification probability results and constructing a three-dimensional aggregation probability matrix based on the joint probability results;

[0105] S7. Correct the aggregation probability matrix according to a preset constant matrix to obtain a corrected origin level classification probability result, a corrected echo level classification probability result, and a corrected disease classification probability result.

[0106] It should be understood that the above method may vary according to different execution processes. For example, the execution order of S4 is not limited. Therefore, the above method does not represent or imply that all steps must be executed in this order. Ordinary technicians in this field can transform or change the execution order of the above steps based on the present invention. Some embodiments of the above method are illustrated below.

[0107] When the endoscopic ultrasound lesion assessment method is specifically implemented, the endoscopic ultrasound lesion assessment system in any of the above embodiments may be referred to to implement lesion assessment, and the specific implementation steps will not be repeated here.

[0108] The present invention also provides a storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the steps of the method for evaluating lesions under ultrasound endoscopy are implemented.

[0109] The effects of the endoscopic ultrasound lesion assessment system, method and storage medium of the present invention are:

[0110] The present invention designs a feature extraction module 20 and a feature aggregation module 30, which specifically adopts multiple groups of Transformer network units to perform feature extraction, and uses hierarchical feature fusion technology to utilize multi-scale and multi-level features to improve the accuracy of image segmentation and target detection.

[0111] The present invention optimizes the design of the decoder and the segmentation prediction unit: the lesions can be accurately segmented; by designing a decoder unit and a segmentation prediction unit including a self-attention mechanism, the accuracy and efficiency of image segmentation are effectively improved.

[0112] The present invention designs multiple groups of category prediction and probability matrix aggregation technology: through multiple groups of decision network branches and probability matrix aggregation units, multi-dimensional classification of origin level, echo level and disease type is comprehensively realized, comprehensive lesion assessment information is provided, and the system complexity is simplified; the correction of the aggregated probability matrix by a constant matrix is ​​used to improve the multi-category classification accuracy of the model, and it can still maintain high precision when dealing with different patients and complex lesion conditions, and has good robustness and generalization performance.

[0113] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0114] The present invention is described with reference to flowcharts and / or block diagrams of methods, systems, and storage media according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

Claims

1. The endoscopic ultrasound lesion assessment system is characterized by: include: An ultrasonic endoscope target recognition module is used to automatically detect targets on the ultrasonic endoscope image using a target detection model to obtain a first target image; A feature extraction module, used to extract features of the first target image layer by layer to obtain multiple trunk feature maps; A feature aggregation module is used to fuse the adjacent main feature maps to obtain multiple aggregated feature maps; A decoding and segmentation module is used to perform multiple decoding operations based on the aggregated feature map to obtain a segmentation prediction map; A multi-group category prediction module is used to perform calculations based on the aggregated feature graph to obtain the origin level classification probability result, the echo level classification probability result, and the disease classification probability result, calculate the joint probability of the origin level classification probability result, the echo level classification probability result, and the disease classification probability result, and construct a three-dimensional aggregated probability matrix based on the joint probability; correct the aggregated probability matrix according to a preset constant matrix to obtain the corrected origin level classification probability result, the corrected echo level classification probability result, and the corrected disease classification probability result; The multiple groups of category prediction modules include: A feature fusion unit, used for fusing and splicing the aggregate feature graphs and weighting the fused and spliced ​​aggregate feature graphs to obtain a first fused coding feature graph; The first decision network branch is used to perform decision classification of the origin level according to the first fusion coding feature map to obtain an origin level classification probability result; The second decision network branch is used to make a decision classification of the echo level according to the first fusion coding feature map to obtain an echo level classification probability result; The third decision network branch is used to make a decision classification of the disease according to the first fusion coding feature map to obtain a disease classification probability result; The probability matrix aggregation unit is used to construct a three-dimensional aggregation probability matrix using the origin level classification probability result, the echo level classification probability result and the disease classification probability result as dimensions; the unit of the aggregation probability matrix represents the joint probability of the classification probability of the origin level classification result, the classification probability of the echo level classification result and the classification probability of the disease classification result corresponding to the unit; A constant matrix correction unit is used to perform a weighted fusion operation on the aggregate probability matrix and a preset constant matrix to obtain a corrected origin level classification probability result, a corrected echo level classification probability result, and a corrected disease classification probability result; The feature fusion unit is specifically used to align several aggregate feature map dimensions through a bilinear interpolation operation, perform a fusion and splicing operation after alignment to obtain an aggregate feature map 1, and input the aggregate feature map 1 into a self-attention mechanism weighted layer to obtain a first fused encoding feature map; The first decision network branch, the second decision network branch and the third decision network branch all include: three groups of Transformer encoding layers, the input of which is the first fused encoding feature map, which is used to continuously extract and deepen the global context information and key feature representation to obtain feature map 1; 1 fully connected layer, which is used to convert the feature map 1 into a one-dimensional feature vector; 1 probability output layer, which is used to convert the one-dimensional feature vector into a category probability representation and output a classification probability result; The feature extraction module includes 5 groups of Transformer network units arranged in sequence, the first group of Transformer network units is used to implement feature extraction on the first target image through convolution operation to obtain a first trunk feature map, the second group of Transformer network units is used to implement feature extraction on the first trunk feature map through convolution operation to obtain a second trunk feature map, the third group of Transformer network units is used to implement feature extraction on the second trunk feature map through convolution operation to obtain a third trunk feature map, the fourth group of Transformer network units is used to implement feature extraction on the third trunk feature map through convolution operation to obtain a fourth trunk feature map, and the fifth group of Transformer network units is used to implement feature extraction on the fourth trunk feature map through convolution operation to obtain a fifth trunk feature map; The feature aggregation module is used to perform feature fusion operations on the fifth trunk feature map, the fourth trunk feature map, the third trunk feature map, the second trunk feature map, and the first trunk feature map in the manner of performing feature fusion operations on every two adjacent trunk feature maps, and sequentially obtain a fourth aggregated feature map, a third aggregated feature map, a second aggregated feature map, and a first aggregated feature map; performing the feature fusion includes sequentially performing upsampling and splicing operations, self-attention module weighted operation operations, and ReLu activation function activation operations; The decoding segmentation module is specifically used to: fuse and splice the third aggregate feature map and the upsampled fourth aggregate feature map, and perform feature decoding after fusion and splicing to obtain a third decoding feature map; fuse and splice the second aggregate feature map and the upsampled third decoding feature map, and perform feature decoding after fusion and splicing to obtain a second decoding feature map; fuse and splice the first aggregate feature map and the upsampled second decoding feature map, and perform feature decoding after fusion and splicing to obtain a first decoding feature map; perform an upsampling operation on the first decoding feature map to restore it to the size of the first target image, and then activate it through an activation function to obtain a segmentation prediction map; The decoding segmentation module includes multiple groups of decoder units and a group of segmentation prediction units. The decoder unit includes: a position encoding layer for providing spatial position information for each input feature block; a self-attention mechanism layer for capturing global and local feature interactions between input feature blocks; a feedforward neural network layer for performing nonlinear transformation on the features output by the self-attention mechanism layer and outputting a decoding feature map; the segmentation prediction unit is used to upsample the results of multiple decoding operations, restore them to the same size as the first target image, and then obtain a segmentation prediction map after activation by an activation function.

2. A method for evaluating lesions under endoscopic ultrasound, characterized in that: include: Automatically detecting targets on the ultrasound endoscope image using the target detection model to obtain a first target image; Using a feature extraction module to extract features of the first target image layer by layer to obtain multiple backbone feature maps; The adjacent main feature maps are fused to obtain multiple aggregated feature maps; The decoding segmentation module is used to perform multiple decoding operations based on the aggregated feature map to obtain a segmentation prediction map; According to the aggregate feature map, operations are performed to obtain the probability results of origin level classification, the probability results of echo level classification, and the probability results of disease classification; specifically: the aggregate feature map is fused and spliced ​​and the fused and spliced ​​aggregate feature map is weighted to obtain a first fused coding feature map; the first decision network branch performs decision classification of the origin level according to the first fused coding feature map to obtain the probability result of origin level classification; the second decision network branch performs decision classification of the echo level according to the first fused coding feature map to obtain the probability result of echo level classification; The third decision network branch performs decision classification of the disease according to the first fusion coding feature map to obtain a disease classification probability result; The first decision network branch, the second decision network branch and the third decision network branch all include: three groups of Transformer encoding layers, the input of which is the first fused encoding feature map, which is used to continuously extract and deepen the global context information and key feature representation to obtain feature map 1; 1 fully connected layer, which is used to convert the feature map 1 into a one-dimensional feature vector; 1 probability output layer, which is used to convert the one-dimensional feature vector into a category probability representation and output a classification probability result; Calculate the joint probability of the origin level classification probability results, the echo level classification probability results and the disease classification probability results and construct a three-dimensional aggregation probability matrix based on them; specifically: use the origin level classification probability results, the echo level classification probability results and the disease classification probability results as dimensions to construct a three-dimensional aggregation probability matrix; the unit of the aggregation probability matrix represents the joint probability of the classification probability of the origin level classification result, the classification probability of the echo level classification result and the classification probability of the disease classification result corresponding to the unit; Correcting the aggregation probability matrix according to a preset constant matrix to obtain a corrected origin level classification probability result, a corrected echo level classification probability result, and a corrected disease classification probability result; Correcting the aggregation probability matrix according to the preset constant matrix specifically comprises: performing a weighted fusion operation on the aggregation probability matrix and the preset constant matrix; The feature extraction module includes 5 groups of Transformer network units arranged in sequence, the first group of Transformer network units is used to implement feature extraction on the first target image through convolution operation to obtain a first trunk feature map, the second group of Transformer network units is used to implement feature extraction on the first trunk feature map through convolution operation to obtain a second trunk feature map, the third group of Transformer network units is used to implement feature extraction on the second trunk feature map through convolution operation to obtain a third trunk feature map, the fourth group of Transformer network units is used to implement feature extraction on the third trunk feature map through convolution operation to obtain a fourth trunk feature map, and the fifth group of Transformer network units is used to implement feature extraction on the fourth trunk feature map through convolution operation to obtain a fifth trunk feature map; The specific process of fusing the main feature maps of adjacent order to obtain multiple aggregated feature maps is as follows: the fifth main feature map, the fourth main feature map, the third main feature map, the second main feature map, and the first main feature map are operated in a feature fusion manner according to every two main feature maps of adjacent order, and the fourth aggregated feature map, the third aggregated feature map, the second aggregated feature map, and the first aggregated feature map are obtained in sequence; the feature fusion includes sequentially performing upsampling and splicing operations, self-attention module weighted operation operations, and ReLu activation function activation operations; The specific process of using the decoding segmentation module to perform multiple decoding operations based on the aggregate feature map to obtain the segmentation prediction map is as follows: the third aggregate feature map and the upsampled fourth aggregate feature map are fused and spliced, and feature decoding is performed after fusion and splicing to obtain the third decoding feature map; the second aggregate feature map and the upsampled third decoding feature map are fused and spliced, and feature decoding is performed after fusion and splicing to obtain the second decoding feature map; the first aggregate feature map and the upsampled second decoding feature map are fused and spliced, and feature decoding is performed after fusion and splicing to obtain the first decoding feature map; the first decoding feature map is upsampled to restore the size of the first target image, and then activated by an activation function to obtain the segmentation prediction map; The decoding segmentation module includes multiple groups of decoder units and a group of segmentation prediction units. The decoder unit includes: a position encoding layer for providing spatial position information for each input feature block; a self-attention mechanism layer for capturing global and local feature interactions between input feature blocks; a feedforward neural network layer for performing nonlinear transformation on the features output by the self-attention mechanism layer and outputting a decoding feature map; the segmentation prediction unit is used to upsample the results of multiple decoding operations, restore them to the same size as the first target image, and then obtain a segmentation prediction map after activation by an activation function.

3. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for evaluating lesions under ultrasound endoscopy described in claim 2 are implemented.

Images