Medical image lesion classification method and system

Through the method of enhancing global class activation graph and feature data, combined with U-Net and Transformer models, the coordinated optimization of medical image segmentation and classification is achieved, which solves the problems of low diagnostic accuracy and high burden on doctors in the existing technology, and improves medicine Image processing accuracy and robustness.

CN118762241BActive Publication Date: 2024-11-29SHANDONG UNIV
View PDF 16 Cites 0 Cited by

Patent Information

Application Number
CN202411244455.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-29
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

Existing medical image processing technology is difficult to effectively deal with sensor noise, individual differences and unclear soft tissue boundaries, resulting in low diagnostic accuracy and high burden on doctors to read films, and deep learning has shortcomings in local feature extraction and contextual information expression.

Method used

By obtaining the global class activation map, assisting the segmentation network to learn location information, enhancing the feature data, expanding the data volume of a few categories, and re-passing it into the segmented network through the global class activation map, realizing the coordinated optimization of segmentation and classification, combining U-Net and The Transformer model uses multi-scale feature fusion and boundary constraint loss functions, uses genetic algorithm ideas to filter and copy feature parameters, and performs full connection learning.

Benefits of technology

Improve the accuracy and robustness of medical image segmentation and classification, enhance the ability to express high-dimensional information of medical images, reduce artificial judgment errors, improve the accuracy and reliability of diagnosis, and is suitable for segmentation and diagnosis of different goals. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118762241B_ABST
    Figure CN118762241B_ABST
Patent Text Reader

Abstract

The present invention belongs to the general field of image processing technology. The present invention provides a method and system for classifying lesions in medical images. By acquiring a global class activation map, the location information of the target area can be completely located, and the classification and segmentation networks can be assisted in learning the location information; by enhancing feature data, the amount of data of a few categories in the full-class data can be expanded; finally, the global class activation map is re-transmitted into the segmentation network, thereby achieving the collaborative optimization of segmentation and classification, and playing a role of mutual collaborative improvement.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the general field of image processing technology, and in particular to a medical image lesion classification method and system. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] Medical imaging technology plays a vital role in the doctor's diagnosis process. With the advancement of technology, new medical imaging methods including computed tomography (CT), magnetic resonance imaging (MRI), X-ray and ultrasound examination have emerged and have been widely used. These modern imaging technologies, combined with advanced image processing methods, provide doctors with an intuitive, real-time, non-invasive and non-destructive way to detect human lesions. However, traditional image segmentation techniques such as edge detection, region growing, and local thresholding often have difficulty in coping with challenges such as sensor noise, individual differences, and unclear soft tissue boundaries in medical imaging. Therefore, in order to improve the accuracy of diagnosis, reduce the burden of doctors reading films, and reduce human judgment errors, computer-aided diagnosis (CAD) technology has received widespread attention and continues to develop in the research field. In the CAD system, accurate segmentation of organs or tissues in medical images is not only a key step in the workflow, but also a core link in medical image processing. The key lies in how to quantitatively analyze clinical parameters.

[0004] With the advancement of computer vision technology, deep learning has become the core technology of medical image processing. Convolutional Neural Networks (CNN) abandons traditional manual feature annotation and realizes automatic feature extraction; however, CNN has certain deviations when extracting local features, lacks in-depth understanding of the overall structure of the image, and is insufficient in the continuous expression of contextual information. Summary of the invention

[0005] In order to address the shortcomings of the prior art, the present invention provides a medical image lesion classification method and system, which can completely locate the position information of the target area by obtaining the global class activation map, and assist the segmentation network in learning the position information; through feature data enhancement, the amount of data of minority categories in the full-class data is expanded; finally, the global class activation map is re-transmitted into the segmentation network to achieve collaborative optimization of segmentation and classification, thereby playing a role of mutual collaborative improvement.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for classifying lesions in medical images.

[0008] A medical image lesion classification method includes the following processes:

[0009] The three-dimensional CT image is segmented by using a pre-trained segmentation network to obtain a segmented lesion target area, the lesion target area is processed by using an edge bounding box to obtain a positioning image, and a distance matrix of the lesion target area is calculated; in the segmentation network, the symmetric KL divergence loss of the class activation map under different window widths and window positions is calculated to obtain a global class activation map and pass it to the segmentation network to provide positioning information;

[0010] A medical image lesion classification result is obtained based on the multi-scale features obtained by the segmentation network, the distance matrix, the positioning image and the pre-trained classification network.

[0011] As a further limitation of the first aspect of the present invention, when training the classification network, the feature parameters after global pooling of each cycle are saved in the feature warehouse, the change in entropy value of the same feature parameter in different cycles is calculated, the feature parameters with strong expression categories are screened out, the mean of the screened out feature parameters is used as the basic feature parameter for expressing the global category, two feature parameters of the same category are randomly copied according to a set ratio to fill the position of the feature parameters that have not been screened out, to obtain enhanced feature data, and the enhanced feature data is input into the classification network for fully connected learning.

[0012] As a further limitation of the first aspect of the present invention, the three-dimensional CT image is adjusted to multiple preset window widths and window positions, and different network models are used for segmentation and classification prediction. The classification prediction results under various window widths and window positions are obtained and the entropy value uncertainty is calculated. When the number of network models whose entropy value uncertainty is less than the set threshold is greater than 1, the parameters of the network models greater than 1 are fused using an exponentially weighted moving average method. When the number of network models whose entropy value uncertainty is less than the set threshold is equal to 1, the network model is retained as the final network model; wherein each of the network models includes a segmentation network and a classification network.

[0013] As a further limitation of the first aspect of the present invention, a global class activation map is obtained by calculating the symmetric KL divergence loss of the class activation map, comprising:

[0014] When the classification model identifies category k, the class activation map of input image i for:

[0015] ;

[0016] in, For channels c The feature maps in the corresponding categories k The weight of Represents the input image i Features before the global average pooling layer, h For the input image i Height, w For the input image i The width of C is the total number of channels;

[0017] Constructing a loss function based on class activation maps ,include:

[0018] ;

[0019] ;

[0020] ;

[0021] , For the input image i No. j The class activation map of each category, K is the total number of categories, For the input image i No. k The class activation map of each category, m is the total number of input images, when near When , the loss should be close to zero, and when they are far apart, the loss should be close to infinity. During training, right The gradient of is:

[0022] ;

[0023] when When , the optimization function obtains the optimal solution, which is the global class activation map. exist near When θ is close to zero, the global class activation map is used to supervise the localization. The obtained global class activation map is passed to the upsampling process of the segmentation network to assist the segmentation task.

[0024] As a further limitation of the first aspect of the present invention, the entropy change of the same feature in different cycles is calculated , filter out characteristic parameters with strong expression categories, including:

[0025] ;

[0026] ;

[0027] in, represents the network weight, represents the network weight after global average pooling, GAP(·) represents the global pooling operation, and retains the entropy value change for the same data in different training cycles. Less than the set threshold The average value of the retained features is calculated as the basic feature expressing the global category.

[0028] As a further limitation of the first aspect of the present invention, before segmenting the three-dimensional CT image using the pre-trained segmentation network, a pre-processing and data enhancement process is also included, including:

[0029] The 3D CT image is resampled to the average voxel spacing of the corresponding data set to maintain the spatial semantics of the 3D CT image. Assuming that a d-layer w×h ​​3D CT image is input, w is the width and h is the height, then the total number of voxels N is N=w×h×d. The w×h×d voxels are divided into non-overlapping patches, where each patch size is p×m×m, and the number of patches is t=(w / p)×(h / m)×(d / m);

[0030] The resampled 3D CT images were cropped to the corresponding percentile of the entire image value, and the Z-score was normalized using the mean and standard deviation of each modality image;

[0031] Remove sections without any organs before or after the entire sequence;

[0032] The three-dimensional CT images are enhanced using data augmentation methods such as random elastic deformation, random rotation, random scaling or random mirroring.

[0033] As a further limitation of the first aspect of the present invention, segmenting the three-dimensional CT image using a pre-trained segmentation network includes:

[0034] The segmented patch is linearly embedded into a C-dimensional vector through a convolutional layer, which is recorded as a token;

[0035] The tokens are normalized and normalized through layers, and then feature extraction is performed using regular windows and moving windows. The regular window extracts information from independent tokens, and the moving window uses a shifted window to bridge adjacent tokens.

[0036] Merge the extracted features with adjacent tokens in a non-overlapping manner along the spatial axis to obtain complete spatial tokens;

[0037] After every two feature merging, a skip connection is introduced, and the skip connection is connected to the encoder of the same order as the upsampling. The features of the skip connection are subjected to the feature fusion module, stripe and surface pooling to obtain the key K and value V after enhanced attention as a supplement to the upsampling process.

[0038] In a second aspect, the present invention provides a medical image lesion classification system.

[0039] A medical image lesion classification system, comprising:

[0040] The image segmentation unit is configured to: segment the three-dimensional CT image using a pre-trained segmentation network to obtain a segmented lesion target area, process the lesion target area using an edge bounding box to obtain a positioning image, calculate a distance matrix of the lesion target area, and in the segmentation network, calculate the symmetric KL divergence loss of the class activation map under different window widths and window positions to obtain a global class activation map and pass it to the segmentation network to provide positioning information;

[0041] The lesion classification unit is configured to obtain a medical image lesion classification result based on the multi-scale features obtained by the segmentation network, the distance matrix, the positioning image and the pre-trained classification network.

[0042] As a further limitation of the second aspect of the present invention, when training the classification network, the feature parameters after global pooling of each cycle are saved in the feature warehouse, the change in entropy value of the same feature parameter in different cycles is calculated, the feature parameters with strong expression categories are screened out, the mean of the screened out feature parameters is used as the basic feature parameter for expressing the global category, two feature parameters of the same category are randomly copied according to the set ratio to fill the position of the feature parameters that have not been screened out, to obtain enhanced feature data, and the enhanced feature data is input into the classification network for fully connected learning.

[0043] As a further limitation of the second aspect of the present invention, the three-dimensional CT image is adjusted to multiple preset window widths and window positions, and different network models are used for segmentation and classification prediction, respectively, to obtain classification prediction results under various window widths and window positions and calculate the entropy value uncertainty, when the number of network models whose entropy value uncertainty is less than a set threshold is greater than 1, the parameters of the network models greater than 1 are fused using an exponentially weighted moving average method, and when the number of network models whose entropy value uncertainty is less than a set threshold is equal to 1, the network model is retained as the final network model; wherein each of the network models includes a segmentation network and a classification network.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention trains and selects network models with different window widths and window positions in parallel, which can not only evaluate the classification results through uncertainty, but also further optimize the network model parameters by fusing the parameters under different perspectives, thereby improving the robustness of the network model. Different from the problem of single perspective of setting window width and window position based on traditional experience, the method proposed by the present invention has the characteristic of better data compatibility.

[0046] 2. The present invention combines the traditional U-Net segmentation network with the Transformer in natural language processing, and uses the Transformer's unique global information learning ability to make up for the shortcomings of U-Net, greatly improving the learning of the segmentation network model in the high-dimensional information expression of medical images, and laying the foundation for the accurate prediction of disease types.

[0047] 3. The present invention uses three-dimensional CT images and a feature selection network with enhanced edge learning to obtain accurate lesion prediction areas. The segmented mask is used to obtain the lesion location and distance matrix. The boundary constraint loss function converges to the target area boundary, and then the multi-scale features are fused into the classification network to obtain the category probability and class activation map CAM (Class Activation Mapping, CAM) of the lesion. By obtaining the CAM under different window widths and window positions, the symmetric KL divergence loss is calculated to obtain the CAM that represents the global state, and it is passed to the learning of the segmentation network to provide positioning information. The CAM map re-enters the feature selection network in the subsequent training sampling process so that the networks can promote each other, realize the coordinated optimization of segmentation and classification, and improve the segmentation and classification accuracy.

[0048] 4. The present invention considers feature information in three directions in three-dimensional space, and realizes cross-scale feature learning by introducing a multi-scale feature fusion module. While improving the segmentation results, it provides richer input information for classification. It optimizes edge learning through a mixed loss function, and has a better representation for learning the edges of complex lesions such as hollowing. It helps the classification task learn better lesion geometry through the central axis distance matrix, improves classification accuracy, and can be widely used in the segmentation of different targets to assist in diagnosis in various medical fields.

[0049] 5. The present invention combines the idea of ​​genetic algorithm and generates extended data at the feature level through screening and copying. Specifically, the feature parameters after the model training cycle are retained in a feature warehouse, the change in entropy value of the same data in different cycles is calculated, and the features of the expression category are screened out. The mean of these features is then used as the feature enhancement benchmark, and two data features of this category are randomly copied to fill the vacant feature positions. Finally, the feature-enhanced data is input into the classification network for fully connected learning. This method can expand data of a few categories and prevent the results from overfitting or biased towards the majority category.

[0050] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0052] Figure 1 A flow chart of a method for segmenting medical images provided in Example 1 of the present invention;

[0053] Figure 2 is a schematic diagram of a patch segmentation method according to Embodiment 1 of the present invention;

[0054] Figure 3 is an overall flow chart of the classification network of Example 1 of the present invention;

[0055] Figure 4 It is a schematic diagram of feature data enhancement according to Example 1 of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0057] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0058] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0059] Embodiment 1:

[0060] This implementation proposes a medical image lesion classification method, such as Figure 1 and Figure 3 As shown, the following process is included:

[0061] S1: Segmenting the three-dimensional CT image using a pre-trained segmentation network to obtain a segmented lesion target area, processing the lesion target area using an edge bounding box to obtain a positioning image, and calculating a distance matrix of the lesion target area;

[0062] S2: Obtain a medical image lesion classification result based on the multi-scale features obtained by the segmentation network, the distance matrix, the positioning image and the pre-trained classification network.

[0063] In this implementation, specifically, the three-dimensional CT image is divided into non-overlapping three-dimensional small blocks (i.e., patches), such as Figure 2As shown in the figure, each 3D patch has the same size, and the number of patches is determined by dividing the size of the original image by the patch size. Each patch feature is linearly embedded into a C-dimensional vector (token). Two types of windows (regular window and moving window) are used to achieve feature extraction. The regular window realizes information extraction from independent tokens, and the moving window uses the shifted window to bridge adjacent tokens. After each feature is extracted, the adjacent tokens are merged in a non-overlapping manner along the spatial axis, showing that the number of tokens is reduced exponentially but the number of channels obtained by linear embedding is doubled.

[0064] In the upsampling part, each decoder module will receive the token generated by the previous decoder, as well as K (key) and V (value) after stripe and face pooling in the encoder at the same stage. This process occurs in the skip connection stage. The accuracy of the segmentation result obtained by the segmentation network of this implementation corresponds to the result of the current window width and window position, and the Dice coefficient of the real data is compared. When it is less than 0.9, it is considered that the current window width and window position setting is not suitable for this data set. Only networks and results greater than 0.9 are retained here. The segmentation results are post-processed to detect the common area coverage and calculate the connected domain, and remove outliers, noise and other information.

[0065] In this implementation, the three-dimensional CT image to be segmented is preprocessed and data enhanced, specifically including:

[0066] Step (1-1) resamples all CT images to the average voxel spacing of the corresponding dataset to maintain the spatial semantics of the three-dimensional CT image. Specifically, assuming that a three-dimensional CT image with d layers w×h is input, w is the width, h is the height, then the total number of voxels N is N=w×h×d, the w×h×d voxels are divided into non-overlapping patches, where the size of each patch is p×m×m, and the number of patches is t=(w / p)×(h / m)×(d / m);

[0067] Step (1-2) crops the resampled 3D CT image to the corresponding percentile of the entire image value, and performs Z-score normalization using the mean and standard deviation of each modality image;

[0068] Step (1-3) adjusts the window width and window position of the three-dimensional CT image; this implementation method takes the lumbar intervertebral disc CT image as an example, and adjusts the window width and window position to five ranges (W: 100, L: 50, W: 350, L: 40; W: 1400, L: -500; W: 1000, L: 400; W: 1000, L: -425);

[0069] Steps (1-4) remove the slices without any organs in the entire sequence by default.

[0070] Steps (1-5) perform data enhancement on the three-dimensional CT image using a data enhancement method of random elastic deformation, random rotation, random scaling or random mirroring.

[0071] In this implementation, a segmentation network is constructed with Transformer and U-Net networks as the backbone network, including 6 downsampling stages and 6 upsampling stages. Stripe pooling modules, face pooling modules, and multi-scale feature fusion (MFF) modules are added in series on the skip connections of the network to build the main framework of the segmentation network. Figure 1 In the figure, after Patch segmentation, the downsampling process is performed. The downsampling process includes the Transformer encoder and the Patch merging module connected in sequence, which are executed 6 times, and then input into the Transformer encoder and the Patch expansion module to enter the upsampling process. The upsampling process includes the Transformer encoder and the Patch expansion module connected in sequence, which are executed 6 times and input into the classifier to obtain the segmentation result. The specific segmentation processing flow includes:

[0072] Step (2-1) linearly embeds the cropped patch image into a C-dimensional vector through a convolutional layer, denoted as token;

[0073] Step (2-2) first normalizes the tokens by layer normalization (LN), and then uses two types of windows to extract features: regular windows and shifted windows. The former extracts information from independent tokens, while the latter uses shifted windows to bridge adjacent tokens.

[0074] Step (2-3) merges the extracted features with adjacent tokens in a non-overlapping manner along the spatial axis to obtain complete spatial tokens. Specifically, first connect each group of 2×2 adjacent tokens and linearly embed them to obtain features with doubled channels, retaining the complete image spatial feature information.

[0075] After every two feature merging in step (2-4), a skip-layer connection stage is introduced, which is connected to the encoder of the same order as the upsampling. In the skip-layer connection process, the complete merged features are retained, so the features can be enhanced through the feature fusion module (MMF), stripe and surface pooling to obtain the key K and value V after attention enhancement. Here, K and V refer to the key vector and the value vector, and their purpose is to supplement the upsampling process;

[0076] In the step (2-5), in the stripe pooling module, three stripe convolutions in different directions are used to extract feature maps in different directions for each scale feature, and the cross-sectional direction feature map, sagittal direction feature map and coronal direction feature map are extracted; and after adding and fusing the three different direction feature maps, the dimension is reduced by unit convolution and the feature weight is obtained by Sigmoid function, and the feature weight is multiplied with the input feature at the corresponding scale to obtain the weighted feature map; among them, when processing three-dimensional images, the traditional method often uses a cube convolution kernel, but in this implementation, combined with the data characteristics, the design sets the three-dimensional convolution kernel to a stripe convolution in three dimensions; for example, the traditional convolution kernel is 3×3×3, and the stripe convolution is 1×1×3, 1×3×1 and 3×1×1.

[0077] Step (2-6) In the face pooling module, three different directions of face convolution are used to extract feature maps of different directions from the weighted feature map output by the stripe pooling module, including the cross-sectional direction, sagittal direction and coronal direction; then the three different directions of feature maps are added and fused, and the dimension is reduced by unit convolution and the feature weight is obtained by Sigmoid activation function, and then multiplied with the input weighted feature map to obtain the final weighted feature map;

[0078] Among them, the surface convolution is to set the three-dimensional convolution kernel into a separate surface convolution in three dimensions; for example, the traditional convolution kernel is 3×3×3, and the surface convolution is 1×3×3, 3×3×1, and 3×1×3;

[0079] Among them, the Sigmoid activation function is:

[0080] (1);

[0081] in, x It is the feature map after dimensionality reduction through unit convolution.

[0082] In this implementation, the weighted feature map output by the face pooling module is input into the MFF module. The MFF module consists of a channel attention (Squeeze-and-Excitation Networks, SE-Net) module and a transposed channel attention (transposed SE-Net) module. It is set at the skip-layer connection of the segmentation network and is responsible for fusing multi-scale features. The feature with the largest weight is selected as the classification guidance information by the weight descending sorting method.

[0083] Step (2-7) The MFF module combines the traditional channel attention module with the transposed channel attention module, performs two operations on the same input feature and then adds them to obtain a composite feature; the segmentation network constructed in this implementation uses a total of 4 layers of skip connections, so a total of 4 MFF modules are set to extract composite features of different scales, and then a total of 16 layers of feature maps are selected as the final classification guidance features after sorting;

[0084] Among them, the channel attention module obtains the features through the global average pooling (GAP) and two fully connected layers. The feature map of ; where the number of neurons in the first fully connected layer is C / 16, and the number of neurons in the second fully connected layer is C. This advantage adds more nonlinear processing and can fit the complex relationship between channels; then, after the sigmoid activation function layer, we get Feature map; finally, the input feature is multiplied by Feature map, obtain feature maps with different importance of different channels, C is the number of channels;

[0085] The transposed channel attention module first transposes the input features to obtain a feature extraction strategy different from the channel attention. After that, it is similar to the traditional channel attention module, except that it needs to multiply the penalty coefficient at the end to convert the features back to the original expression. The specific process includes:

[0086] Step (3-1) In the transposed channel attention module, the weighted feature map output by the face pooling module is Before entering the MFF module, the feature map is converted into ; Capture valuable information from the height dimension to enhance each feature map tensor in the height dimension:

[0087] (2);

[0088] in, is the transpose operation, It is the weighted feature map output by the face pooling module, where C, H, and W are the number of channels, height, and width, respectively.

[0089] Step (3-2) uses global average Compressed feature map , to obtain the eigenvector :

[0090] (3);

[0091] Among them, i and j are both pixel points, which can be expressed as pixel points in the channel direction and width direction.

[0092] Step (3-3) uses two fully connected layers to reduce model complexity and improve generalization ability; first, the first fully connected layer (which contains The activation function) is used for dimensionality reduction, and then the second fully connected layer is used to restore the original size while completing feature learning. The specific expression is:

[0093] (4);

[0094] in, , is a hyperparameter; Represents two fully connected layer operations; represents the first fully connected layer; represents the second fully connected layer.

[0095] Steps (3-4) will and Multiply to get the recalibrated transposed feature map :

[0096] (5);

[0097] Steps (3-2) to (3-4) are performed in both the transposed channel attention module and the channel attention module.

[0098] Steps (3-5) In the transposed channel attention module, an additional penalty coefficient is used After the squeeze stimulus ,Then Restore to Dimensions:

[0099] (6);

[0100] Steps (3-6) transpose the output of the channel attention module And the output of the channel attention module After adding and averaging according to the channel dimension, the composite weighted feature is obtained ;

[0101] (7);

[0102] in, is the average function;

[0103] Steps (3-7) combine the weighted features The weights are sorted in descending order from the channel dimension. The MFF module in each skip layer connection selects the four channel features with the largest weights from the channel dimension, and finally uses The convolution kernel obtains a feature map with four layers of feature selection:

[0104] (8);

[0105] in, , represents the feature map selected from the channel dimension, Indicates that the sorting weight is from large to small.

[0106] In this implementation, after passing through the segmentation network, the segmentation mask of the lesion area is obtained; the maximum connected domain and bounding box regression are used to cut out each intervertebral disc separately; the label function is used to mark all connected areas of the binarized image; the return value of the label function has many attributes, including the area of ​​the connected area, the center coordinates, and the bounding box. This implementation uses the bounding box.

[0107] In this implementation, stripe pooling and surface pooling are used to perform targeted learning on flat and circular geometric features such as lumbar intervertebral discs and lung nodules from the XOZ, OYZ and XYO perspectives to maximize the utilization of features and solve the problem of weak classification boundaries of lesion areas under different degrees of loss in the target area.

[0108] For segmentation tasks, the segmentation of boundary parts is the most difficult. In order to better constrain the boundary of the segmentation result, this implementation method designs a composite boundary constraint function ;

[0109] (9);

[0110] in, is a hyperparameter responsible for adjusting the weight.

[0111] Specifically: Boundary segmentation loss function , the prediction probability of the boundary points and the area of ​​the connected domain where the boundary points are located are combined to solve the problem of difficult boundary segmentation in the target area; specifically:

[0112] (10);

[0113] =exp( )(11);

[0114] in, As a hyperparameter, the pixels of the lesion and the background are sorted according to the error after each batch of forward propagation, that is, the pixel with the largest error is selected; is the nth pixel; is the category of the nth pixel, 1 represents lesion, and 0 represents background; is the area weight of the connected domain, is the area of ​​the connected domain, is a constant, usually set to 85. The larger the area of ​​the connected domain, the greater the area weight of the connected domain. The smaller the size, the smaller the size, and vice versa; Indicates The background of the input image (i.e., CT image) The predicted probability of difficult-to-classify pixels; Indicates The lesion in the input image The predicted probability of difficult-to-classify pixels; is a hyperparameter that controls the difference, that is, strictly enforces .

[0115] Structural Similarity Loss Function , specifically:

[0116] (12);

[0117] in, , and , They are x and y The mean and variance of for x and y covariance; x is the reference image, y is the image to be evaluated, C1 and C2 are constants, the purpose is to avoid instability when the values ​​of x and y are close to 0.

[0118] Set similarity loss function , used to calculate the similarity between two samples, the value range is usually between 0 and 1, specifically:

[0119] =1- (13);

[0120] Among them, X represents the predicted value, Y represents the true value (represented by 0 or 1); |X⋂Y| represents the dot product of the corresponding elements of the two sets, and then the results of the element-by-element multiplication are added and summed.

[0121] In steps (3-8), after 6 downsamplings, the dimension of the input feature is doubled using linear mapping; then the feature size is restored to the size at the sixth downsampling layer through reshaping. The purpose of this is to expand the patch in high dimensions.

[0122] Steps (3-9) in the upsampling stage use the same method as downsampling but in the opposite direction.

[0123] In this implementation, the feature parameters of each cycle of the network are retained in a feature warehouse, and the change in entropy value of the same data in different cycles is calculated to screen out features with strong expression categories. Subsequently, the mean of these features is used as the feature enhancement benchmark, and the features of this type are randomly copied at a ratio of 1:1 (i.e., the set ratio) to fill the vacant positions. Finally, the feature-enhanced data is input into the classification network for fully connected learning.

[0124] The process of data enhancement based on genetic algorithm is as follows: Genetic algorithm mainly combines two methods of selection (screening) and replication to achieve optimization, such as Figure 4 As shown. Specifically, this module acts on the features obtained after global pooling through the backbone network (i.e., CNNs network, a convolutional neural network). The deep features of the network are selected as the target because in the classification task, the features closer to the fully connected layer tend to have a greater impact on the results, and can also more accurately express the high-dimensional features of the category. In the selection stage of the genetic algorithm, the system will screen out the best performing features. The present invention selects the entropy value uncertainty representation and replicates them based on these features to retain and enhance the information that has a greater impact on the classification results. In this way, not only can the classification performance of the model be improved, but also the key features can be captured and utilized more effectively, thereby improving the accuracy and reliability of the classification. The advantage of this method is that it can make full use of the expressive power of deep features, so that the model can make more accurate judgments when facing complex classification tasks. This method of combining selection and replication also provides strong support for subsequent feature optimization and model improvement. Specifically, it includes the following contents:

[0125] First, the hyperparameter N is preset to control the number of feature warehouses. The present invention sets N=10 here. As shown in formula (13), the entropy change of the feature at the same position in multiple cycles is calculated:

[0126] (14);

[0127] (15);

[0128] in, represents the network weight, Represents the network weight after global average pooling. For the results of the same data in different training cycles, the entropy value is calculated as the confidence auxiliary feature selection. Combined with the genetic characteristics in the genetic algorithm, the entropy value change is less than the set threshold. feature parameters, which will maximize the retention of the feature expression of the corresponding category.

[0129] After retaining the filtered feature parameters, the average value of these features is calculated as the basic feature for expressing the global category (it should be noted that the ratio of the retained mean feature parameters to the total number of features is 7:10. This step provides an initial framework for subsequent feature-level data expansion, ensuring the stability and accuracy of the expansion process. Subsequently, the replication operation in the genetic algorithm is applied. It is worth mentioning that since this classification task does not involve the introduction of new categories as perturbation factors, replication is selected as the main feature inheritance method to enhance the robustness of the model. Specifically, for those feature parameters that are not retained, two model parameters of the same category are randomly copied and evenly distributed to these unreserved feature positions at a ratio of 1:1 (i.e., the set ratio), thereby generating new feature-level-based data. This process not only simplifies the complexity of feature expansion, but also ensures that the newly generated features are diverse and representative.

[0130] The CAM under different window widths and window positions is obtained, and the CAM representing the global situation is obtained by calculating the symmetric KL divergence loss, thereby assisting the classification network model in learning global information. Finally, the global CAM is passed to the segmentation network learning (that is, the global CAM is superimposed with the downsampling result of the segmentation network and then input into the MFF module) to provide positioning information.

[0131] The global CAM process based on symmetric KL divergence is as follows: Inspired by the positioning ability of CAM in classification tasks, it is proposed to achieve global lesion attention by optimizing the distance of CAM generated by the same three-dimensional CT image in different lesion tasks. To this end, the present invention designs an optimization function to measure the distance of CAM generated by the same three-dimensional CT image according to different lesion tasks. It should be noted that global lesion localization is not targeted at data after feature enhancement. This is because the enhanced features obtained using the genetic algorithm idea and the features obtained by image extraction cannot obtain an interpretable CAM, so this part is not used in the present invention; in addition, in addition to the CAM in different tasks, the present invention also adds a CAM overlay class as additional supervisory information, which is obtained by averaging the CAM obtained by the same data in different tasks. The reason for adding this category information is that compared with the CAM of different tasks, the overlay class can better cover a larger target area and play a positive supervisory role in learning the global CAM; we assume represents the CAM of sample i when identifying category 𝑗, when near When , the loss should be close to zero; when they are far apart, the loss should be close to infinity. At the same time, the loss function should be symmetric because the exchange and Does not affect the corresponding distance.

[0132] Assume the weight matrix of the classification layer is , when the classification model identifies category k, the CAM of input image i can be calculated as follows:

[0133] (16);

[0134] in, is the weight of the feature map corresponding to category k in channel c, It represents the features of input image i before the global average pooling layer, and C is the total number of channels. Figure 3 middle, is the weight of the feature map in channel 1 corresponding to category 1, is the weight of the feature map in channel 2 corresponding to category 1, is the weight of the feature map in channel c corresponding to category 1, is the weight of the feature map corresponding to category k in channel 1, It is the weight of the feature map in channel 1 corresponding to category k.

[0135] The present invention constructs the following CAM-based loss function, and the specific expressions are shown in formulas (17), (18) and (19):

[0136] (17);

[0137] (18);

[0138] (19);

[0139] Obviously, this loss function is symmetric, that is ,also, It also satisfies two other properties, namely, assuming represents the CAM of sample i when identifying category 𝑗, when near When , the loss should be close to zero; when they are far apart, the loss should be close to infinity. During training, right The gradient of can be calculated as follows:

[0140] (20);

[0141] When , the optimization function obtains the optimal solution, that is, exist near When , it is close to zero, thereby achieving the role of global CAM supervision positioning. The obtained global CAM will also be passed to the upsampling process of the segmentation network to assist the segmentation task.

[0142] In this implementation, the entire three-dimensional segmentation result is regarded as a whole, which is composed of voxels. A distance matrix is ​​established based on a voxel-based geometric distance matrix, and the distance from the lesion boundary to the geometric center or the central axis is used as a metric.

[0143] The process of establishing the distance matrix specifically includes two methods, based on the geometric center and based on the central axis. The process based on the geometric center is as follows: (1) Extract the boundaries of each layer in the region of interest, establish a coordinate system for each point, and provide geometric information; (2) Calculate the Euclidean distance from the segmentation boundary voxel point to other voxels, and calculate the average distance to locate the center coordinates; (3) Randomly downsample the edge voxel points (this implementation method takes 200 points as an example) to obtain the edge feature points of each layer. While reducing the computational cost, randomly selecting feature points is conducive to improving the robustness of subsequent models; (4) Calculate the Euclidean distance from the edge voxel point to the center coordinate. It should be noted that the points that are not selected on the edge use the nearest neighbor clustering method to give the nearest point distance as a feature, and non-edge points (background points and internal points) are given a larger distance as an outlier, thereby obtaining a distance matrix.

[0144] The process based on the central axis is as follows: (1) Extract the boundaries of each layer in the region of interest, establish a coordinate system for each point, and provide geometric information; (2) Randomly downsample edge voxel points (200 points are used as an example in the lumbar disc embodiment) to obtain edge feature points of each layer. While reducing the computational cost, the random selection of feature points is conducive to improving the robustness of the subsequent model; (3) Use a combination of distance transform and dilation to calculate the distance from each foreground pixel to the background pixel. The result is an image in which the value of each pixel is its distance to the nearest background. The skeleton is extracted from the distance transform map by finding the local maximum point. These points represent the foreground pixels farthest from the background, that is, part of the skeleton; (4) Uniformly sample 200 points from the extracted skeleton and calculate the Euclidean distance from the edge voxel point to the central coordinate. It should be noted that the points that are not selected on the edge use the nearest neighbor clustering method to give the nearest point distance as a feature, and non-edge points (background points and internal points) are given a larger distance as abnormal points, thereby obtaining a distance matrix.

[0145] In this implementation, a classification network is constructed based on the 3D-Resnet18 layer as the main framework, and the SE channel attention module (Squeeze-and-Excitation block) and pyramid pooling (Pyramid Scene Parseing, PSP) module are added to increase the feature learning of the intervertebral disc lesion site. The feature map filtered by the MFF module, the cropped target area and the distance matrix are fused as the input of the classification network for classification prediction.

[0146] Specifically: SE channel attention modules and PSP modules are added to the layers of the classification network to learn global features. The PSP module can aggregate contextual information from different regions, thereby improving the ability to obtain global information; four pyramid scale features are fused, the first row is the coarsest feature, and the following three rows are pooling features of different scales; in order to ensure the weight of the global feature, if the pyramid has N levels, use The convolution reduces the corresponding level channel to 1 / N of the original, and then obtains the size before pooling through bilinear interpolation, and finally connects them together.

[0147] In this implementation, the output of the classification network is divided into three directions; the first is to output the classification prediction probability, calculate the entropy value of the model results corresponding to different window widths and window positions and compare them, select the results below the threshold and use EMA to fuse the network parameters to form a new classification network parameter model, thereby outputting the classification results of fusion of multiple perspectives; the second is to multiply and superimpose the weights after global pooling with each feature layer to obtain the CAM under the corresponding window width and window position, and obtain the global CAM through symmetric KL divergence calculation; the third is to perform high-dimensional expansion of the data through feature data enhancement.

[0148] In this implementation, the classification results obtained from training under the five window width and window position perspectives are expressed with entropy values ​​to represent uncertainty, and compared with the preset parameter T (influenced by factors such as the number, quality, and size of training data). When it is less than the set threshold, the model and its parameters under this perspective are retained, and the EMA (Exponential Moving Average) is used to weight the retained network parameters to obtain a fused network, thereby obtaining the classification results under the fusion perspective. For the CAM graph, we obtain an I-dimensional vector P after global average pooling of the feature map of the last convolutional layer of the retained model, and then connect a classifier to select the category with the highest probability as the final output category. At this time, each unit in P has a weight ( ), After obtaining the probability of being classified, each weight represents the influence of each unit value in P on the final classification result; each unit corresponds to a channel of the feature map, so the weight reflects the influence of each channel in the feature map on the final classification. Therefore, a weighted sum is taken for each channel of the feature map, and then the obtained image is superimposed with the original image to complete the class activation mapping.

[0149] By calculating the symmetric KL divergence loss, the CAM representing the global is obtained, thereby assisting the classification network model in learning the global information. The global CAM map is restored to the original image size and then fused into the upsampling process of the segmentation network, and fused with the upsampled features. Training is continued on the basis of the previous parameter model to achieve the purpose of locating the area of ​​interest, and assist the segmentation network in re-learning; wherein the CAM map restores the image size to the same as the feature map in the upsampling stage according to the bounding box positioning information. The present invention realizes the auxiliary classification of segmentation features and segmentation results, the CAM map obtained by classification assists segmentation, and the collaborative learning of segmentation and auxiliary classification.

[0150] This implementation combines the idea of ​​genetic algorithm to generate extended data at the feature level through screening and copying. Specifically, the feature parameters after the model training cycle are retained in a feature warehouse, the change in entropy value of the same data in different cycles is calculated, and the features of the expression category are screened out. Then the mean of these features is used as the feature enhancement benchmark, and the remaining two data features of the class are randomly copied to fill the vacant feature positions. Finally, the feature-enhanced data is input into the classification network for fully connected learning. This method can expand the data of a few categories and prevent the results from overfitting or biased towards the majority class.

[0151] This implementation not only considers the feature information in three directions in three-dimensional space, but also realizes cross-scale feature learning by introducing the proposed MFF module, which improves the segmentation results while providing richer input information for classification; the geometric features of the target area are represented by the geometric center or central axis distance matrix, which increases the learnable information of the model and improves the classification accuracy of the model; a new hybrid loss function is proposed, which not only optimizes edge learning, but also limits the edge of the target area caused by holes, which can be widely used in the segmentation of different human targets and assist in diagnosis in various medical fields; by fusing multi-view network parameters and updating parameters based on the semi-supervised learning EMA mechanism, a more robust classification network model is obtained.

[0152] Embodiment 2:

[0153] This implementation method proposes a medical image lesion classification system, including:

[0154] The image segmentation unit is configured to: segment the three-dimensional CT image using a pre-trained segmentation network to obtain a segmented lesion target area, process the lesion target area using an edge bounding box to obtain a positioning image, calculate a distance matrix of the lesion target area, and in the segmentation network, calculate the symmetric KL divergence loss of the class activation map under different window widths and window positions to obtain a global class activation map and pass it to the segmentation network to provide positioning information;

[0155] The lesion classification unit is configured to: obtain the medical image lesion classification result according to the multi-scale features obtained by the segmentation network, the distance matrix, the positioning image and the pre-trained classification network; when training the classification network, save the feature parameters after global pooling of each cycle into the feature warehouse, calculate the change of entropy value of the same feature parameter in different cycles, screen out the feature parameters with strong expression categories, use the mean of the screened feature parameters as the basic feature parameters for expressing the global category, randomly copy two feature parameters of the same category according to the set ratio to fill the position of the feature parameters that have not been screened, obtain enhanced feature data, and input the enhanced feature data into the classification network for full connection learning.

[0156] The specific working process of each unit is shown in the steps of Example 1, which will not be repeated here.

[0157] It is understandable that the above-mentioned units can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple functionally smaller units to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the medical image lesion system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0158] According to another embodiment of the present application, the system described in the present implementation mode and the medical image lesion classification method of the embodiment of the present application can be constructed by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for classifying lesions in medical images, characterized in that: The process includes: The three-dimensional CT image is segmented by using a pre-trained segmentation network to obtain a segmented lesion target area, the lesion target area is processed by using an edge bounding box to obtain a positioning image, and a distance matrix of the lesion target area is calculated. In the segmentation network, the symmetric KL divergence loss of the class activation map under different window widths and window positions is calculated to obtain a global class activation map and pass it to the segmentation network to provide positioning information; The three-dimensional CT image is adjusted to a plurality of preset window widths and window positions, and different network models are used for segmentation and classification prediction respectively, and the classification prediction results under various window widths and window positions are obtained and the entropy uncertainty is calculated. When the number of network models whose entropy uncertainty is less than the set threshold is greater than 1, the parameters of the network models whose entropy uncertainty is greater than 1 are fused using the exponentially weighted moving average method. When the number of network models whose entropy uncertainty is less than the set threshold is equal to 1, the network model is retained as the final network model; wherein each of the network models includes a segmentation network and a classification network; A medical image lesion classification result is obtained based on the multi-scale features obtained by the segmentation network, the distance matrix, the positioning image and the pre-trained classification network.

2. The medical image lesion classification method according to claim 1, characterized in that: When training the classification network, the feature parameters after global pooling of each cycle are saved in the feature warehouse, the change of entropy value of the same feature parameter in different cycles is calculated, the feature parameters with strong expression categories are screened out, the mean of the screened feature parameters is used as the basic feature parameter for expressing the global category, two feature parameters of the same category are randomly copied according to the set ratio to fill the position of the feature parameters that have not been screened out, and the enhanced feature data is obtained, and the enhanced feature data is input into the classification network for full connection learning.

3. The medical image lesion classification method according to claim 1 or 2, characterized in that: By calculating the symmetric KL divergence loss of the class activation map, a global class activation map is obtained, including: When the classification model identifies category k, the class activation map of input image i is: in, is the weight of the feature map in channel c corresponding to category k, F i represents the features of input image i before the global average pooling layer, C is the total number of channels, h is the height of input image i, and w is the height of input image i; Construct a symmetric Kulbak-Leibler loss function based on class activation maps, including: is the class activation map of the jth category of input image i, K is the total number of categories, is the class activation map of the kth category of input image i, m is the total number of input images, during the training process, L CSKL right The gradient of is: when When , the optimization function obtains the optimal solution, and the optimal solution is used as the global class activation map. The obtained global class activation map is passed to the upsampling process of the segmentation network to assist the segmentation task.

4. The medical image lesion classification method according to claim 2, characterized in that: Calculate the change in entropy value of the same feature parameter in different cycles and select feature parameters with strong expression categories, including: W t =GAP(W); Among them, W represents the network weight, W t represents the network weight after global average pooling, and retains the entropy change H(W) for the results of the same data in different training cycles. t ) is less than the set threshold τ, and the average value of the retained features is calculated as the basic feature expressing the global category.

5. The medical image lesion classification method according to claim 1 or 2, characterized in that: Before segmenting the 3D CT image using the pre-trained segmentation network, the pre-processing and data enhancement process is also included, including: The 3D CT image is resampled to the average voxel spacing of the corresponding data set to maintain the spatial semantics of the 3D CT image. Assuming that a d-layer w×h ​​3D CT image is input, w is the width and h is the height, then the total number of voxels N is N=w×h×d. The w×h×d voxels are divided into non-overlapping patches, where each patch size is p×m×m, and the number of patches is t=(w / p)×(h / m)×(d / m); The resampled 3D CT images were cropped to the corresponding percentile of the entire image value, and the Z-score was normalized using the mean and standard deviation of each modality image; Remove sections without any organs before or after the entire sequence; The three-dimensional CT images are enhanced using data augmentation methods such as random elastic deformation, random rotation, random scaling or random mirroring.

6. The medical image lesion classification method according to claim 5, characterized in that: The 3D CT images are segmented using a pre-trained segmentation network, including: The segmented patch is linearly embedded into a C-dimensional vector through a convolutional layer, which is recorded as a token; The tokens are normalized and normalized through layers, and then feature extraction is performed using regular windows and moving windows. The regular window extracts information from independent tokens, and the moving window uses a shifted window to bridge adjacent tokens. Merge the extracted features with adjacent tokens in a non-overlapping manner along the spatial axis to obtain complete spatial tokens; After every two feature merging, a skip connection is introduced, and the skip connection is connected to the encoder of the same order as the upsampling. The features of the skip connection are subjected to the feature fusion module, stripe and surface pooling to obtain the key K and value V after enhanced attention as a supplement to the upsampling process.

7. A medical image lesion classification system, characterized in that: include: The image segmentation unit is configured to: segment the three-dimensional CT image using a pre-trained segmentation network to obtain a segmented lesion target area, process the lesion target area using an edge bounding box to obtain a positioning image, calculate the distance matrix of the lesion target area, and in the segmentation network, obtain a global class activation map by calculating the symmetric KL divergence loss of the class activation map under different window widths and window positions, and pass it to the segmentation network to provide positioning information; adjust the three-dimensional CT image to a plurality of preset window widths and window positions, respectively, use different network models for segmentation and classification prediction, obtain classification prediction results under various window widths and window positions, and calculate entropy uncertainty. When the number of network models whose entropy uncertainty is less than a set threshold is greater than 1, use an exponentially weighted moving average method to fuse the parameters of the network model whose entropy uncertainty is greater than 1. When the number of network models whose entropy uncertainty is less than a set threshold is equal to 1, only retain the network model as the final network model; wherein each of the network models includes a segmentation network and a classification network; The lesion classification unit is configured to obtain a medical image lesion classification result based on the multi-scale features obtained by the segmentation network, the distance matrix, the positioning image and the pre-trained classification network.

8. The medical image lesion classification system according to claim 7, characterized in that: In the lesion classification unit, when the classification network is trained, the feature parameters after global pooling of each cycle are saved in the feature warehouse, the change in entropy value of the same feature parameter in different cycles is calculated, the feature parameters with strong expression categories are screened out, the mean of the screened feature parameters is used as the basic feature parameter for expressing the global category, two feature parameters of the same category are randomly copied according to the set ratio to fill the position of the feature parameters that have not been screened out, and the enhanced feature data is obtained, and the enhanced feature data is input into the classification network for full connection learning.

Citation Information

Patent Citations

  • Liver cancer focus data processing method and system and storage medium

    CN110599447A

  • New coronal pneumonia CT detection, identification and positioning system and computing equipment

    CN112233117A

  • Medical image multi-organ segmentation method and system

    CN113808146A

  • Target positioning method and system based on multi-scale attention constraint

    CN114202649A

  • Cerebrovascular image segmentation method based on multi-scale attention network

    CN114723698A