Method for improving explainability of medical image deep learning network based on partial guidance

Abstract

The application discloses a method for improving the explainability of a medical image deep learning network based on partial guidance. The method comprises the following steps: medical image data preprocessing, gray normalization and size normalization of an original medical image; deep classification network model structure building, first, a feature extraction module based on a convolution structure is built, then a trainable class activation map module based on an attention mechanism is built after the feature extraction module, and finally a classification layer is connected after an attention weighting fusion path of the trainable class activation map module; in the deep classification network model training process, pixel-level strong labeled data information is introduced, and a hybrid supervision method is used to train the classification network model; and a pre-training and step-by-step hierarchical training method is used to adjust model parameters, optimize model performance and save an optimal model. The deep classification network model obtained by the application can not only provide strong explanations in line with doctors' cognition, but also has better classification performance.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and deep learning, and in particular to a method for improving the interpretability of deep learning networks for medical images based on partial guidance. Background Technology

[0002] Currently, deep classification network models are widely used in the medical field, greatly reducing the workload of doctors and assisting them in quickly processing large amounts of medical image data, enabling patients to receive more timely and accurate diagnostic results. However, existing deep classification network models face several limitations when applied to medical image processing, such as the following two points.

[0003] First, deep classification network models still suffer from poor interpretability. Because of the opacity within the network, they are considered black-box models; they can provide predictions, but the prediction process is difficult to interpret. Doctors cannot understand or trust the model's predictions, hindering their application in subsequent diagnoses. Visual interpretability methods can intuitively show the areas the network model focuses on when making predictions, thus providing a basis for model decision-making and enhancing the interpretability of deep classification network models. However, when applied to medical images, this method is affected by issues such as the lack of foreground prominence and organ coexistence, resulting in a gap between the visual evidence supporting the predictions and the basis for doctors' decisions. Furthermore, visual interpretability methods mainly focus on how to provide better interpretations, without exploring how to use model interpretations to improve model performance, thereby aligning model interpretation with performance and human cognition. Therefore, it is necessary to research a method that can improve the interpretability of deep classification network models and utilize interpretation to enhance model performance.

[0004] Secondly, training deep classification network models places extremely high demands on the quality and quantity of medical image annotations. Medical image annotation requires experienced physicians and can be categorized into strongly annotated and weakly annotated data based on annotation intensity. Strongly annotated data requires physicians to annotate each pixel layer by layer; while containing very rich information, this type of annotation requires a significant investment of time and effort, making it costly to obtain. Weakly annotated data only requires annotation of the approximate location or category information of the target, making it relatively easy to annotate and obtain; however, physicians still face a considerable burden in interpreting the images during the annotation process. Currently, semi-supervised learning and weakly supervised learning, aimed at reducing annotation costs, have been extensively studied, but neither method simultaneously utilizes the characteristics of strongly and weakly annotated data. In practical applications, while strongly annotated data is available for some data sets, models trained on large amounts of weakly annotated data suffer from information loss, making it difficult to improve model performance. Therefore, combining the use of two different types of annotated data to improve model performance is worthy of further research. Summary of the Invention

[0005] The purpose of this invention is to overcome the aforementioned problems and provide a method for improving the interpretability of deep learning networks for medical images based on partial guidance. This method outputs the region of interest within the deep classification network model to enhance its interpretability, while simultaneously utilizing a hybrid supervision method that combines strong and weak annotations to train the model and improve its classification performance.

[0006] To achieve the above objectives, the present invention proposes the following technical solution:

[0007] A method for improving the interpretability of deep learning networks for medical images based on partial guidance includes the following steps:

[0008] (1) Medical image data preprocessing: grayscale normalization and size normalization of the original medical images;

[0009] (2) Building the deep classification network model structure: First, a feature extraction module based on convolutional structure is built. Then, a trainable class active map (TCAM) module based on attention mechanism is built after the feature extraction module. Finally, a classification layer is connected after the attention weighted fusion path of TCAM.

[0010] (3) During the training of the deep classification network model, pixel-level strong annotation information of some data is introduced, and the deep classification network model is trained using a hybrid supervision method.

[0011] (4) Use pre-training and step-by-step hierarchical training to adjust model parameters, optimize model performance and save the optimal model.

[0012] Furthermore:

[0013] The original medical images used in step (1) are tomographic images, such as breast DCE-MR images and coronary CTA images, in which the patient's private information has been removed. The grayscale values ​​of the breast DCE-MR images used can be normalized to their maximum and minimum values, and the grayscale values ​​of the coronary CTA images used can be normalized within a specific CT value window. Specifically, the CT value window is selected from -100 to 500 HU, and the grayscale values ​​of pixels within this CT value range are normalized to between 0 and 1. Pixels with a grayscale value greater than 500 HU are set to 1, and pixels with a grayscale value less than -100 HU are set to 0. This invention performs binarization processing on strong annotations, setting the pixel values ​​of the annotated foreground targets to 1 and the pixel values ​​of the background areas to 0.

[0014] The different medical images used in step (1) have different resolutions. In this invention, bilinear interpolation is used to unify the images input to the network to a size of 512*512, and the corresponding strong annotations are unified to a size of 16*16.

[0015] The weak annotations used in this invention record image category information using 0 or 1. Specifically, images containing lesions (such as breast cancer lesions) are weakly labeled as 1, while images without lesions are weakly labeled as 0; images containing target organs (such as the aorta) are weakly labeled as 1, while images without target organs are weakly labeled as 0.

[0016] This invention saves scaled and grayscale normalized medical image data as a more easily processed PNG image format for training deep classification network models.

[0017] The deep classification network model built in step (2) mainly consists of three parts: first, a feature extraction module; then, a trainable class activation map module connected after the feature extraction module; and finally, two different classification layers connected after the trainable class activation map module according to different purposes. Both classification layers include a global mean pooling layer and a fully connected layer. The fully connected layer of the first classification layer is implemented by a linear layer with 512 input channels and 2 output channels, while the fully connected layer of the second classification layer is implemented by a linear layer with 1024 input channels and 2 output channels. The first classification layer is used during pre-training, and the second classification layer is used during step-by-step layered training. Specifically:

[0018] First, there is a feature extraction module, which includes a convolutional layer with a 7x7 kernel and a stride of 2, a max pooling layer with a 3x3 kernel and a stride of 2, and four convolutional blocks. Each convolutional block contains four convolutional layers with 3x3 kernels, a ReLU activation layer, and a batch normalization layer. The second and third convolutional layers are connected using residual connections. Second, there is a classification layer, including a global mean pooling layer and a fully connected layer. This invention designs two types of classification layers based on training characteristics: the first type of classification layer's fully connected layer is implemented by a linear layer with 512 input channels and 2 output channels, while the second type of classification layer's fully connected layer is implemented by a linear layer with 1024 input channels and 2 output channels. Finally, there is a trainable class activation map module based on an attention mechanism.

[0019] More specifically, the attention-based trainable class activation graph module includes a high-level semantic feature input path, a category information input path, a trainable weight layer, a trainable bias layer, a class activation graph output path, and an attention-weighted fusion path.

[0020] This invention places the class activation map module after the feature extraction module. After the input image passes through the feature extraction module, high-level semantic features with reduced resolution and increased feature channels are obtained. These high-level semantic features enter the TCAM module via the high-level semantic feature input path, while the category information provided by weak image annotations enters the TCAM module via the category information input path. The trainable weight layer is implemented using a 512*2 element-trainable matrix, with matrix elements initialized by the parameters of the fully connected layer in the first classification layer. The trainable bias layer is implemented using a 512*16*16 tensor, with parameters initialized to 0. Category information is selected from the weights belonging to the corresponding category in the trainable weight layer. Then, the high-level semantic features are weighted by the corresponding category weights and the trainable bias layer, implementing an attention mechanism that applies different weights to different features within a specific category. Finally, the weighted feature channels are summed channel by channel to obtain the class activation map.

[0021] The specific weighting formula is as follows:

[0022]

[0023] Among them, the trainable weight layer is ω. c ∈R k*c The trainable bias layer is b∈R k*m*n The high-level semantic features are A∈R k*m*n R is the real number field. L attention The TCAM module generates a category activation map, where y represents the image's category label. Here, k is the number of feature channels (512 in this invention), m*n is the size of the feature map (16*16 in this invention), and c is the number of categories (2 in this invention).

[0024] The class activation map output path outputs the generated class activation map.

[0025] The role of the attention-weighted fusion pathway is to fuse the class activation map generated by the attention mechanism and the high-level semantic features extracted by the feature extraction module. Specifically, the obtained class activation map is multiplied by the high-level semantic features to obtain the high-level semantic features fused by the attention mechanism. The weighted fusion formula is as follows:

[0026] A' = L attention ×A

[0027] Here, A' represents the fused high-level semantic features. In this invention, the classification layer is placed after the TCAM module, and the fused high-level semantic features are entered into the classification layer for classification.

[0028] Step (3) introduces pixel-level strong annotations into the training process of the deep classification network model. The attention mechanism forces the network to focus on regions consistent with those humans focus on when making decisions, thus enhancing the interpretability of the deep network. Deep classification networks require a large amount of image-level annotation during training, but image-level annotations cannot provide the location information of the target indicated by the image labels. This invention introduces pixel-level strong annotation information from some data into the training process of the deep classification network in a supervised manner, guiding the network to focus on the target region, thereby improving the interpretability of the network.

[0029] Specifically, in step (2), the TCAM module generates and outputs a class activation map. In step (3), the generated class activation map is constrained to be consistent with the strong annotation by the loss function, thus introducing strong annotation information. This invention uses mean squared error as the loss function, and the formula is as follows:

[0030]

[0031] Among them, Loss MSE Let the mean squared error loss function be . For the pixel in the i-th row and j-th column of the category activation map, The pixel in the i-th row and j-th column is the pixel with strong pixel-level annotation. In this invention, the size of the category activation map and the strong pixel-level annotation is 16*16, and i,j∈[0,15].

[0032] Step (3) using a hybrid supervision method specifically refers to: using a large amount of image-level supervision information for weak supervision learning, while using pixel-level supervision information of some data for strong supervision learning.

[0033] Weakly supervised learning process: For an image X input to the network, its corresponding weak label is y. c Let c represent the image category. X's features are extracted by the feature extraction module, then processed by the TCAM module to obtain the fused high-level semantic features A'. A' enters the classification layer for classification, and the classification layer outputs the classification result y. The classification result y and the weakly labeled y' are then compared. c Calculate the classification cross-entropy loss. CE This part pertains to weakly supervised learning, and the loss function formula is as follows:

[0034] Loss CE =-[y c logy+(1-y c log(1-y)]

[0035] Strongly supervised learning process: Because obtaining pixel-level strong annotations is costly, this invention only uses strong-annotated data from a subset of the datasets for training; that is, not all images used for training are provided with corresponding strong-annotated data. If a pixel-level strong annotation exists for X, it is denoted as y. pixelAfter step (2), X outputs the category activation map corresponding to the image category. This invention only guides the output category activation map for images containing organs and lesions. If the image input to the network contains a target, and the image also has pixel-level strong annotations, then the mean squared error loss is constrained on the output category activation map. This part is strong supervised learning. At this time, the total loss of the network is:

[0036] Loss = λ1Loss CE +λ2Loss MSE

[0037] Where λ1 is the cross-entropy loss coefficient and λ2 is the mean squared error loss coefficient, this invention can set λ1 to 1 and λ2 to 2. If the image input to the network contains a target, but the pixel-level strong annotation of this image does not exist, then only the classification loss is constrained, and the output class activation map is not constrained. If the image input to the network does not contain a target, then only the classification loss is constrained, and the output class activation map is not constrained. In both of the above cases, the total loss is:

[0038] Loss = λ1Loss CE

[0039] Step (4) involves training the model in two stages: pre-training and step-by-step hierarchical training.

[0040] In the pre-training phase, the feature extraction module and classification layer of the deep classification network model are trained first. Specifically, the feature extraction module and the first classification layer can form a general classification network. Training uses medical image data and corresponding weak annotations, and the loss function is the cross-entropy loss function Loss from step (3). CE The trained classification network parameters exhibit good feature extraction capabilities and classification performance. The parameters of the classification layer are used as the initial parameters for the trainable weight layers of the TCAM module. Specifically, the parameter matrix of the fully connected layer in the classification layer is imported into the parameter matrix of the TCAM module. Training is performed for 50 epochs with a learning rate of 0.0001, which decays to one-tenth of its original value every 20 epochs.

[0041] The step-by-step, layered training phase fine-tunes the parameters of the deep classification network model, including the following two steps: First, fine-tuning is performed on the TCAM module and the last convolutional layer of the fourth convolutional block. Specifically, the feature extraction module and classification layer in the network model inherit the parameters from the pre-training phase. The trainable weight layers of the TCAM module are initialized with the parameters of the fully connected layers of the classification layer, and the trainable bias layers are initialized with 0. Then, the model parameters, except for the TCAM module and the last convolutional layer of the fourth convolutional block, are set to be untrainable. Training uses medical image data and corresponding weak and partial strong annotations, and the loss function is the total loss function Loss from step (3). Training is performed for 50 rounds, with the learning rate set to 0.0001, decreasing to one-tenth of its original value every 20 rounds. Second, fine-tuning is performed on the TCAM module and the second classification layer. Specifically, except for the classification layer, the network model inherits the parameters obtained in the first step of training, and then the model parameters, except for the TCAM module and the classification layer, are set to be untrainable. This training step uses the second classification layer designed in step (2). The TCAM module does not select a specific weight layer based on category information. The trainable weight layers of both categories are weighted and summed with high-level semantic features to generate category activation maps for both categories. Then, the category activation maps of both categories are fused with the high-level semantic features. The fusion result is concatenated into 1024 channels and entered into the classification layer for classification. The training uses medical image data and corresponding weak annotations, and the loss function is the cross-entropy loss function Loss from step (3). CE The training run consists of 50 rounds, with a learning rate of 0.0001. The learning rate decays to one-tenth of its original value every 20 rounds.

[0042] The main beneficial effects of this invention are as follows: This invention provides a method for improving the interpretability of deep learning networks for medical images based on partial guidance. It proposes using an attention mechanism and a hybrid supervision method to enhance the interpretability of the deep network reasoning process, thereby improving the performance of the network model by guiding network interpretation. This invention adds a trainable class activation map module based on an attention mechanism after the convolution-based feature extraction module. Through a hybrid supervision method, it introduces pixel-level strong annotations and sufficient weakly labeled data to train the network model, forcing the network model's region of interest to align with human regions of interest. This influence is further propagated to subsequent layers through feature weighting, affecting the network's final decision. This invention uses an attention mechanism to assign higher weights to important information and lower weights to unimportant information, continuously adjusting the weights during training to suit different situations. The attention mechanism improves the model's ability to select useful features, thereby enhancing the model's performance, generalization ability, and robustness.

[0043] Compared with the prior art, the advantages of the present invention are:

[0044] 1) Introducing pixel-level annotation information for some data aligns the network's focus with human decision-making, improving the interpretability of the convolutional neural network process.

[0045] 2) Different layers of the classification network are trained sequentially with data of different labeling levels. The different features extracted and utilized by the different layers of the network are used to improve the classification performance of the network. Attached Figure Description

[0046] Figure 1 This is a flowchart of the method of the present invention;

[0047] Figure 2 This is a structural diagram of the feature extraction module;

[0048] Figure 3 is a structure diagram of a trainable class activation graph module. Figure 3(a) is a structure diagram of a trainable class activation graph module used in the first step of step-by-step hierarchical training, and Figure 3(b) is a structure diagram of a trainable class activation graph module used in the second step of step-by-step hierarchical training.

[0049] Figures 4(a) to 4(c) Figure 4(a) shows the network structure and training flowchart of the pre-trained deep classification network model, Figure 4(b) shows the network structure and training flowchart of the first step of the step-by-step hierarchical training, and Figure 4(c) shows the network structure and training flowchart of the second step of the step-by-step hierarchical training. Detailed Implementation

[0050] The following is a comparison chart. Figure 1 The following section provides a detailed description of the embodiments of the present invention. The examples described below are merely illustrative and are not intended to limit the application scenarios of the present invention.

[0051] The process of the method of the present invention is as follows: Figure 1 As shown. The following describes a specific implementation of the present invention using a deep learning network model for coronary CTA image classification as an example, including the following steps:

[0052] (1) Data preprocessing

[0053] Coronary CTA images have a wide grayscale range, and the significant differences in grayscale range between different images lead to inconsistencies in image brightness, necessitating grayscale normalization. Since CT values ​​have corresponding physical meanings, retaining the CT value range useful for research is reasonable. The goal of coronary CTA image classification is to determine the presence of the aorta in the image. The CT value range of the aorta is generally between -100 and 500 HU. Therefore, this invention extracts pixels within this grayscale range for normalization. Pixels with CT values ​​greater than 500 HU have a grayscale value of 1, and pixels with CT values ​​less than -100 HU have a grayscale value of 0.

[0054] The network designed in this invention accepts input images of size [B, C, H, W]. Here, B represents the number of images entering the network at one time; this invention uses 16. C represents the number of channels in the image entering the network; CT images are grayscale images with only one channel, so C is 1. H and W represent the height and width of the input image, respectively; this invention uses images with both height and width of 512 pixels. The original size of coronary CTA images is 512*512, and no image size adjustment is needed. For medical images of other sizes, this invention uses bilinear interpolation to scale the image to the specified size. This invention requires the input image to be a 2D image; for medical tomographic images that have been normalized in size and grayscale, the image must be saved as a PNG file.

[0055] The weak annotations corresponding to the coronary CTA image indicate whether the aorta is present; a value of 1 indicates the presence of the aorta, and a value of 0 indicates otherwise. The strong annotations corresponding to the coronary CTA image delineate the region containing the aorta in the image. Pixels within the aorta region have a grayscale value of 1, while pixels outside the aorta region have a grayscale value of 0. The strong annotations are the same size as the original image and are saved as PNG format. Before use, the 512*512 strong annotations need to be downsampled to 16*16 using bilinear interpolation.

[0056] (2) Pre-training

[0057] In the pre-training phase, the feature extraction module and the first classification layer of the deep classification network model are trained first. The feature extraction module is as follows: Figure 2 As shown, it mainly consists of a convolutional layer with a 7x7 kernel and a stride of 2, a max pooling layer with a 3x3 kernel and a stride of 2, and four convolutional blocks. Each convolutional block contains four convolutional layers with a 3x3 kernel, a ReLU activation layer, and a max pooling layer with a 3x3 kernel and a stride of 2. The second and third convolutional layers use residual connections. The fourth convolutional block is followed by the first classification layer to form a complete classification model. The network structure and training process of the pre-trained classification model are shown in Figure 4(a).

[0058] During pre-training, the resolution of the input image changes from 512*512 to 256*256 after passing through the first convolutional layer, and the number of channels increases from 1 to 64.

[0059] After passing through the pooling layer, the number of channels remains the same, but the resolution decreases again, from 256*256 to 128*128.

[0060] Next, the features are fed into the first convolutional block. The first convolutional block performs four convolution operations without changing the resolution and number of feature channels. Therefore, the resolution of the feature channels output by the first convolutional block is still 128*128, and the number of channels is 64.

[0061] The features then enter the second convolutional block. The first convolutional layer of the second convolutional block doubles the number of feature channels and halves the resolution, resulting in a feature channel resolution of 64*64 and 128 channels. The last three convolutional layers of the second convolutional block do not change the feature channel resolution or number of channels. Therefore, the output feature channel resolution of the second convolutional block is 64*64, and the number of channels is 128.

[0062] The features then enter the third convolutional block. The first convolutional layer of the third convolutional block also doubles the number of feature channels and halves the resolution, making the feature channel resolution 32*32 and the number of channels 256. The last three convolutional layers of the third convolutional block do not change the resolution and number of feature channels. Therefore, the feature channel resolution output by the third convolutional block is 32*32, and the number of channels is 256.

[0063] The features are then fed into the fourth convolutional block. The first convolutional layer of this fourth block doubles the number of feature channels and halves the resolution, resulting in a feature channel resolution of 16x16 and 512 channels. The last three convolutional layers of the fourth block do not change the feature channel resolution or number of channels. Therefore, the output feature channel resolution of the fourth convolutional block is 16x16, and the number of channels is 512. At this point, the feature extraction module's functionality is complete.

[0064] The classification layer uses the features extracted by the feature extraction module for classification. First, the 512-channel, 16*16 feature is transformed into a 512-channel, 1*1 feature by a global mean pooling layer. Then, it enters a fully connected layer to become a 2-channel, 1*1 prediction result. The two channels represent the prediction result of whether the input image belongs to class 0 or class 1, respectively. The larger the value, the greater the probability that the input image belongs to that class.

[0065] After the classification model outputs its prediction of the input image, it then performs softmax normalization on the prediction. This operation sets the sum of the two class predictions to 1, thus obtaining the probability that the input image is predicted to belong to each class by the classification model.

[0066] This invention calculates the cross-entropy loss function using the probabilities predicted by the network and the actual weak labels, and updates the network model parameters through the backpropagation algorithm. During training, the Adam optimization algorithm is used with a learning rate set to 0.001. A learning rate decay strategy is also employed during training, reducing the learning rate to one-tenth of its original value every 20 training epochs. The training objective of this invention is to bring the cross-entropy loss function close to 0. Based on the observed convergence of the loss function, 50 training epochs are selected, and the training parameters with the minimum loss function are saved as the model's pre-training parameters.

[0067] (3) Train the TCAM module and the last convolutional layer of the fourth convolutional block.

[0068] When training the TCAM module and the last convolutional layer of the fourth convolutional block, the network model consists of a feature extraction module, a TCAM module, and a first classification layer. After obtaining the pre-trained parameters of the classification network in step (2), the parameters of the above modules can be initialized. Specifically, the network structure and training process are shown in Figure 4(b). In this invention, the feature extraction module part and the classification layer part of the pre-trained parameters are imported into the corresponding modules respectively, and the parameters of the trainable weight layer in the TCAM module are also initialized by the classification layer parameters of the pre-trained parameters.

[0069] During training, the input image first enters the feature extraction module for feature extraction. As described in step (2), the feature extraction module outputs high-level semantic features with a resolution of 16*16 and 512 channels.

[0070] Next, as shown in Figure 3(a), the high-level semantic features and the weak annotations corresponding to the input image enter the TCAM module through the high-level semantic feature input path and the category information input path, respectively.

[0071] This invention selects corresponding trainable weight layer parameters based on the category of the weakly labeled image. Taking an input image containing the aorta as an example, its corresponding weak label is 1, and the parameters in the first column of the parameter matrix of the trainable weight layer are selected. The selected parameters contain 512 trainable weight values, which are multiplied by the 512-channel high-level semantic features according to the number of channels. That is, each channel of the high-level semantic features is multiplied by a weight, resulting in a feature that is still 16*16 in size and has 512 channels.

[0072] Next, the obtained features are added element-wise to the trainable bias layer, which is a 512*16*16 tensor. After addition, the features still have 512 channels and a size of 16*16. To obtain the class activation map corresponding to the input image category, the obtained features need to be added according to the number of channels. This invention adds elements at the same position on all 512 channels to obtain a 16*16 2D class activation map, which is then output through the class activation map output path.

[0073] This invention aims to learn location-related features using category activation maps. Specifically, it fuses the obtained category activation map with high-level semantic features, applying different weights to each position of the high-level semantic features. First, the invention expands the 16*16 category activation map to a 512-channel 16*16 size using a copying method. Then, it directly multiplies the expanded category activation map with the corresponding elements of the high-level semantic features. After feature fusion, the TCAM module outputs a 512*16*16 fused high-level semantic feature.

[0074] The fused high-level semantic features are then classified in the classification layer. First, the 512-channel, 16*16-size features are transformed into 512-channel, 1*1-size features through a global mean pooling layer, and then enter a fully connected layer to become 2-channel, 1*1-size prediction results.

[0075] During training, the network model has two outputs. The first output is a 16*16 class activation map output through the TCAM module class activation map output path, and the second output is the two-channel classification prediction result output through the classification layer.

[0076] The first output represents the region that the network model focuses on when making decisions. The output is a 16*16 resolution image, and the grayscale value of the image represents the degree of attention the network pays to that region of the input image; the stronger the attention, the larger the grayscale value. This invention aims to ensure that the network's region of interest is precisely the region where the target is located. Therefore, this invention uses pixel-level strong annotations to guide the output class activation map.

[0077] The pixel-level strong annotations were originally binary images of size 512*512. In order to calculate the loss with the class activation map, this invention first downsamples the resolution of the strong annotations to 16*16. The downsampling operation causes the grayscale range of the strong annotations to be between 0 and 1. In order to retain more information, this invention does not perform binarization processing.

[0078] This invention calculates the mean squared error loss using the class activation map output by the network and the downsampled strong annotations. Because this invention only uses pixel-level strong annotations for a portion of the data, this step of loss calculation is unnecessary for input images without strong annotations.

[0079] The second output represents the network's prediction of the category of the input image. Similar to step (2), the prediction result is first normalized by softmax to obtain the prediction probabilities of the two categories. Then, the cross-entropy loss is calculated using the network's predicted probabilities and the actual weak labels.

[0080] This invention aims to train the TCAM module to provide interpretations consistent with human cognition; therefore, the TCAM parameters are set to be trainable during training. Furthermore, because the features extracted by the later convolutional blocks of the feature extraction module are more sophisticated, and the output of the feature extraction module is the input of the TCAM module, this invention also sets the parameters of the last convolutional layer of the fourth convolutional block of the feature extraction module to be trainable. All other parameters of the network model, except those mentioned above, are set to be non-trainable, maintaining the pre-trained parameters unchanged.

[0081] The total loss during training is calculated in three ways: First, if the input image is of class 1 and its corresponding strong label exists, the mean squared error loss is calculated for the first output and the cross-entropy loss is calculated for the second output. The total loss is the cross-entropy loss plus twice the mean squared error loss. Second, if the input image is of class 1 but its corresponding strong label does not exist, the output result of the first path is not constrained, and the total loss is equal to the cross-entropy loss calculated by the second path. Third, if the input image is of class 0, the output result of the first path is not constrained regardless of whether its corresponding strong label exists, and the total loss is also equal to the cross-entropy loss calculated by the second path.

[0082] This invention calculates the total loss and then updates the trainable parameters of the network model using the backpropagation algorithm. During training, the RMSprop optimization algorithm is used, with a learning rate set to 0.0001. A learning rate decay strategy is also employed, reducing the learning rate to one-tenth of its original value every 20 training epochs. The goal of this invention is to train the network until the total loss approaches zero. Based on observations of the loss function's convergence, 50 training epochs are selected, and the training parameters with the minimum loss function are saved.

[0083] (4) Train the TCAM module and the second classification layer

[0084] The purpose of step (3) above is to train the interpretability of the class activation map output by the TCAM module. It is also necessary to retrain a classification layer that matches the interpretability of the network.

[0085] When training the TCAM module and the second classification layer, the network model consists of a feature extraction module, a TCAM module, and a second classification layer. After obtaining the training results in step (3), the parameters of the above modules are re-initialized. Specifically, the network structure and training flowchart are shown in Figure 4(c). Because step (3) adjusts the parameters of the last convolutional layer of the fourth convolutional block of the feature extraction module, when inheriting parameters, the other layers of the feature extraction module, except for the last convolutional layer of the fourth convolutional block, inherit the parameters of the corresponding layers obtained from the pre-training in step (2), and the last convolutional layer of the fourth convolutional block inherits the parameters obtained from the training in step (3). The TCAM module inherits the parameters of the corresponding module trained in step (3). The second classification layer is randomly initialized because it has not been trained in the previous training.

[0086] During training, the input image first enters the feature extraction module for feature extraction. As described in step (2), the feature extraction module outputs high-level semantic features with a resolution of 16*16 and 512 channels.

[0087] Next, as shown in Figure 3(b), the high-level semantic features enter the TCAM module through the high-level semantic feature input path.

[0088] In this training process, this invention uses two types of weights in a trainable weight layer to weight the high-level semantic features. Specifically, taking an input image containing the aorta as an example, its corresponding weak label is 1. The high-level semantic features are weighted not only using the trainable weights corresponding to class 1, but also using the trainable weights corresponding to class 0. Each type of weight contains 512 trainable weight values, which are multiplied by the 512 channels of the high-level semantic features according to the number of channels, resulting in two sets of features with a resolution of 16*16 and 512 channels.

[0089] The two sets of features are added element-wise to the trainable bias layer to obtain two sets of features of the same size. To obtain class activation maps for the two categories, the two sets of features are further summed channel-wise. The two 16*16 2D class activation maps obtained by the above operations correspond to the network model's attention to the two categories of the input image, and are output through the class activation map output path.

[0090] Next, this invention fuses both category activation maps with high-level semantic features. Similar to the process in (3), the category activation maps are first expanded to 512 channels, and then the expanded category activation maps are directly multiplied with the elements corresponding to the high-level semantic features to obtain two sets of fused features. The two sets of features are stacked according to channels to obtain a fused high-level semantic feature of size 1024*16*16.

[0091] The fused high-level semantic features are then classified in the second classification layer. First, the 1024-channel, 16*16-size features are transformed into 1024-channel, 1*1-size features through a global mean pooling layer, and then enter a fully connected layer to become 2-channel, 1*1-size prediction results.

[0092] During training, the network model also has two outputs. The first output is a 2*16*16 class activation map output through the TCAM module class activation map output path, and the second output is the two-channel classification prediction result output through the classification layer.

[0093] This training aims to train a classification layer that matches the TCAM module. During training, the parameters of the TCAM module and the second classification layer are set to be trainable, while other parameters of the network model are set to be non-trainable. No constraints are imposed on the first output during training. Similar to step (3), for the second output, this invention first performs softmax normalization on the prediction results to obtain the prediction probabilities of the two categories, and then calculates the cross-entropy loss using the network's predicted probabilities and the actual weak labels.

[0094] After calculating the loss, the trainable parameters of the network model are updated using the backpropagation algorithm. During training, the RMSprop optimization algorithm is used, with a learning rate set to 0.0001. A learning rate decay strategy is employed, reducing the learning rate to one-tenth of its original value every 20 training epochs. The goal of this training is to bring the loss close to zero. Based on observations of the loss function's convergence, 50 training epochs are selected, and the training parameters with the minimum loss function are saved as the final training parameters.

[0095] The above description, in conjunction with specific embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions, and those skilled in the art will be able to design other alternative embodiments without departing from the scope of the claims. Any reference numerals placed between parentheses in the claims should not be interpreted as limiting the scope of the claims.

Claims

1. A method for improving the interpretability of deep learning networks for medical images based on partial guidance, characterized in that, Includes the following steps: (1) Medical image data preprocessing: grayscale normalization and size normalization of the original medical images; (2) Building the deep classification network model structure: First, a feature extraction module based on convolutional structure is built. Then, a trainable class activation map module (TCAM) based on attention mechanism is built after the feature extraction module. Finally, a classification layer is connected after the attention weighted fusion path of TCAM. (3) During the training of the deep classification network model, pixel-level strong annotation information of some data is introduced, and the deep classification network model is trained using a hybrid supervision method; (4) Use pre-training and step-by-step hierarchical training to adjust model parameters, optimize model performance, and save the optimal model; The trainable class activation graph module consists of a high-level semantic feature input path, a category information input path, a trainable weight layer, a trainable bias layer, a class activation graph output path, and an attention-weighted fusion path. The trainable weight layer is implemented using a 512*2 element-trainable matrix, with matrix elements initialized by the parameters of the fully connected layer in the first classification layer. The trainable bias layer is implemented using a 512*16*16 tensor, with parameters initialized to 0. After the input image passes through the feature extraction module, high-level semantic features with reduced resolution and increased feature channels are obtained. These high-level semantic features then enter the TCAM module via the high-level semantic feature input path. Simultaneously, category information provided by weak image annotations enters the TCAM module via the category information input path. Category information selects weights belonging to the corresponding category from the trainable weight layer. The high-level semantic features are then weighted by the corresponding category weights and the trainable bias layer, implementing an attention mechanism that applies different weights to different features within a specific category. Next, the weighted feature channels are summed channel by channel to obtain a category activation map. The category activation map output path outputs the generated category activation map. Finally, the category activation map generated by the attention mechanism and the high-level semantic features extracted by the feature extraction module are fused through a weighted fusion path. Specifically, the obtained category activation map is copied to 512 channels and then multiplied by the high-level semantic features channel by channel to obtain the attention mechanism-weighted fused high-level semantic features. In step (3), the weakly supervised learning process is as follows: for an image X input to the network, its corresponding weak label is y. c c represents the image category; X extracts features through the feature extraction module, then passes through the TCAM module to obtain the fused high-level semantic features A'. A' enters the classification layer for classification, and the classification layer outputs the classification result y; the classification result y and the weak label y c Calculate the classification cross-entropy loss. CE Strongly supervised learning process: Training is performed using strongly labeled data. If a pixel-level strong label exists for X, it is recorded as y. pixel After passing through the feature extraction module and the TCAM module, X outputs the class activation map corresponding to the image category. Only for images containing organs and lesions, the output class activation map is subject to mean squared error loss. MSE The calculation.

2. The method for improving the interpretability of deep learning networks for medical images based on partial guidance according to claim 1, characterized in that, The deep classification network model built in step (2) consists of three parts. First, there is a feature extraction module. Then, a trainable class activation map module is connected after the feature extraction module. Finally, two different classification layers are connected after the trainable class activation map module according to different purposes. Both classification layers include a global mean pooling layer and a fully connected layer. The fully connected layer of the first classification layer is implemented by a linear layer with 512 input channels and 2 output channels. The fully connected layer of the second classification layer is implemented by a linear layer with 1024 input channels and 2 output channels. The first classification layer is used during pre-training, and the second classification layer is used during step-by-step layered training. The feature extraction module includes a convolutional layer with a 7*7 kernel and a stride of 2, a max pooling layer with a 3*3 kernel and a stride of 2, and four convolutional blocks. Each convolutional block contains four convolutional layers with a 3*3 kernel, a ReLU activation layer, and a batch normalization layer. The second and third convolutional layers are connected by residual connections.

3. The method for improving the interpretability of deep learning networks for medical images based on partial guidance according to claim 1, characterized in that, Different loss functions are used to constrain the input image based on whether it has corresponding pixel-level strong annotations. Specifically, if the input image contains a target and also has pixel-level strong annotations, then the mean squared error loss is constrained on the output class activation map. In this case, the total loss of the network is: Loss=λ1Loss CE +λ2Loss MSE Where λ1 is the coefficient of cross-entropy loss and λ2 is the coefficient of mean square error loss; If the image input to the network contains a target, but the pixel-level strong annotation of this image does not exist, then only the classification loss is constrained; if the image input to the network does not contain a target, then only the classification loss is constrained; in both cases, the total loss is: Loss=λ1Loss CE 。 4. The method for improving the interpretability of deep learning networks for medical images based on partial guidance according to claim 1, 2, or 3, characterized in that, In step (4), the pre-training stage first trains the feature extraction module and classification layer of the deep classification network model; specifically, the feature extraction module and the first classification layer constitute a classification network; the training uses medical image data and corresponding weak annotations, and the loss function is the cross-entropy loss function. CE The parameters of the classification layer are used as the initial parameters of the trainable weight layers of the TCAM module.

5. The method for improving the interpretability of deep learning networks for medical images based on partial guidance according to claim 1, 2, or 3, characterized in that, In step (4), the step-by-step layered training includes two steps: The first step is to fine-tune the TCAM module and the last convolutional layer of the fourth convolutional block. Specifically, the feature extraction module and classification layer in the network model inherit the parameters from the pre-training stage. The trainable weight layer of the TCAM module is initialized by the fully connected layer parameters of the classification layer, and the trainable bias layer is initialized with 0. Then, the model parameters other than the TCAM module and the last convolutional layer of the fourth convolutional block are set to be untrainable. The second step is to fine-tune the TCAM module and the second classification layer. Specifically, except for the classification layer, the network model inherits the parameters obtained from the first step of training. Then, the model parameters except for the TCAM module and the classification layer are set to be untrainable. This step uses the second classification layer designed in step (2). The TCAM module does not select a specific weight layer through category information. The trainable weight layers of the two categories are weighted and summed on the high-level semantic features to generate the category activation maps of the two categories. Then, the category activation maps of the two categories are fused with the high-level semantic features. The fusion result is spliced ​​into 1024 channels and entered into the classification layer for classification.

6. The method for improving the interpretability of deep learning networks for medical images based on partial guidance according to claim 1, characterized in that, In step (4), the step-by-step layered training includes two steps: The first step is to fine-tune the TCAM module and the last convolutional layer of the fourth convolutional block. Specifically, the feature extraction module and classification layer in the network model inherit the parameters from the pre-training stage. The trainable weight layer of the TCAM module is initialized by the fully connected layer parameters of the classification layer, and the trainable bias layer is initialized with 0. Then, the model parameters other than the TCAM module and the last convolutional layer of the fourth convolutional block are set to be untrainable. The second step is to fine-tune the TCAM module and the second classification layer. Specifically, except for the classification layer, the network model inherits the parameters obtained from the first step of training. Then, the model parameters except for the TCAM module and the classification layer are set to be untrainable. This step uses the second classification layer designed in step (2). The TCAM module does not select a specific weight layer through category information. The trainable weight layers of the two categories are weighted and summed on the high-level semantic features to generate the category activation maps of the two categories. Then, the category activation maps of the two categories are fused with the high-level semantic features. The fusion result is spliced ​​into 1024 channels and entered into the classification layer for classification.

7. The method for improving the interpretability of deep learning networks for medical images based on partial guidance according to claim 4, characterized in that, In step (4), the step-by-step layered training includes two steps: The first step is to fine-tune the TCAM module and the last convolutional layer of the fourth convolutional block. Specifically, the feature extraction module and classification layer in the network model inherit the parameters from the pre-training stage. The trainable weight layer of the TCAM module is initialized by the fully connected layer parameters of the classification layer, and the trainable bias layer is initialized with 0. Then, the model parameters other than the TCAM module and the last convolutional layer of the fourth convolutional block are set to be untrainable. The second step is to fine-tune the TCAM module and the second classification layer. Specifically, except for the classification layer, the network model inherits the parameters obtained from the first step of training. Then, the model parameters except for the TCAM module and the classification layer are set to be untrainable. This step uses the second classification layer designed in step (2). The TCAM module does not select a specific weight layer through category information. The trainable weight layers of the two categories are weighted and summed on the high-level semantic features to generate the category activation maps of the two categories. Then, the category activation maps of the two categories are fused with the high-level semantic features. The fusion result is spliced ​​into 1024 channels and entered into the classification layer for classification.

Images