A medical image classification model with an improved attention mechanism, a medical image classification method and related devices

Abstract

The application discloses a medical image classification model with an improved attention mechanism, a medical image classification method and related devices, and relates to the technical fields of artificial intelligence and medical image processing. An ASAE module is introduced into an EfficientNet network. The SE part of the ASAE module generates a channel weighting feature map by calculating the attention weights between different channels. The attention part calculates the attention weights of different local regions within a single channel based on a zero attention mechanism and generates a region enhancement feature map. The EfficientNet network outputs a medical image category based on the double-attention enhancement feature map composed of the two kinds of feature maps. Since the attention mechanism is improved, the importance of different channels is focused on, and the key information of local regions within a single channel is captured without significantly increasing the model parameters and inference time. The problem of insufficient precision of traditional models in medical image classification is effectively solved, the classification precision is improved, and the method is especially suitable for disease category judgment of dermoscope images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and medical image processing, in particular to a medical image classification model with improved attention mechanism, a medical image classification method and related devices. BACKGROUND

[0002] Skin cancer is the most common cancer and also one of the diseases with high mortality rate among many cancers. Early detection and treatment of skin lesions can greatly improve the cure rate of skin cancer. At present, dermoscopy is an effective means for skin disease diagnosis, but many skin diseases still need to rely on professional medical knowledge and clinical experience. Therefore, computer algorithm assisted diagnosis of skin diseases is very helpful to improve the accuracy of skin cancer diagnosis. Convolutional neural network performs excellently in image recognition field, and EfficientNet network is widely used in medical image classification research due to its high accuracy and low parameter amount characteristics on ImageNet dataset.

[0003] However, the current medical image classification method based on EfficientNet network architecture still has the problem of difficult balance between inference speed and classification accuracy. SUMMARY

[0004] The purpose of the present application is to provide a medical image classification model with improved attention mechanism, a medical image classification method and related devices to improve the classification accuracy of medical images (especially dermoscopy images) while avoiding significant increase in inference time and parameters.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides a medical image classification model with improved attention mechanism, which is an EfficientNet network introducing an ASAE module, wherein the ASAE module is used to process the feature map generated by the medical image and output a double attention enhanced feature map, and the EfficientNet network is used to output the medical image category based on the double attention enhanced feature map.

[0007] The ASAE module comprises an SE part and an attention part, wherein the double-attention enhanced feature map comprises a channel-weighted feature map and a region-enhanced feature map, the SE part is configured to calculate attention weights between different channels and generate the channel-weighted feature map based on the attention weights between different channels; and the attention part is configured to calculate attention weights of different local regions within a single channel based on a zero attention mechanism, and generate the region-enhanced feature map based on the attention weights of different local regions, wherein the zero attention mechanism refers to adding a column or a row of zero vectors in a key matrix and a value matrix, and taking the zero vectors as a benchmark for attention score calculation.

[0008] In a second aspect, the present application provides a medical image classification method, comprising:

[0009] obtaining a medical image to be classified;

[0010] inputting the medical image to be classified into a trained medical image classification model, and outputting a medical image category, wherein the medical image classification model adopts the image classification model with the improved attention mechanism according to the first aspect.

[0011] In a third aspect, the present application provides a medical image classification device, comprising:

[0012] an image acquisition module configured to obtain a medical image to be classified;

[0013] a classification module configured to input the medical image to be classified into a trained medical image classification model, and output a medical image category, wherein the medical image classification model adopts the image classification model with the improved attention mechanism according to the first aspect.

[0014] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the medical image classification method according to the second aspect.

[0015] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the medical image classification method according to the second aspect.

[0016] In a sixth aspect, the present application provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the medical image classification method according to the second aspect.

[0017] According to the embodiments provided in the present application, the following technical effects are achieved:

[0018] The application provides a medical image classification model with an improved attention mechanism, a medical image classification method and related devices, introduces an ASAE module in an EfficientNet network, the SE part of the ASAE module generates a channel weighted feature map by calculating the attention weights between different channels, the attention part calculates the attention weights of different local regions in a single channel based on a zero attention mechanism and generates a region enhanced feature map, and the EfficientNet network outputs a medical image category based on the double attention enhanced feature map composed of the two kinds of feature maps. The technical solution improves the attention mechanism, pays attention to the importance of different channels and captures the key information of local regions in a single channel without significantly increasing the model parameters and inference time, effectively solves the problem of insufficient precision of traditional models in medical image classification, improves the classification precision, and is especially suitable for disease category judgment of dermoscope images. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 FIG. 1 is a structural schematic diagram of a medical image classification model with an improved attention mechanism in Embodiment 1 of the present application;

[0021] Figure 2 FIG. 2 is a schematic diagram of the modified attention weight calculation method in Embodiment 1 of the present application;

[0022] Figure 3 FIG. 3 is a flow schematic diagram of a medical image classification method provided in Embodiment 2 of the present application;

[0023] Figure 4 FIG. 4 is a structural schematic diagram of a computer device provided in Embodiment 4 of the present application. DETAILED DESCRIPTION

[0024] It is found through research that existing medical image classification methods are mainly based on deep convolutional neural networks (CNN) and Transformer architecture. Among them, EfficientNet and other CNN models improve the calculation efficiency through depth separable convolution and SE (Squeeze-and-Excitation) module, but still have the following problems:

[0025] Limitations of channel attention mechanism: the traditional SE module only pays attention to the relationship between channels and ignores the semantic association of different regions inside a single channel, resulting in the loss of local key information.

[0026] Deviation problem of multi-head attention mechanism: when calculating the attention score, if the scores of all query vectors are generally low, the Softmax normalization may still give higher weights to unimportant information, leading to unreasonable attention allocation and affecting classification accuracy.

[0027] Trade-off between computational efficiency and accuracy: although the existing improvement methods (such as the CBAM module) improve the classification accuracy, they often introduce additional parameters, increasing the computational burden and being not conducive to deployment in resource-limited medical scenarios.

[0028] To this end, the embodiment provides a medical image classification model with an improved attention mechanism, a medical image classification method and related devices, which can improve classification accuracy, avoid attention deviation, and not significantly increase computational cost.

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0030] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0031] Embodiment 1

[0032] The embodiment provides a medical image classification model with an improved attention mechanism, which is an EfficientNet network introducing an ASAE module, wherein the ASAE module is used to process the feature map generated by the medical image and output a double attention enhanced feature map, and the EfficientNet network is used to output a medical image category based on the double attention enhanced feature map.

[0033] The ASAE module comprises an SE part and an attention part, wherein the double-attention enhanced feature map comprises a channel-weighted feature map and a region-enhanced feature map, the SE part is used for calculating attention weights between different channels and generating the channel-weighted feature map based on the attention weights between different channels; and the attention part is used for calculating attention weights of different local regions within a single channel based on a zero attention mechanism, and generating the region-enhanced feature map based on the attention weights of different local regions, wherein the zero attention mechanism refers to adding a column or a row of zero vectors in a key matrix and a value matrix, and taking the zero vectors as a benchmark for attention score calculation.

[0034] The improved medical image classification model will be specifically explained below. Figure 1 The improved medical image classification model will be specifically explained below.

[0035] The backbone architecture of the improved medical image classification model is based on EN (EfficientNet). EN is an image classification model, and its core lies in a novel model scaling method. By balancing the depth, width and resolution of the network, EN reduces the consumption of computing resources while improving model accuracy. Compared with traditional CNN architectures, EN can still achieve better performance with a significant reduction in parameter quantity and computational complexity.

[0036] In the convolution operation, EN adopts multiple efficient convolution methods. For example, it uses depthwise separable convolution (Depthwise Separable Convolution). This convolution method decomposes the standard convolution into two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs convolution operation on each input channel separately, greatly reducing the amount of calculation and the number of parameters, and significantly reducing the computational cost without sacrificing too much performance. In addition, EN also introduces a bottleneck structure (Bottleneck Structure) with an SE module. In the bottleneck structure, the input is first reduced in dimension by a 1x1 convolution layer to reduce the number of channels and thus reduce the computational cost; then depthwise separable convolution and SE operation are performed to extract features; finally, a 1x1 convolution layer is used to restore the channel number to the original size. This structure effectively reduces the amount of calculation and the number of parameters while maintaining good feature extraction capability. EN is composed of a structure of repeatedly stacked MBConv and forms eight different versions of b0-b7.

[0037] The SE module in the MBConv structure serves as a cross-channel attention mechanism, which to some extent enhances the network's attention and filtering ability to different channel information. However, this module has limitations. It only considers the relationship between channels, ignoring the semantic association within a single channel region. This leads to the model's inability to fully focus on the importance of different regions within each single channel, causing some key local information to be missed during processing.

[0038] In view of this, the SE module is improved in this embodiment, mainly reflected in replacing the squeeze and excitation (SE) structure with the attention squeeze and excitation (ASAE) structure. Through this replacement, the new ASAE structure not only can focus on the attention weight information of different channels, but also can more comprehensively capture the semantic information within a single channel, enhancing the model's perception ability of each region in the image, thereby providing more rich and accurate feature information for subsequent image analysis tasks.

[0039] The ASAE module mainly consists of two key parts, namely the attention part and the SE part. In this architecture, the SE part continues its function in the traditional structure, mainly responsible for processing the attention mechanism between channels. By evaluating and weighting the importance of different channels, the model can more effectively focus on the information of different channels, thereby enhancing the ability to grasp the overall features. The attention part focuses on processing the attention of different regions within a single channel. This innovative design enables the model to further focus on key regions within the channel, improving the perception and analysis ability of local details.

[0040] The workflow of the attention part is relatively fine and complex. First, the convolutional features will be segmented. The segmentation granularity is a hyperparameter, which has an important influence on the model performance: the more patches (local regions) divided, the more detailed local information the model can capture, and theoretically the better the classification accuracy. After segmentation, a linear mapping operation is used to convert the segmented patches into a series of patch vectors, which carry the feature information of the local regions of the image. To enable the model to better understand the position information of these patch vectors in the image, a position encoding is added. It is worth noting that the dimension of the position encoding is also a hyperparameter, and its reasonable setting is crucial for the model to accurately capture spatial information. Different dimension settings can have a significant impact on the model's performance, and the formula is as follows:

[0041] F=R C×H×W (1)

[0042]

[0043] V i = L(P i ), i = 1, 2, 3... (3)

[0044] u i = concat(v i , e i ), i = 1, 2, 3... (4)

[0045] where F represents the convolution feature, N is the number of patches of the same segmentation size, V i is the patch vector after linear convolution, u i is the patch vector added with the position encoding e i .

[0046] In the traditional multi-head attention mechanism calculation process, the query vector q and the key vector k are multiplied to calculate the score, and this score is essentially a relative score. For example, assuming there are three query vectors q1, q2, q3, when calculating the attention score of q1, the dot product of q1 and each key vector will be calculated. If none of the three query vectors has important semantic information in the current task scenario, the scores calculated will generally be low. However, after the softmax function is processed, these scores will be converted into attention weights in the form of probability distribution. Even if the three scores are low, the attention weight corresponding to the relatively highest score will still occupy a large proportion in the entire probability distribution, thereby obtaining a large attention allocation. This situation will lead to the occurrence of attention deviation, causing the model to be inaccurate when focusing on information, and thus negatively affecting the extraction of subsequent key information, interfering with the accurate understanding and analysis of important content by the model.

[0047] Traditionally, the improvement strategies for multi-head attention mechanism mostly focus on adjusting the attention calculation method, adding additional attention modules, etc. However, considering the problems existing in the traditional multi-head attention mechanism, this embodiment innovatively proposes a unique improvement method. In the key vector (k) and value vector (v) involved in the multi-head attention mechanism, a column (or a row) of zero vectors is artificially added. This idea of introducing a benchmark by adding zero vectors is relatively unique among many improvement methods. This operation is equivalent to introducing a benchmark reference in the multi-head attention mechanism. When performing attention calculation, the score can be adjusted based on this benchmark. When the problem of attention deviation caused by generally low scores is encountered again, the corresponding score can be made as low as possible by calculating the score with the zero vector. Due to the existence of "zero attention", the attention weight corresponding to these vectors with low scores due to unimportant information will become very small after the softmax function processing, and even tend to 0 in extreme cases. In this way, the model can more reasonably allocate attention, avoid excessive attention to unimportant information, and thus improve the accuracy of key information extraction. The calculation sequence is as shown in Figure 2 query vector, key vector, value vector and calculated score, respectively. After adding a row (column) of zero vectors, we have

[0048]

[0049] where K' and V' are the k matrix and v matrix after adding zero vectors, respectively, and 01×d n represents a zero vector of length n.

[0050]

[0051] Then apply the softmax function to the new attention matrix S' to get the new attention weight matrix A':

[0052] A' = softmax(S') (8)

[0053] For each row i in A', the elements are calculated as follows:

[0054]

[0055] As can be seen from equation (9), when the original S' i，j is generally low, due to the newly added 0 score item, the attention weight of the element with low score will be significantly reduced after softmax, and the situation of low weight still getting high weight score will not occur.

[0056] ​The embodiment improves the Transformer attention mechanism and designs the ASAE module, effectively improves the accuracy of medical image classification, and does not increase the model parameters, maintains the computing efficiency, applies the improved medical image classification model to the skin disease classification field, can further improve the skin disease classification accuracy, and is suitable for automatic diagnosis of dermoscope images and other medical images.

[0057] Embodiment 2

[0058] As shown in Figure 3 The embodiment provides a medical image classification method, which comprises the following steps:

[0059] S1: acquiring a medical image to be classified;

[0060] S2: inputting the medical image to be classified into a trained medical image classification model, and outputting a medical image category, wherein the medical image classification model adopts the image classification model with an improved attention mechanism in the embodiment 1.

[0061] More specifically, the medical image adopted by the embodiment is a dermoscope image, so as to realize the classification of skin diseases.

[0062] The training process of the medical image classification model specifically comprises the following steps:

[0063] Acquiring a medical image training data set, analyzing and preprocessing the skin disease data;

[0064] Inputting the medical image training data set into the medical image classification model, applying the transfer learning idea to transfer the EfficientNet series network weight parameters based on the ImageNet data set to the skin disease data set, and adjusting the model parameters through back propagation until the medical image classification model converges.

[0065] The improvement of the medical image classification model in the embodiment does not increase additional parameters, but the accuracy is improved by a considerable margin compared with the unimproved one (the existing neural network model is still inaccurate in judging skin diseases, while the new method of the embodiment has better effect and higher classification accuracy, but the inference event does not increase significantly; the attention mechanism in the transformer model is improved, which can better pay attention to the features of the medical image), effectively improves the medical image classification accuracy, and maintains the model efficiency, and has high clinical application value.

[0066] Embodiment 3

[0067] Based on the same inventive concept, the embodiment of the present application further provides a medical image classification device for implementing the medical image classification method of the above-mentioned embodiment 2. The implementation scheme for solving problems provided by the device is similar to the implementation scheme described in the above-mentioned method, so the specific limitation of the medical image classification device provided by the present embodiment can be referred to the limitation of the medical image classification method in the above, which will not be repeated here.

[0068] The medical image classification device provided by the present embodiment comprises:

[0069] An image acquisition module is configured to acquire a medical image to be classified.

[0070] A classification module is configured to input the medical image to be classified into a trained medical image classification model and output a medical image category, wherein the medical image classification model is the image classification model with the improved attention mechanism according to the first aspect.

[0071] Embodiment 4

[0072] The present embodiment provides a computer device, which can be a server or a terminal, and the internal structure diagram thereof can be as shown in Figure 4 The computer device comprises a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data related to the medical image classification method. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement the medical image classification method in embodiment 2.

[0073] Those skilled in the art can understand, Figure 4The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above method embodiments.

[0074] Embodiment 5

[0075] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the medical image classification method in the embodiment 2.

[0076] Embodiment 6

[0077] The embodiment provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the medical image classification method in the embodiment 2.

[0078] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0079] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0080] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0081] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0082] The principles and implementation modes of the present application are described by applying specific examples herein. The above description of the embodiments is only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. A medical image classification model with an improved attention mechanism, characterized in that, The medical image classification model with an improved attention mechanism is an EfficientNet network that incorporates an ASAE module. The ASAE module is used to process the feature maps generated from medical images and output dual-attention enhanced feature maps. The EfficientNet network is used to output medical image categories based on the dual-attention enhanced feature maps. The ASAE module includes an SE part and an attention part. The dual attention-enhanced feature map includes a channel-weighted feature map and a region-enhanced feature map. The SE part is used to calculate the attention weights between different channels and generate the channel-weighted feature map based on the attention weights between different channels. The attention part is used to calculate the attention weights of different local regions within a single channel based on a zero-attention mechanism and generate the region-enhanced feature map based on the attention weights of different local regions. The zero-attention mechanism refers to adding a column or row of zero vectors to the key matrix and the value matrix, and using the zero vectors as the basis for calculating the attention score. Calculating the attention weights for different local regions within a single channel specifically includes: The feature map of each individual channel input is divided into several local regions of the same size; A linear mapping is performed on each local region to obtain a local region vector; Add positional encoding to each of the local region vectors to obtain the enhanced vector; The attention weights of the local regions are calculated based on each of the enhancement vectors.

2. A medical image classification method, characterized in that, The medical image classification method includes: Acquire medical images to be classified; The medical image to be classified is input into a trained medical image classification model, which outputs the medical image category. The medical image classification model adopts the image classification model with improved attention mechanism as described in claim 1.

3. The medical image classification method according to claim 2, characterized in that, The training process of the medical image classification model specifically includes: Obtain a medical image training dataset; The medical image training dataset is input into the medical image classification model, and the model parameters are fine-tuned through backpropagation until the medical image classification model converges.

4. The medical image classification method according to claim 2, characterized in that, The medical image is a dermoscopy image.

5. A medical image classification device, characterized in that, The medical image classification device includes: The image acquisition module is used to acquire medical images to be classified. The classification module is used to input the medical image to be classified into a trained medical image classification model and output the medical image category, wherein the medical image classification model adopts the image classification model with improved attention mechanism as described in claim 1.

6. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the medical image classification method according to any one of claims 2-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the medical image classification method according to any one of claims 2-4.

8. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the medical image classification method according to any one of claims 2-4.

Images