Class activation map generation model training method, generation method and related device

Abstract

The embodiment of the present invention provides a class activation map generation model training method, generation method and related equipment, which relate to the field of image processing technology. Multiple groups of sample data are obtained, and multiple prototype vectors corresponding to each labeled category are initialized. The skin feature map and the corresponding skin structure mask are input into the medical attention guidance module. After the attention mechanism is processed, a fused feature map after fusion of attention is obtained. The fused feature map corresponding to the key frame is input into the fully connected classification layer to obtain the predicted category of the key frame. The fused feature map and each prototype vector are input into the class activation map generation module for weighted fusion to obtain the class activation map of each labeled category. The joint training loss is calculated for each predicted category, each labeled category, each lesion mask and each type of activation map to update the class activation map generation model. Through the multi-prototype activation mechanism, the overlap between different class activation maps can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a class activation map generation model training method, a generation method and related equipment. Background Art

[0002] The incidence of skin diseases continues to rise, becoming a major global health issue. Traditional pathological examination, the gold standard for diagnosis, faces challenges such as high invasiveness, high costs, and high physician subjectivity.

[0003] In recent years, the combination of non-invasive skin ultrasound imaging and artificial intelligence-assisted diagnosis has become a research hotspot. In medical image interpretability research, class activation map technology has attracted widespread attention due to its ability to intuitively express the model's focus areas.

[0004] However, in current class activation map technology, when there are multiple lesions in ultrasound images, different class activation maps often overlap, resulting in distorted interpretation. Summary of the Invention

[0005] In view of this, an object of an embodiment of the present invention is to provide a class activation map generation model training method, generation method and related devices to at least partially improve the above-mentioned problems.

[0006] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0007] In a first aspect, an embodiment of the present invention provides a method for training a class activation map generation model, comprising:

[0008] Acquire multiple sets of sample data; wherein each set of the sample data includes a skin ultrasound video and a labeled category of the skin ultrasound video, the skin ultrasound video includes multiple consecutive skin ultrasound images, the skin ultrasound images include key frames and non-key frames, and each set of the sample data also includes a skin structure mask of each skin ultrasound image and a lesion mask of the key frame;

[0009] Initializing a plurality of prototype vectors corresponding to each of the labeled categories according to the labeled categories of all the sample data and preset prototype hyperparameters; wherein the prototype hyperparameters are used to specify the number of prototype vectors corresponding to each of the labeled categories, and the prototype vectors represent different morphological subtype features under the same labeled category;

[0010] For each skin ultrasound image in each set of sample data, the skin feature map and the corresponding skin structure mask are input into a medical attention guidance module, and processed by an attention mechanism to obtain a fused feature map after fusion of attention; wherein the skin feature map is extracted from the skin ultrasound image;

[0011] input the fusion feature map corresponding to the key frame into a full connection classification layer to obtain a predicted category of the key frame;

[0012] input the fusion feature map and each prototype vector into a class activation map generation module to perform weighted fusion to obtain a class activation map of each labeled category;

[0013] calculate a joint training loss based on each predicted category, each labeled category, each lesion mask, and each class activation map, and update the class activation map generation model according to the joint training loss.

[0014] Optionally, the skin feature map and the corresponding skin structure mask are input into a medical attention guide module to obtain a fusion feature map after fusion attention through attention mechanism processing, and the method comprises the following steps.

[0015] input the skin structure mask and the skin feature map into a spatial attention unit to obtain a spatial attention map through spatial attention processing;

[0016] input the skin feature map into a channel attention unit to obtain a channel attention map through channel attention processing;

[0017] point multiply the skin feature map, the spatial attention map, and the channel attention map to obtain a fusion feature map after fusion attention.

[0018] Optionally, the spatial attention map is calculated by the following formula:

[0019]

[0020] wherein, the skin structure mask is, the skin feature map is, is an adjustable hyperparameter, is convolution, is an activation function;

[0021] the channel attention map is calculated by the following formula:

[0022]

[0023] wherein, the skin feature map is, is a global average pooling, is a max pooling, and is convolution, is a ReLU activation.

[0024] Optionally, the inputting the fusion feature map and each prototype vector into a class activation map generation module comprises:

[0025] calculating an inner product match between each feature point of the fusion feature map and each prototype vector to obtain a prototype activation map of each prototype vector;

[0026] inputting the fusion feature map into a weight prediction unit to obtain a weight map of each prototype vector;

[0027] for each annotation class, weighted summing each prototype activation map and each weight map to obtain a class activation map of the annotation class.

[0028] Optionally, the prototype activation map is calculated by the following formula:

[0029]

[0030] wherein, is the jth prototype activation map of the annotation class k, is the fusion feature map, is the jth prototype vector of the annotation class k, represents and is the inner product of

[0031] The weight map is calculated by the following formula:

[0032]

[0033] wherein, is the jth prototype activation map of the annotation class k, is the fusion feature map, is an activation function, is convolution;

[0034] The class activation map is calculated by the following formula:

[0035]

[0036] wherein, is the class activation map of the annotation class k, and K is the number of prototype vectors of the annotation class k.

[0037] Optionally, the calculation formula of the joint training loss is:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] wherein, is a cross-entropy loss, is a probability of predicting a labeled class i, is the reciprocal of the proportion of samples of the labeled class i, is an auxiliary adjustment factor; is an IoU similarity loss, is the class activation map, is the lesion mask; is a temporal consistency loss, are class activation maps of two consecutive skin ultrasound images in the skin ultrasound video respectively, is an L2 norm; is a multi-lesion decoupling regularization term, represents and are different annotation types, are class activation maps of different annotation types of the same skin ultrasound image respectively, is a Hadamard product, is an L1 norm; , , , are loss weights.

[0044] In a second aspect, an embodiment of the present application provides a class activation map generation method, comprising:

[0045] obtaining a key frame in a skin ultrasound video to be classified;

[0046] inputting the key frame into a pre-trained feature extraction model to obtain a key frame feature;

[0047] inputting the key frame feature into a pre-trained hierarchical model to obtain a skin structure mask;

[0048] inputting the key frame feature and the skin structure mask into a class activation map generation model to obtain a class activation map of each class of the key frame; the class activation map generation model is trained by the class activation map generation model training method of the first aspect.

[0049] In a third aspect, an embodiment of the present application provides a class activation map generation model training device, comprising:

[0050] The sample data acquisition unit is configured to acquire a plurality of groups of sample data, wherein each group of sample data comprises a skin ultrasound video and a labeled category of the skin ultrasound video, the skin ultrasound video comprises a plurality of continuous skin ultrasound images, the skin ultrasound image comprises a key frame and a non-key frame, and each group of sample data further comprises a skin structure mask of each skin ultrasound image and a lesion mask of the key frame.

[0051] The prototype vector initialization unit is configured to initialize a plurality of prototype vectors corresponding to each labeled category according to the labeled categories of all the sample data and preset prototype hyperparameters, wherein the prototype hyperparameters are used to specify the number of prototype vectors corresponding to each labeled category, and the prototype vectors represent different morphological subtype features under the same labeled category.

[0052] The feature fusion unit is configured to input a skin feature map and a corresponding skin structure mask into a medical attention guide module for each skin ultrasound image in each group of sample data, and obtain a fusion feature map after fusion attention through an attention mechanism.

[0053] The category prediction unit is configured to input the fusion feature map corresponding to the key frame into a fully connected classification layer to obtain a predicted category of the key frame.

[0054] The class activation map prediction unit is configured to input the fusion feature map and each prototype vector into a class activation map generation module to perform weighted fusion and obtain a class activation map of each labeled category.

[0055] The loss calculation unit is configured to calculate a joint training loss from each predicted category, each labeled category, each lesion mask and each class activation map, and update the class activation map generation model according to the joint training loss.

[0056] In a fourth aspect, an embodiment of the present application provides a class activation map generation device, comprising:

[0057] The key frame acquisition unit is configured to acquire a key frame in a skin ultrasound video to be classified.

[0058] The key frame feature extraction unit is configured to input the key frame into a pre-trained feature extraction model to obtain a key frame feature.

[0059] The skin structure extraction unit is configured to input the key frame feature into a pre-trained hierarchical model to obtain a skin structure mask.

[0060] The class activation map generation unit is configured to input the key frame features and the skin structure mask into a class activation map generation model to obtain a class activation map of each class of the key frame, and the class activation map generation model is trained by the class activation map generation model training method of the first aspect.

[0061] In a fifth aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the class activation map generation model training method of any one of the first aspect when executing the computer program.

[0062] The class activation map generation model training method and the generation method provided by the embodiments of the present application can assign multiple learnable prototype vectors to each class through a multi-prototype parameterized activation mechanism, thereby capturing the diversity within the class, more finely matching the local features related to the class in the image, and reducing the overlapping part of the generated class activation map and improving the interpretability of the class activation.

[0063] In order to make the above objectives, characteristics and advantages of the present application more apparent and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0065] Figure 1 A schematic structural block diagram of an electronic device provided by the embodiments of the present application is shown in FIG. 1;

[0066] Figure 2 A flowchart of a class activation map generation model training method provided by the embodiments of the present application is shown in FIG. 2;

[0067] Figure 3 A schematic structural block diagram of a class activation map generation model provided by the embodiments of the present application is shown in FIG. 3;

[0068] Figure 4 A schematic structural block diagram of a class activation map generation model provided by the embodiments of the present application is shown in FIG. 4;

[0069] Figure 5 A flowchart of a class activation map generation model training method provided by the embodiments of the present application is shown in FIG. 5;

[0070] Figure 6A flowchart of a class activation map generation method according to an embodiment of the present application is shown in FIG. 6.

[0071] Figure 7 A schematic structural block diagram of a class activation map generation model according to an embodiment of the present application is shown in FIG. 7.

[0072] Figure 8 A training flowchart of a class activation map generation model according to an embodiment of the present application is shown in FIG. 8.

[0073] Figure 9 A flowchart of a class activation map generation method according to an embodiment of the present application is shown in FIG. 6.

[0074] Figure 10 A schematic structural block diagram of a class activation map generation model training device according to an embodiment of the present application is shown in FIG. 9.

[0075] Figure 11 A schematic structural block diagram of a class activation map generation device according to an embodiment of the present application is shown in FIG. 10.

[0076] Icon: 100-electronic device; 101-memory; 102-communication interface; 103-processor; 104-communication bus; 300-class activation map generation model; 310-medical attention guiding module; 311-space attention unit; 312-channel attention processing; 320-class activation map generation module; 321-weight prediction unit; 330-full connection classification layer; 500-class activation map generation model training device; 510-sample data acquisition unit; 520-prototype vector initialization unit; 530-feature fusion unit; 540-class prediction unit; 550-class activation map prediction unit; 560-loss calculation unit; 600-class activation map generation device; 610-key frame acquisition unit; 620-key frame feature extraction unit; 630-skin structure extraction unit; 640-class activation map generation unit. DETAILED DESCRIPTION

[0077] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0078] The following detailed description of embodiments of the application in the drawings provided is not intended to limit the scope of the application as claimed, but merely represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the application.

[0079] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0080] It should be noted that in this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitation, the element defined by the sentence "including a…" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0081] As described in the background, in the traditional class activation map technology, the class activation map generation depends on the weighted sum (usually based on GAP) between the classifier weights and the feature map, which ignores the morphological differences within the class (such as different subtypes of melanoma), resulting in poor performance of the generated class activation map when there are multiple lesions or intra-class variations.

[0082] Based on the above, the embodiments of the application provide a class activation map generation model training method, a generation method and related equipment, which improve the class specificity, spatial precision and multi-lesion decoupling ability of the class activation map through multi-prototype parameterization modeling, and enhance the accuracy and clinical credibility of medical image interpretation.

[0083] To realize the flow steps and functions of various examples of the application, please refer to Figure 1 , Figure 1An exemplary structural block diagram of an electronic device is provided for embodiments of the present application. The electronic device 100 includes a memory 101 and a processor 103, which are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0084] The electronic device 100 can be, but is not limited to, a personal computer (PC), a server, a distributed computer, and the like. It can be understood that the electronic device 100 is also not limited to a physical server, but can also be a virtual machine on a physical server, a virtual machine constructed based on a cloud platform, and the like, which can provide a computer having the same function as the server or the virtual machine. The operating system of the electronic device 100 can be, but is not limited to, a Windows system, a Linux system, and the like.

[0085] The memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and the like.

[0086] The communication connection between the electronic device 100 and an external device is realized through at least one communication interface 102 (which can be wired or wireless).

[0087] The processor 103 can be an integrated circuit chip with processing capability. In implementation process, each step of the embodiments of the present application can be completed by integrated logic circuit of hardware in the processor 103 or by instructions in the form of software. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0088] It can be understood that Figure 1 The structure shown is only schematic, and the electronic device 100 can further include more or fewer components than those shown, or have a different configuration of components than those shown. Figure 1 The components shown in the figures can be implemented in hardware, software, or a combination thereof. Figure 1 The components shown in the figures can be implemented in hardware, software, or a combination thereof. Figure 1 The components shown in the figures can be implemented in hardware, software, or a combination thereof.

[0089] The following will be exemplarily described with respect to the class activation map generation model training method provided by the present application. Referring to Figure 2 , the method execution subject can be the electronic device 100 shown in the above Figure 1 , and the method includes the following steps as shown in the above Figure 2 .

[0090] S210: Obtain a plurality of groups of sample data.

[0091] Each group of sample data includes a skin ultrasound video and a labeled class of the skin ultrasound video, the skin ultrasound video includes a plurality of continuous skin ultrasound images, the skin ultrasound images include key frames and non-key frames, and each group of sample data further includes a skin structure mask of each skin ultrasound image and a lesion mask of the key frames.

[0092] S220: Initialize a plurality of prototype vectors corresponding to each labeled class according to the labeled classes of all sample data and preset prototype hyperparameters.

[0093] The prototype hyperparameters are used to specify the number of prototype vectors corresponding to each labeled class, and the prototype vectors represent different morphological subtype features under the same labeled class.

[0094] S230: For each skin ultrasound image in each set of sample data, input the skin feature map and the corresponding skin structure mask into the medical attention guide module, and obtain the fusion feature map after attention mechanism processing.

[0095] Wherein, the skin feature map is extracted from the skin ultrasound image.

[0096] S240: Input the fusion feature map corresponding to the key frame into the full connection classification layer to obtain the predicted category of the key frame.

[0097] S250: Input the fusion feature map and each prototype vector into the class activation map generation module for weighted fusion to obtain the class activation map of each annotation category.

[0098] S260: Calculate the joint training loss according to the predicted category, the annotation category, the lesion mask and the class activation map, and update the class activation map generation model according to the joint training loss.

[0099] To train the class activation map generation model, first prepare multiple sets of sample data and initialize multiple prototype vectors.

[0100] Each set of sample data is a piece of skin ultrasound video including multiple continuous skin ultrasound images, one of which is a key frame, for example, a frame with blood flow peak value is taken as a key frame.

[0101] In addition, each skin ultrasound image also has a corresponding skin structure mask, which represents the layered dissection area of the skin epidermis, dermis and subcutaneous tissue. The skin structure mask is used to provide prior guidance for subsequent feature extraction of the skin ultrasound image. The skin structure mask in the sample data can be obtained by doctor's annotation of the skin ultrasound image, or can be obtained by segmentation of the skin ultrasound image by a pre-trained layered model.

[0102] In addition, each piece of skin ultrasound video is also annotated with an annotation category, for example, the annotation categories include melanoma, hemangioma, basal cell carcinoma, and the lesion mask of the key frame, which is obtained by artificial annotation by doctors.

[0103] So far, the sample data for training is prepared, and next, a plurality of prototype vectors are initialized, which are used for modeling intra-class variation, such as different morphological subtypes of melanoma. The prototype vectors are initialized according to the sample data and prototype hyperparameters. In the sample data, how many labeled categories are labeled, that is, how many categories the skin ultrasound video is classified into. For example, the classification of skin ultrasound has three categories: melanoma, hemangioma, and basal cell carcinoma (of course, there are more than three in reality, which is just an example here). The prototype hyperparameters are how many prototypes each category has, which is predefined, and the number of prototypes in each category can be different or the same. In order to simplify the description, the number of prototypes in each category is the same in the embodiment of the present application, for example, there are 3 prototype vectors in each category. Then 9 prototype vectors are initialized . In the subsequent training process, the is updated and optimized until the training is completed.

[0104] After the sample data and the initialized prototype vectors are prepared, the training process of the class activation map generation model is entered. Referring to Figure 3 , the class activation map generation model 300 includes a medical attention guide module 310, a class activation map generation module 320, and a fully connected classification layer 330. After forward propagation through the medical attention guide module 310 and the class activation map generation module 320, the joint training loss is calculated, and then the parameters of the medical attention guide module 310 and the class activation map generation module 320 are updated by back propagation.

[0105] Each skin ultrasound image in each set of sample data is processed to generate a class activation map for each labeled category. Then, loss calculation and back propagation update are performed.

[0106] The skin feature map of the skin ultrasound image is extracted using a feature extraction model, which is a well-trained model and can be a MixConv neural network or a ResNet neural network. The skin feature map and the corresponding skin structure mask are input into the medical attention guide module, and after attention mechanism processing, the fusion feature map can highlight the region guide of different structure layers of the epidermis, dermis, and subcutaneous tissue.

[0107] Since the sample data only classifies the skin ultrasound video as a whole, we only need to classify the key frames in the skin ultrasound video and perform subsequent loss calculation with the overall classification. It can be understood that each ultrasound image in the skin ultrasound video can also be classified and labeled, and in the processing, all frames in the skin ultrasound video are classified. In the embodiment of the present application, the fusion feature map corresponding to the key frame is input into the fully connected classification layer to obtain the predicted category of the key frame.

[0108] After the class activation map generation module processes the fusion feature map and each prototype vector, a class activation map of each labeled class can be generated.

[0109] Finally, the joint training loss is calculated based on the predicted classes, the labeled classes, the lesion masks, and the class activation maps, and the class activation map generation model is updated based on the joint training loss.

[0110] The method assigns multiple learnable prototype vectors to each class through the multi-prototype parameterized activation mechanism, thereby capturing the diversity within the class, more finely matching the local features related to the class in the image, and reducing the overlapping parts of the generated class activation map to improve the interpretability of the class activation.

[0111] In order to capture more rich information of the skin feature map, the skin feature map can be subjected to spatial attention processing and channel attention processing. Therefore, referring to Figure 4 , the medical attention guide module 310 can include a spatial attention unit 311 and a channel attention processing unit 312. Referring to Figure 5 The above step S230 can include the following steps:

[0112] S231: input the skin structure mask and the skin feature map into the spatial attention unit, and obtain a spatial attention map after spatial attention processing.

[0113] S232: input the skin feature map into the channel attention unit, and obtain a channel attention map after channel attention processing.

[0114] S233: point multiply the skin feature map, the spatial attention map, and the channel attention map to obtain a fusion feature map after fusion attention.

[0115] There are multiple ways to perform spatial attention processing on the skin feature map using the skin structure mask as prior information. In an optional implementation, the processing process can be represented by the following formula:

[0116]

[0117] wherein, is the skin structure mask, is the skin feature map, is an adjustable hyperparameter, is convolution, is an activation function.

[0118] The skin structure mask and the skin feature map are input into the formula to obtain the spatial attention map.

[0119] The channel attention processing of the skin feature map adopts a CBAM channel attention mechanism. Exemplarily, a channel attention map is calculated by the following formula:

[0120]

[0121] wherein, is the skin feature map, is global average pooling, is max pooling, and is convolution, is ReLU activation.

[0122] After the spatial attention map and the channel attention map, the skin feature map can be fused. The fusion process is as follows:

[0123]

[0124] wherein, is the fused feature map, is the skin feature map, is the spatial attention map, is the channel attention map.

[0125] For the fusion of the prototype vector and the fused feature map, reference can be made to Figure 6 The above step S250 can include the following steps:

[0126] S251: Calculate the inner product match of each feature point of the fused feature map and each prototype vector to obtain a prototype activation map of each prototype vector.

[0127] S252: Input the fused feature map into a weight prediction unit to obtain a weight map of each prototype vector.

[0128] S253: For each labeled category, weighted sum each prototype activation map and each weight map to obtain a class activation map of the labeled category.

[0129] Exemplarily, the embodiment of the present application includes three categories, namely melanoma, hemangioma and basal cell carcinoma, each category has three prototype vectors, and we describe the generation of a class activation map of one category. For example, melanoma has three prototype vectors , , .

[0130] First, calculate the inner product match of each feature point of the fused feature map and , , to obtain 、 、 each prototype activation map is calculated by the following equation:

[0131]

[0132] wherein, is the jth prototype activation map of the labeled class k, is the fused feature map, is the jth prototype vector of the labeled class k, denotes the inner product of and

[0133] For example, the three prototype activation maps of melanoma are calculated as 、 、 .

[0134] Meanwhile, the weight prediction unit predicts the weight map of the prototype vector, see Figure 7 , the class activation map generation module 320 comprises a weight prediction unit 321. The fused feature map is input into the weight prediction unit 321, and the weight map of each prototype vector is predicted. The weight prediction unit 321 can use a light convolution, for example, the weight map can be calculated by the following equation:

[0135]

[0136] wherein, is the jth prototype activation map of the labeled class k, is the fused feature map, is an activation function, is a convolution.

[0137] The parameters of the weight prediction unit 321 will also be updated in the training process.

[0138] For example, the weight maps of the three prototype vectors of melanoma are predicted as 、 、 .

[0139] Finally, the dot product and summation of each prototype activation map and each weight map of each class are obtained, i.e. the class activation map of each class. The class activation map can be calculated by the following equation:

[0140]

[0141] wherein, is the class activation map of the annotation class k.

[0142] For example, the class activation map of melanoma is .

[0143] The formula for calculating the joint training loss in step S260 can be:

[0144]

[0145]

[0146]

[0147]

[0148]

[0149] wherein, is the cross-entropy loss, is the probability of predicting the annotation class i, is the reciprocal of the proportion of samples of the annotation class i, is the auxiliary adjustment factor; is the IoU similarity loss, is the class activation map, is the lesion mask; is the temporal consistency loss, are the class activation maps of two consecutive skin ultrasound images in the skin ultrasound video, is the L2 norm; is the multi-lesion decoupling regularization term, denotes and are different annotation types, are the class activation maps of different annotation types of the same skin ultrasound image, is the Hadamard product, is the L1 norm; , , , are the loss weights.

[0150] Continuing the above example to illustrate the joint training loss, assuming that the proportions of skin ultrasound videos annotated as melanoma, hemangioma, and basal cell carcinoma in the sample data are 20%, 20%, and 40% respectively, then are , , , The proportion of the classification prediction of the key frame in the sample data in this batch is melanoma, hemangioma, and basal cell carcinoma. The cross-entropy loss can be calculated through these data. The IoU similarity loss is calculated by the class activation map of the key frame and the lesion mask. Therefore, the results of the key frame are used for the cross-entropy loss and the IoU similarity loss.

[0151] The temporal consistency loss is used to prevent the class activation map from jumping over time by introducing a temporal smoothing loss. The temporal consistency loss is calculated by using two consecutive images in the same skin ultrasound video.

[0152] The multi-lesion decoupling regularization term is calculated by using the class activation maps corresponding to the same skin ultrasound image. For example, the class activation maps of a skin ultrasound image include the class activation map of melanoma, the class activation map of hemangioma, and the class activation map of basal cell carcinoma. The sum of the L2 norms between the three class activation maps is calculated.

[0153] Referring to Figure 8 , the class activation map generation model training method is further described from the forward propagation and the backward propagation. First, the prototype vector is initialized according to the labeled category and the prototype hyperparameter. The skin feature map and the skin structure mask extracted from the skin ultrasound image in the sample are input into the medical attention guide module to obtain a feature fusion map. The prototype vector and the feature fusion map are input into the class activation map generation module to generate a class activation map. Meanwhile, the feature fusion map is input into a fully connected classification layer to obtain a key frame prediction category. The joint training loss is calculated according to the key frame prediction category, the class activation map, the labeled category, and the lesion mask. Finally, the joint training loss is used for the forward propagation to update the parameters of the fully connected classification layer, the class activation map generation module, the medical attention guide module, and the prototype vector.

[0154] Further, the embodiment of the present application further provides a class activation map generation method, referring to Figure 9 , the class activation map generation method comprises the following steps:

[0155] S410: obtaining a key frame in a skin ultrasound video to be classified.

[0156] S420: inputting the key frame into a pre-trained feature extraction model to obtain a key frame feature.

[0157] S430: inputting the key frame feature into a pre-trained hierarchical model to obtain a skin structure mask.

[0158] S440: inputting the key frame feature and the skin structure mask into a class activation map generation model to obtain a class activation map of each category of the key frame; the class activation map generation model is trained by the class activation map generation model training method.

[0159] The skin is scanned by an ultrasonic device to obtain a skin ultrasonic video to be classified, key frames are extracted from the skin ultrasonic video, features of the key frames are extracted by using a pre-trained feature extraction model, the key frame features are input into a pre-trained hierarchical model to obtain a skin structure mask, and finally the key frame features and the skin structure mask are input into a class activation map generation model to obtain class activation maps of each category of the key frames.

[0160] Further, the embodiment of the present application also provides a class activation map generation model training device, which is used for Figure 10 The class activation map generation model training device 500 comprises:

[0161] A sample data acquisition unit 510 is configured to acquire a plurality of groups of sample data, wherein each group of sample data comprises a skin ultrasonic video and a labeled category of the skin ultrasonic video, the skin ultrasonic video comprises a plurality of continuous skin ultrasonic images, the skin ultrasonic images comprise key frames and non-key frames, and each group of sample data further comprises a skin structure mask of each skin ultrasonic image and a lesion mask of the key frames.

[0162] A prototype vector initialization unit 520 is configured to initialize a plurality of prototype vectors corresponding to each labeled category according to the labeled categories of all sample data and preset prototype hyperparameters, wherein the prototype hyperparameters are used to specify the number of prototype vectors corresponding to each labeled category, and the prototype vectors represent different morphological subtype features under the same labeled category.

[0163] A feature fusion unit 530 is configured to input a skin feature map and a corresponding skin structure mask into a medical attention guide module for each skin ultrasonic image in each group of sample data, and obtain a fusion feature map after fusion attention through an attention mechanism.

[0164] A category prediction unit 540 is configured to input the fusion feature map corresponding to the key frames into a fully connected classification layer to obtain a predicted category of the key frames.

[0165] A class activation map prediction unit 550 is configured to input the fusion feature map and the prototype vectors into a class activation map generation module to perform weighted fusion to obtain class activation maps of each labeled category.

[0166] A loss calculation unit 560 is configured to calculate a joint training loss from the predicted categories, the labeled categories, the lesion masks and the class activation maps, and update the class activation map generation model according to the joint training loss.

[0167] Further, the embodiment of the present application also provides a class activation map generation device, which is used for Figure 11 The class activation map generation device 600 comprises:

[0168] The key frame acquisition unit 610 is configured to acquire a key frame in a skin ultrasound video to be classified.

[0169] The key frame feature extraction unit 620 is configured to input the key frame into a pre-trained feature extraction model to obtain key frame features.

[0170] The skin structure extraction unit 630 is configured to input the key frame features into a pre-trained hierarchical model to obtain a skin structure mask.

[0171] The class activation map generation unit 640 is configured to input the key frame features and the skin structure mask into a class activation map generation model to obtain a class activation map of each class of the key frame, and the class activation map generation model is trained by the class activation map generation model training method.

[0172] In summary, the class activation map generation model training method, the generation method and the related device provided by the embodiments of the present application introduce a multi-prototype parameterization modeling mechanism, assign multiple learnable prototype vectors to each class, better capture the diversity within the same class, improve the matching accuracy of the class activation map and the local region of the image, and effectively reduce the overlapping area between multiple class activation maps; a medical attention guide module is introduced, including a spatial attention unit and a channel attention unit, the spatial attention of the feature map is guided by the skin structure mask, the focusing on the skin structure region is improved, and the feature fusion is performed in combination with the medical prior knowledge (the skin structure mask), so that the class activation map is more in line with the anatomical structure of the medical image; the time consistency loss is used to constrain the class activation map in the video sequence in the time dimension, and the class activation map is prevented from jumping sharply with time.

[0173] In the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only schematic. For example, the flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operations of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from those shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0174] In addition, each functional module in various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0175] If the functions are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a computer readable storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0176] The above is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0177] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended to include all changes falling within the meaning and scope of the equivalent elements of the claims. Any reference signs in the claims should not be regarded as limiting the claims involved.

Claims

1. A class activation map generation model training method, characterized in that: include: Acquire multiple sets of sample data; wherein each set of the sample data includes a skin ultrasound video and a labeled category of the skin ultrasound video, the skin ultrasound video includes multiple consecutive skin ultrasound images, the skin ultrasound images include key frames and non-key frames, and each set of the sample data also includes a skin structure mask of each skin ultrasound image and a lesion mask of the key frame; Initializing a plurality of prototype vectors corresponding to each of the labeled categories according to the labeled categories of all the sample data and preset prototype hyperparameters; wherein the prototype hyperparameters are used to specify the number of prototype vectors corresponding to each of the labeled categories, and the prototype vectors represent different morphological subtype features under the same labeled category; For each skin ultrasound image in each set of sample data, the skin feature map and the corresponding skin structure mask are input into a medical attention guidance module, and processed by an attention mechanism to obtain a fused feature map after fusion of attention; wherein the skin feature map is extracted from the skin ultrasound image; Inputting the fused feature map corresponding to the key frame into a fully connected classification layer to obtain a predicted category of the key frame; Inputting the fused feature map and each of the prototype vectors into a class activation map generation module for weighted fusion to obtain a class activation map of each of the labeled categories; A joint training loss is calculated for each of the predicted categories, each of the labeled categories, each of the lesion masks, and each of the class activation maps, and the class activation map generation model is updated according to the joint training loss.

2. The class activation map generation model training method according to claim 1, characterized in that The skin feature map and the corresponding skin structure mask are input into the medical attention guidance module, and processed by the attention mechanism to obtain a fused feature map after fusion of attention, including: Inputting the skin structure mask and the skin feature map into a spatial attention unit, and performing spatial attention processing to obtain a spatial attention map; Inputting the skin feature map into a channel attention unit and performing channel attention processing to obtain a channel attention map; Perform dot multiplication on the skin feature map, the spatial attention map, and the channel attention map to obtain a fused feature map after fusion of attention.

3. The class activation map generation model training method according to claim 2, wherein The spatial attention map is calculated by the following formula: in, for the skin structure mask, is the skin feature map, is a tunable hyperparameter, for convolution, is the activation function; The channel attention map is calculated by the following formula: in, is the skin feature map, is the global average pooling, is the maximum pooling, and for convolution, is the ReLU activation.

4. The class activation map generation model training method according to claim 1, wherein The step of inputting the fused feature map and each prototype vector into a class activation map generation module for weighted fusion to obtain a class activation map of each labeled category includes: Calculate the inner product matching between each feature point of the fused feature map and each prototype vector to obtain a prototype activation map of each prototype vector; Inputting the fused feature map into a weight prediction unit to obtain a weight map of each prototype vector; For each of the labeled categories, the prototype activation maps and the weight maps are weighted and summed to obtain a class activation map of the labeled category.

5. The class activation map generation model training method according to claim 4, characterized in that The prototype activation map is calculated by the following formula: in, is the j-th prototype activation map of the labeled category k, is the fusion feature map, is the j-th prototype vector of the labeled category k, express and The inner product of The weight map is calculated by the following formula: in, is the j-th prototype activation map of the labeled category k, is the fusion feature map, is the activation function, for convolution; The class activation map is calculated by the following formula: in, is the class activation map of the labeled category k, and K is the number of prototype vectors of the labeled category k.

6. The class activation map generation model training method according to claim 1, characterized in that The calculation formula of the joint training loss is: in, is the cross entropy loss, is the probability of predicting the labeled category i, is the reciprocal of the proportion of samples labeled category i, is the auxiliary adjustment factor; is the IoU similarity loss, is the class activation graph, mask for the lesion; is the temporal consistency loss, are the class activation maps of two consecutive skin ultrasound images in the skin ultrasound video, is the L2 norm; is the multi-lesion decoupling regularization term, express and For different annotation types, are class activation maps of different annotation types for the same skin ultrasound image, is the Hadamard product, is the L1 norm; , , , is the weight of each loss.

7. A method for generating a class activation map, characterized in that: include: Obtain key frames in the skin ultrasound video to be classified; Inputting the key frame into a pre-trained feature extraction model to obtain key frame features; Inputting the key frame features into a pre-trained hierarchical model to obtain a skin structure mask; The key frame features and the skin structure mask are input into a class activation map generation model to obtain class activation maps of each category of the key frame; the class activation map generation model is trained by the class activation map generation model training method according to any one of claims 1 to 6.

8. A class activation map generation model training device, characterized in that: include: A sample data acquisition unit, configured to acquire multiple sets of sample data; wherein each set of sample data includes a skin ultrasound video and a labeled category of the skin ultrasound video, the skin ultrasound video includes multiple consecutive skin ultrasound images, the skin ultrasound images include key frames and non-key frames, and each set of sample data also includes a skin structure mask of each skin ultrasound image and a lesion mask of the key frame; A prototype vector initialization unit is used to initialize multiple prototype vectors corresponding to each of the labeled categories according to the labeled categories of all the sample data and preset prototype hyperparameters; wherein the prototype hyperparameters are used to specify the number of prototype vectors corresponding to each of the labeled categories, and the prototype vectors represent different morphological subtype features under the same labeled category; a feature fusion unit, configured to input, for each skin ultrasound image in each set of sample data, a skin feature map and a corresponding skin structure mask into a medical attention guidance module, and process the skin feature map and the corresponding skin structure mask through an attention mechanism to obtain a fused feature map after fusion of attention; wherein the skin feature map is extracted from the skin ultrasound image; A category prediction unit, configured to input the fused feature map corresponding to the key frame into a fully connected classification layer to obtain a predicted category of the key frame; A class activation map prediction unit is used to input the fused feature map and each prototype vector into a class activation map generation module for weighted fusion to obtain a class activation map of each labeled category; The loss calculation unit is used to calculate the joint training loss of each of the predicted categories, each of the labeled categories, each of the lesion masks, and each of the class activation maps, and update the class activation map generation model according to the joint training loss.

9. A class activation map generating device, characterized in that: include: A key frame acquisition unit, used to acquire key frames in the skin ultrasound video to be classified; A key frame feature extraction unit, configured to input the key frame into a pre-trained feature extraction model to obtain key frame features; a skin structure extraction unit, configured to input the key frame features into a pre-trained hierarchical model to obtain a skin structure mask; A class activation map generation unit is used to input the key frame features and the skin structure mask into a class activation map generation model to obtain class activation maps of each category of the key frame; the class activation map generation model is trained by the class activation map generation model training method according to any one of claims 1 to 6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the class activation map generation model training method according to any one of claims 1 to 6 is implemented.

Images