Class activation mapping graph generation method and device, equipment and storage medium thereof
By extracting and clustering images of medical images, class activation maps are generated, which solves the problem of inaccurate class activation maps and low generation efficiency in the prior art, and achieves more efficient and accurate class activation map generation.
Patent Information
- Application Number
- CN202411667922.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-03-25
AI Technical Summary
The existing class activation map generation technology is easy to pay attention to image areas that do not need to be paid attention to, resulting in the generated class activation map inaccurate enough, increasing the amount of calculation, and inefficient generation.
By obtaining the target input image, input it into the pre-trained feature extraction model for image feature extraction to generate an image feature map. Then, based on the vector value of each pixel point in the image feature map, a preset clustering algorithm is used for clustering processing, and a representative feature of each cluster cluster is obtained, and a class activation map is generated based on the image feature map.
The calculation processing amount during the generation of class activation maps is reduced, the accuracy and generation efficiency of class activation maps are improved, and attention to unimportant image feature areas is avoided.
Smart Images
Figure CN119693662A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of R&D design and image processing technology, and is applied to the scenario of class activation map generation when extracting features from medical images, and in particular to a class activation map generation method, device, equipment and storage medium thereof. Background Art
[0002] In 2015, Zhou et al. proposed class activation mapping (CAM). This method is a technique for generating heat maps to highlight specific areas of an image class. It locates the relevant important areas in the input image samples that affect the model's decision category judgment. For specific inputs, the class activation mapping method can better reflect the areas that the deep model focuses on. Through this form of explanation map, it is possible to locate the important areas of relevant decisions, which is a better method of local explanation.
[0003] However, due to factors such as noise, the existing class activation map generation technology causes the generated class activation map to easily focus on areas that do not actually need to be paid attention to, thereby reducing the effectiveness and correctness of the class activation map. Moreover, paying more attention to areas that do not need to be paid attention to will also increase the amount of generation calculations, reducing the generation efficiency of the class activation map. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose a class activation map generation method, device, equipment and storage medium thereof to solve the problem that the existing class activation map generation technology is prone to focus on image areas that do not need to be paid attention to or are unimportant, resulting in the generated class activation map being inaccurate, the amount of generation calculation being increased and the generation efficiency being low.
[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for generating a class activation map, which adopts the following technical solution:
[0006] A method for generating a class activation map comprises the following steps:
[0007] Acquire a target input image, wherein the target input image includes a medical image for which class activation map generation is to be performed;
[0008] Inputting the target input image into a pre-trained feature extraction model to perform image feature extraction to obtain an image feature extraction result, wherein the feature extraction model is a feature extraction model based on a CNN convolutional neural network;
[0009] Generate an image feature map according to the image feature extraction result;
[0010] Based on the vector value of each pixel point in the image feature map, a preset clustering algorithm is used to perform clustering processing on all the pixel points in the image feature map to obtain a clustering processing result;
[0011] According to the clustering processing results, the representative features corresponding to each cluster are obtained;
[0012] According to the representative features corresponding to each cluster, a class activation map is generated based on the image feature map.
[0013] Furthermore, the step of clustering all the pixels in the image feature map using a preset clustering algorithm based on the vector value of each pixel in the image feature map to obtain a clustering result specifically includes:
[0014] Identify a vector value of each pixel, wherein the vector value includes an RGB value;
[0015] Using a K-means clustering algorithm, and clustering all pixels in the image feature map according to the vector value of each pixel, wherein the K value in the K-means clustering algorithm represents the number of cluster categories;
[0016] After clustering is completed, the pixel point sets contained in each cluster are obtained.
[0017] Furthermore, before executing the step of obtaining the pixel point sets respectively included in each cluster after clustering is completed, the method further includes:
[0018] Presetting a number of clustering result output channels equal to the number of clustering categories;
[0019] The step of obtaining the pixel point set contained in each cluster after clustering is completed specifically includes:
[0020] Identify the pixels contained in each cluster;
[0021] Binarization is performed on the pixels included in each cluster and the pixels not included in each cluster to obtain the binary image feature map corresponding to each cluster;
[0022] According to different clusters, different output channels are selected to output the corresponding binary image feature maps;
[0023] According to the binary image feature maps corresponding to all clusters, the pixel point sets contained in each cluster are obtained.
[0024] Furthermore, after executing the step of obtaining the pixel point set respectively included in each cluster according to the binary image feature maps respectively corresponding to all cluster clusters, the method further includes:
[0025] Based on the vector values of the pixels contained in all clusters, the mode vector value of each cluster is determined;
[0026] Identify all pixel points corresponding to the majority vector value of each cluster as important pixel points contained in the corresponding cluster, and identify the pixel points corresponding to the non-majority vector value as non-important pixel points contained in the corresponding cluster;
[0027] On the target binary image feature map, perform RGB value whitening on the non-important pixels contained in each cluster.
[0028] Obtain a binary image feature map corresponding to each cluster, retaining only important pixels.
[0029] Furthermore, the step of obtaining representative features corresponding to each cluster according to the clustering processing result specifically includes:
[0030] According to the spatial relationship between the key pixels contained in each cluster, the cluster center areas corresponding to all clusters are screened out;
[0031] The cluster center regions corresponding to all clusters are taken as representative feature regions and marked in the target input image to obtain an image marked with representative feature regions;
[0032] The image with the representative feature regions marked is input into the feature extraction model, and feature extraction is performed on the representative feature regions in the image to obtain representative features corresponding to each cluster.
[0033] Furthermore, the step of screening out the cluster center areas corresponding to all clusters respectively according to the spatial relationship between the key pixels contained in each cluster specifically includes:
[0034] Take each key pixel in the current cluster as the target pixel in turn;
[0035] In a circular area with the target pixel as the center point and the adjustable radius as the boundary, the number of other key pixel points is counted to obtain a number statistical value, wherein the adjustable radius value means that the radius value is dynamically adjusted for each round of statistics, and then the statistics are re-performed;
[0036] Until the number statistics corresponding to the target pixel point is greater than the number statistics corresponding to other key pixel points, the position information and the current radius value of the target pixel point are recorded;
[0037] Taking the position information as the center point and the current radius value as the radius, a circular area is divided as the cluster center area of the current cluster.
[0038] Furthermore, the step of marking the cluster center regions corresponding to all clusters as representative feature regions in the target input image to obtain an image marked with the representative feature regions specifically includes:
[0039] In the binary image feature map corresponding to each cluster, the corresponding representative feature areas are marked respectively;
[0040] Performing whole-image mapping synthesis on the binary image feature maps corresponding to all clusters and having corresponding representative feature regions marked, to obtain a mapping synthesis map marking the representative feature regions corresponding to all clusters;
[0041] According to the regional position information of the representative feature regions corresponding to all clusters in the mapping synthesis image, all the representative feature regions are marked in the target input image to obtain the image with the marked representative feature regions.
[0042] In order to solve the above technical problems, the embodiment of the present application also provides a class activation map generation device, which adopts the following technical solution:
[0043] A class activation map generating device, comprising:
[0044] A target input image acquisition module, used to acquire a target input image, wherein the target input image includes a medical image to be subjected to class activation map generation;
[0045] An image feature extraction module is used to input the target input image into a pre-trained feature extraction model to perform image feature extraction and obtain an image feature extraction result, wherein the feature extraction model is a feature extraction model based on a CNN convolutional neural network;
[0046] An image feature map generation module, used to generate an image feature map according to the image feature extraction result;
[0047] A pixel clustering processing module, used to perform clustering processing on all pixels in the image feature map based on the vector value of each pixel in the image feature map using a preset clustering algorithm to obtain a clustering processing result;
[0048] A representative feature acquisition module is used to obtain representative features corresponding to each cluster according to the clustering processing results;
[0049] The class activation map generation module is used to generate a class activation map based on the image feature map according to the representative features corresponding to each cluster.
[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0051] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned class activation map generation method when executing the computer-readable instructions.
[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0053] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the class activation map generation method as described above.
[0054] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0055] The class activation map generation method described in the embodiment of the present application is to obtain a target input image; input the target input image into a pre-trained feature extraction model to extract image features and obtain image feature extraction results; generate an image feature map based on the image feature extraction results; cluster all pixels in the image feature map based on the vector value of each pixel in the image feature map to obtain a clustering result; obtain representative features corresponding to each cluster cluster according to the clustering result; and generate a class activation map based on the image feature map according to the representative features corresponding to each cluster cluster. The class activation map generation method described in the present application first performs image feature clustering, and only realizes class activation map generation by focusing on the representative features corresponding to each cluster cluster. Compared with the existing class activation map generation technology, it is easy to focus on image areas that do not need to be focused on or are not important when generating class activation maps, resulting in the problem that the generated class activation map is not accurate enough. In this application, there is no need to focus on non-important image feature areas, which reduces the amount of computational processing when generating class activation maps, and at the same time, improves the correctness and generation efficiency of class activation maps. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0057] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0058] Figure 2 is a flow chart of an embodiment of a method for generating a class activation map according to the present application;
[0059] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;
[0060] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 303 is shown;
[0061] Figure 5 It is a flowchart of a specific embodiment of screening important pixels in the class activation map generation method described in the present application;
[0062] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown;
[0063] Figure 7 yes Figure 6 A flowchart of a specific embodiment of step 601 is shown;
[0064] Figure 8 yes Figure 6 A flowchart of a specific embodiment of step 602 is shown;
[0065] Fig. 9 is a structural schematic diagram of an embodiment of a class activation map generating device according to the present application;
[0066] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0068] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0069] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0070] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.
[0071] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0072] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.
[0073] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .
[0074] It should be noted that the class activation map generation method provided in the embodiment of the present application is generally executed by a server, and accordingly, the class activation map generation device is generally arranged in the server.
[0075] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0076] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for generating a class activation map according to the present application. The method for generating a class activation map comprises the following steps:
[0077] Step 201: Acquire a target input image, wherein the target input image includes a medical image for which class activation map generation is to be performed.
[0078] In this embodiment, the class activation map refers to a CAM (Class Activation Mapping) map, which is a visualization technique for explaining the decision-making process of a deep learning model. CAM generates a class activation map to show which areas of the input image contribute most to the output of the model when the model makes a specific decision. The CAM method, i.e., the class activation map generation method, is used to understand the decision-making process of a convolutional neural network CNN in image classification tasks.
[0079] Applying the class activation map generation method described in the present application to the class activation map generation scenario when performing feature extraction on medical images can help doctors and patients understand how to classify image areas in medical images, for example: classifying image areas corresponding to different human organs in medical images, and for example: classifying normal areas and pathological areas in the same organ in medical images.
[0080] Step 202: input the target input image into a pre-trained feature extraction model to perform image feature extraction to obtain an image feature extraction result, wherein the feature extraction model is a feature extraction model based on a CNN convolutional neural network.
[0081] In this embodiment, a feature extraction model based on a CNN convolutional neural network is used to perform feature extraction on the target input image to obtain corresponding image features, wherein the image features include: the vector value of each pixel in the image, i.e., the RGB value, the position information of each pixel in the image, the area size of a specific target organ in the image, the position information of the specific target organ, etc.
[0082] Image feature extraction is performed by inputting the target input image into a pre-trained feature extraction model to obtain an image feature extraction result, so as to facilitate subsequent clustering processing in combination with the image feature extraction result.
[0083] Step 203: Generate an image feature map according to the image feature extraction result.
[0084] Step 204: Based on the vector value of each pixel in the image feature map, a preset clustering algorithm is used to perform clustering processing on all the pixels in the image feature map to obtain a clustering processing result.
[0085] Specifically, by performing clustering processing on all pixels in the image feature map, a clustering processing result is obtained so that pixels with the same or similar image features can be clustered together, which is convenient for subsequent generation of class activation maps for image features with different clustering results, and also convenient for classifying different regions in the target input image in combination with different clustering results when actually performing image region classification.
[0086] Step 205: Obtain representative features corresponding to each cluster according to the clustering processing result.
[0087] Step 206: Generate a class activation map based on the image feature map according to the representative features corresponding to each cluster.
[0088] By first performing clustering processing on all pixel points in the image feature map to obtain a clustering processing result, and then obtaining representative features corresponding to each cluster cluster based on the clustering processing result, and finally, generating a class activation map based on the image feature map according to the representative features corresponding to each cluster cluster, the class activation map generation method described in the present application realizes class activation map generation by only focusing on the representative features corresponding to each cluster cluster.
[0089] Compared with the existing class activation map generation technology, it is easy to focus on unnecessary or unimportant image areas when generating class activation maps, resulting in the problem that the generated class activation maps are not accurate enough. The present application does not need to focus on non-important image feature areas, reduces the computational processing amount when generating class activation maps, and at the same time, improves the accuracy and generation efficiency of class activation maps.
[0090] In this embodiment, by obtaining a target input image; inputting the target input image into a pre-trained feature extraction model, performing image feature extraction, and obtaining an image feature extraction result; generating an image feature map according to the image feature extraction result; clustering all pixels in the image feature map based on the vector value of each pixel in the image feature map to obtain a clustering result; obtaining representative features corresponding to each cluster cluster according to the clustering result; generating a class activation map based on the image feature map according to the representative features corresponding to each cluster cluster. The class activation map generation method described in the present application first performs image feature clustering, and only realizes the generation of class activation maps by focusing on the representative features corresponding to each cluster cluster. Compared with the existing class activation map generation technology, when generating class activation maps, it is easy to focus on image areas that do not need to be focused on or are not important, resulting in the problem that the generated class activation map is not accurate enough. In this application, there is no need to focus on non-important image feature areas, which reduces the amount of computational processing when generating class activation maps, and at the same time, improves the correctness and generation efficiency of class activation maps.
[0091] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 204 shown includes the following steps:
[0092] Step 301, identifying a vector value of each pixel, wherein the vector value includes an RGB value;
[0093] Step 302, using a K-means clustering algorithm, and performing clustering processing on all pixels in the image feature map according to the vector value of each pixel, wherein the K value in the K-means clustering algorithm represents the number of cluster categories;
[0094] Specifically, the K-means clustering algorithm is used, and all the pixels in the image feature map are clustered according to the vector value of each pixel. By freely setting the K value, the number of classifications in the image area can be controlled in actual medical image classification applications.
[0095] Step 303: Obtain the pixel point sets contained in each cluster after clustering is completed.
[0096] In this embodiment, the vector value, that is, the RGB value, can be used for clustering. For example, if the K value is 10, then 10 RGB value ranges are set accordingly, and all pixels in the image feature map are clustered according to the 10 RGB value ranges.
[0097] By clustering all the pixels in the image feature map, a clustering result is obtained so that the pixels with the same or similar image features can be clustered together, which is convenient for subsequent generation of class activation maps for image features with different clustering results, and also convenient for classifying different regions in the target input image in combination with different clustering results when actually classifying image regions.
[0098] In this embodiment, before executing the step of obtaining the pixel point sets respectively included in each cluster after clustering is completed, that is, before step 303, the method further includes: presetting clustering result output channels equal in number to the clustering categories.
[0099] Continue to refer Figure 4 , Figure 4 yes Figure 3 The flowchart of a specific embodiment of step 303 shown includes the following steps:
[0100] Step 401, identifying the pixels contained in each cluster;
[0101] Step 402, binarizing the pixels included in each cluster and the pixels not included in each cluster to obtain a binary image feature map corresponding to each cluster;
[0102] Specifically, the binarization process includes not performing RGB value change processing on the pixels included in each cluster, and performing RGB value whitening processing on the pixels not included in each cluster. The RGB value whitening processing is to reset the RGB values of the pixels to 255, 255, 255.
[0103] Step 403, according to different clusters, select different output channels to output corresponding binary image feature maps;
[0104] Step 404: Obtain the pixel point set included in each cluster according to the binary image feature maps corresponding to all clusters.
[0105] By pre-setting a number of clustering result output channels equal to the number of clustering categories, it is ensured that different output channels are used in the subsequent process to output binary image feature maps corresponding to different clustering clusters, and image feature point processing is performed in combination with the binary image feature maps, thereby ensuring that corresponding data support is provided for the subsequent generation of class activation maps.
[0106] Continue to refer Figure 5 In some optional implementations, after step 404, a step of screening important pixels is also included. Figure 5 This is a flowchart of a specific embodiment of the important pixel point screening in the class activation map generation method described in the present application, including the following steps:
[0107] Step 501, determining the mode vector value of each cluster based on the vector values of the pixels included in all clusters;
[0108] Specifically, the image pixels clustered in each cluster after the aforementioned K-means clustering process are pixels within a certain RGB value range; on this basis, the pixels with different RGB values are counted, and the RGB value with the most corresponding pixels is determined through the mode vector value, and the pixel corresponding to the RGB value is determined as the important pixel.
[0109] Step 502, identifying all pixel points corresponding to the mode vector value of each cluster as important pixel points included in the corresponding cluster, and identifying pixel points corresponding to non-mode vector values as non-important pixel points included in the corresponding cluster;
[0110] Step 503, on the target binary image feature map, RGB value whitening processing is performed on the non-important pixels contained in each cluster;
[0111] Step 504: Obtain a binary image feature map corresponding to each cluster, retaining only important pixels.
[0112] In this embodiment, based on the above-mentioned K-means clustering process, an important pixel point screening is performed again, thereby further streamlining the number of pixel points included in each cluster.
[0113] Continue to refer Figure 6 , Figure 6 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes the following steps:
[0114] Step 601, based on the spatial relationship between the key pixels contained in each cluster, the cluster center areas corresponding to all clusters are screened out;
[0115] Step 602, taking the cluster center regions corresponding to all clusters as representative feature regions and marking them in the target input image, so as to obtain an image marked with the representative feature regions;
[0116] Step 603: input the image with the representative feature regions marked into the feature extraction model, perform feature extraction on the representative feature regions in the image, and obtain representative features corresponding to each cluster.
[0117] By screening out the cluster center regions corresponding to all cluster clusters, that is, the cluster center regions corresponding to the image classification targets, the cluster center regions corresponding to each image classification category are used as the representative feature regions of each image classification category. Then, feature extraction is performed on the representative feature regions in the image to obtain the representative features corresponding to each cluster cluster, that is, the representative features of each image classification category are obtained, so as to facilitate the subsequent generation of class activation maps according to the representative features of each image classification category.
[0118] Specifically, it goes through K-means clustering, important pixel screening and cluster center area determination, and then only obtains the features of the image in the cluster center area as the representative features. This ensures that there is no need to pay attention to non-important image feature areas when generating class activation maps later, reduces the computational processing amount when generating class activation maps, and at the same time, improves the accuracy and generation efficiency of class activation maps.
[0119] Continue to refer Figure 7 , Figure 7 yes Figure 6 The flowchart of a specific embodiment of step 601 shown includes the following steps:
[0120] Step 701, taking each key pixel in the current cluster as a target pixel in turn;
[0121] Specifically, first, different clusters are selected in turn as the current cluster. Then, in step 701, different target pixel points are selected in turn from the current cluster. After the cluster center area corresponding to one cluster is determined, the cluster center areas of other clusters are determined in the same way.
[0122] Step 702, within a circular area with the target pixel as the center point and an adjustable radius as the boundary, count the number of other key pixel points to obtain a number statistical value, wherein the adjustable radius value means that each round of statistics is performed, the radius value is dynamically adjusted, and then the statistics are re-performed;
[0123] Step 703, until there is a target pixel point corresponding to a statistical value that is greater than the statistical values of other key pixels, the position information of the target pixel point and the current radius value are recorded;
[0124] Specifically, a relatively small initial radius value is set first. Since the initial radius value is too small, the number statistics corresponding to multiple pixels may be repeated. Then, the radius value is increased successively until there is a pixel point whose number statistics are greater than the number statistics corresponding to other pixels. This indicates that the densest area of the cluster is determined.
[0125] Step 704: Taking the location information as the center point and the current radius value as the radius, a circular area is divided as the cluster center area of the current cluster.
[0126] Specifically, the most dense area of the cluster is essentially used as the cluster center area of the current cluster.
[0127] By taking the most dense area of each cluster as the cluster center area of the current cluster, it is more scientific. Moreover, since the radius value is in an adjustment state when determining the cluster center area of each cluster, the sizes of the cluster center areas corresponding to the different clusters finally obtained may also be different, ensuring that the selection of the cluster center area and the area size are non-mandatory settings and more flexible.
[0128] Continue to refer Figure 8 , Figure 8 yes Figure 6 The flowchart of a specific embodiment of step 602 shown includes the following steps:
[0129] Step 801, in the binary image feature map corresponding to each cluster, mark the corresponding representative feature area respectively;
[0130] Step 802, performing whole-image mapping synthesis on the binary image feature maps corresponding to all clusters and having corresponding representative feature regions marked, to obtain a mapping synthesis map marking the representative feature regions corresponding to all clusters;
[0131] Step 803 , marking all representative feature regions in the target input image according to the region position information of the representative feature regions corresponding to all clusters in the mapping synthesis image, and obtaining the image with the marked representative feature regions.
[0132] It should be understood that steps 801 to 803 implement the corresponding marking of the representative feature areas in the target input image. Thereafter, based on the image feature map in step 203, the image features corresponding to each representative feature area are obtained, and a class activation map is generated based on the image features corresponding to each representative feature area.
[0133] Specifically, image features are obtained from the corresponding area positions of the image feature map in step 203 through the representative feature areas marked in the target input image, and class activation maps are generated based on the image features, i.e., the image features corresponding to each representative feature area.
[0134] Specifically, when generating a class activation map, a Relu activation function is generally used for activation, which will not be described in detail here.
[0135] The present application obtains a target input image; inputs the target input image into a pre-trained feature extraction model to extract image features and obtain image feature extraction results; generates an image feature map based on the image feature extraction results; clusters all pixels in the image feature map based on the vector value of each pixel in the image feature map to obtain clustering results; obtains representative features corresponding to each cluster cluster according to the clustering results; generates a class activation map based on the image feature map according to the representative features corresponding to each cluster cluster. The class activation map generation method described in the present application first performs image feature clustering, and only realizes class activation map generation by focusing on the representative features corresponding to each cluster cluster. Compared with the existing class activation map generation technology, it is easy to focus on image areas that do not need to be focused on or are not important when generating class activation maps, resulting in the problem that the generated class activation map is not accurate enough. The present application does not need to focus on non-important image feature areas, reduces the amount of computational processing when generating class activation maps, and at the same time, improves the correctness and generation efficiency of class activation maps.
[0136] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0137] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0138] In the embodiment of the present application, by obtaining a target input image; inputting the target input image into a pre-trained feature extraction model, performing image feature extraction, and obtaining an image feature extraction result; generating an image feature map according to the image feature extraction result; clustering all pixels in the image feature map based on the vector value of each pixel in the image feature map to obtain a clustering result; obtaining representative features corresponding to each cluster cluster according to the clustering result; generating a class activation map based on the image feature map according to the representative features corresponding to each cluster cluster. The class activation map generation method described in the present application first performs image feature clustering, and only realizes the generation of class activation maps by focusing on the representative features corresponding to each cluster cluster. Compared with the existing class activation map generation technology, it is easy to focus on image areas that do not need to be focused on or are not important when generating class activation maps, resulting in the problem that the generated class activation maps are not accurate enough. In the present application, there is no need to focus on non-important image feature areas, which reduces the amount of computational processing when generating class activation maps, and at the same time, improves the correctness and generation efficiency of class activation maps.
[0139] Further references Fig. 9 , as a response to the above Figure 2 The present application provides an embodiment of a class activation map generation device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0140] like Fig. 9 As shown, the class activation map generation device 900 described in this embodiment includes: a target input image acquisition module 901, an image feature extraction module 902, an image feature map generation module 903, a pixel point clustering processing module 904, a representative feature acquisition module 905 and a class activation map generation module 906. Among them:
[0141] A target input image acquisition module 901 is used to acquire a target input image, wherein the target input image includes a medical image to be subjected to class activation map generation;
[0142] An image feature extraction module 902 is used to input the target input image into a pre-trained feature extraction model to perform image feature extraction and obtain an image feature extraction result, wherein the feature extraction model is a feature extraction model based on a CNN convolutional neural network;
[0143] An image feature map generating module 903 is used to generate an image feature map according to the image feature extraction result;
[0144] The pixel clustering processing module 904 is used to perform clustering processing on all the pixels in the image feature map based on the vector value of each pixel in the image feature map by using a preset clustering algorithm to obtain a clustering processing result;
[0145] The representative feature acquisition module 905 is used to obtain the representative features corresponding to each cluster according to the clustering processing result;
[0146] The class activation map generation module 906 is used to generate a class activation map based on the image feature map according to the representative features corresponding to each cluster.
[0147] The present application obtains a target input image; inputs the target input image into a pre-trained feature extraction model to extract image features and obtain image feature extraction results; generates an image feature map based on the image feature extraction results; clusters all pixels in the image feature map based on the vector value of each pixel in the image feature map to obtain clustering results; obtains representative features corresponding to each cluster cluster according to the clustering results; generates a class activation map based on the image feature map according to the representative features corresponding to each cluster cluster. The class activation map generation method described in the present application first performs image feature clustering, and only realizes class activation map generation by focusing on the representative features corresponding to each cluster cluster. Compared with the existing class activation map generation technology, it is easy to focus on image areas that do not need to be focused on or are not important when generating class activation maps, resulting in the problem that the generated class activation map is not accurate enough. The present application does not need to focus on non-important image feature areas, reduces the amount of computational processing when generating class activation maps, and at the same time, improves the correctness and generation efficiency of class activation maps.
[0148] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0149] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0150] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.
[0151] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c which are interconnected and communicated via a system bus. It should be noted that: Fig.10 Only a computer device 10 having a component memory 10a, a processor 10b, and a network interface 10c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.
[0152] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0153] The memory 10a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 10a can be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 10a can also be an external storage device of the computer device 10, such as a plug-in hard disk equipped on the computer device 10, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10a can also include both the internal storage unit of the computer device 10 and its external storage device. In this embodiment, the memory 10a is generally used to store the operating system and various application software installed on the computer device 10, such as a computer-readable instruction of a class activation map generation method, etc. In addition, the memory 10a can also be used to temporarily store various types of data that have been output or are to be output.
[0154] The processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run the computer-readable instructions stored in the memory 10a or process data, such as running the computer-readable instructions of the class activation map generation method.
[0155] The network interface 10c may include a wireless network interface or a wired network interface. The network interface 10c is generally used to establish a communication connection between the computer device 10 and other electronic devices.
[0156] The computer device proposed in this embodiment belongs to the field of research and development design and image processing technology, and is applied to the class activation map generation scenario when extracting features from medical images. This application obtains a target input image; inputs the target input image into a pre-trained feature extraction model to perform image feature extraction and obtain image feature extraction results; generates an image feature map based on the image feature extraction results; clusters all pixels in the image feature map based on the vector value of each pixel in the image feature map to obtain clustering results; obtains representative features corresponding to each clustering cluster based on the clustering results; generates a class activation map based on the image feature map based on the representative features corresponding to each clustering cluster. The class activation map generation method described in this application first performs image feature clustering, and only achieves class activation map generation by focusing on the representative features corresponding to each clustering cluster. Compared with the existing class activation map generation technology, it is easy to focus on unnecessary or unimportant image areas when generating class activation maps, resulting in the problem that the generated class activation maps are not accurate enough. The present application does not need to focus on non-important image feature areas, reduces the computational processing amount when generating class activation maps, and at the same time, improves the accuracy and generation efficiency of class activation maps.
[0157] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the class activation map generation method as described above.
[0158] The computer-readable storage medium proposed in this embodiment belongs to the field of research and development design and image processing technology, and is applied to the class activation map generation scenario when extracting features from medical images. This application obtains a target input image; inputs the target input image into a pre-trained feature extraction model to perform image feature extraction and obtain image feature extraction results; generates an image feature map based on the image feature extraction results; clusters all pixels in the image feature map based on the vector value of each pixel in the image feature map to obtain clustering results; obtains representative features corresponding to each clustering cluster based on the clustering results; generates a class activation map based on the image feature map based on the representative features corresponding to each clustering cluster. The class activation map generation method described in this application first performs image feature clustering, and only achieves class activation map generation by focusing on the representative features corresponding to each clustering cluster. Compared with the existing class activation map generation technology, it is easy to focus on unnecessary or unimportant image areas when generating class activation maps, resulting in the problem that the generated class activation maps are not accurate enough. The present application does not need to focus on non-important image feature areas, reduces the computational processing amount when generating class activation maps, and at the same time, improves the accuracy and generation efficiency of class activation maps.
[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0160] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A method for generating a class activation map, characterized in that: The steps include: Acquire a target input image, wherein the target input image includes a medical image for which class activation map generation is to be performed; Inputting the target input image into a pre-trained feature extraction model to perform image feature extraction to obtain an image feature extraction result, wherein the feature extraction model is a feature extraction model based on a CNN convolutional neural network; Generate an image feature map according to the image feature extraction result; Based on the vector value of each pixel point in the image feature map, a preset clustering algorithm is used to perform clustering processing on all the pixel points in the image feature map to obtain a clustering processing result; According to the clustering processing results, the representative features corresponding to each cluster are obtained; According to the representative features corresponding to each cluster, a class activation map is generated based on the image feature map.
2. The method for generating a class activation map according to claim 1, characterized in that: The step of clustering all the pixels in the image feature map using a preset clustering algorithm based on the vector value of each pixel in the image feature map to obtain a clustering result specifically includes: Identify a vector value of each pixel, wherein the vector value includes an RGB value; Using a K-means clustering algorithm, and clustering all pixels in the image feature map according to the vector value of each pixel, wherein the K value in the K-means clustering algorithm represents the number of cluster categories; After clustering is completed, the pixel point sets contained in each cluster are obtained.
3. The method for generating a class activation map according to claim 2, characterized in that: Before executing the step of obtaining the pixel point sets respectively included in each cluster after clustering is completed, the method further includes: Presetting a number of clustering result output channels equal to the number of clustering categories; The step of obtaining the pixel point set contained in each cluster after clustering is completed specifically includes: Identify the pixels contained in each cluster; Binarization is performed on the pixels included in each cluster and the pixels not included in each cluster to obtain the binary image feature map corresponding to each cluster; According to different clusters, different output channels are selected to output the corresponding binary image feature maps; According to the binary image feature maps corresponding to all clusters, the pixel point sets contained in each cluster are obtained.
4. The method for generating a class activation map according to claim 3, characterized in that: After executing the step of obtaining the pixel point set respectively included in each cluster according to the binary image feature maps respectively corresponding to all cluster clusters, the method further includes: Based on the vector values of the pixels contained in all clusters, the mode vector value of each cluster is determined; Identify all pixel points corresponding to the majority vector value of each cluster as important pixel points contained in the corresponding cluster, and identify the pixel points corresponding to the non-majority vector value as non-important pixel points contained in the corresponding cluster; On the target binary image feature map, perform RGB value whitening on the non-important pixels contained in each cluster. Obtain a binary image feature map corresponding to each cluster, retaining only important pixels.
5. The method for generating a class activation map according to claim 1, characterized in that: The step of obtaining representative features corresponding to each cluster according to the clustering processing result specifically includes: According to the spatial relationship between the key pixels contained in each cluster, the cluster center areas corresponding to all clusters are screened out; The cluster center regions corresponding to all clusters are taken as representative feature regions and marked in the target input image to obtain an image marked with representative feature regions; The image with the representative feature regions marked is input into the feature extraction model, and feature extraction is performed on the representative feature regions in the image to obtain representative features corresponding to each cluster.
6. The method for generating a class activation map according to claim 5, characterized in that: The step of screening out the cluster center areas corresponding to all clusters respectively according to the spatial relationship between the key pixels contained in each cluster specifically includes: Take each key pixel in the current cluster as the target pixel in turn; In a circular area with the target pixel as the center point and the adjustable radius as the boundary, the number of other key pixel points is counted to obtain a number statistical value, wherein the adjustable radius value means that the radius value is dynamically adjusted for each round of statistics, and then the statistics are re-performed; Until the number statistics corresponding to the target pixel point is greater than the number statistics corresponding to other key pixel points, the position information and the current radius value of the target pixel point are recorded; Taking the position information as the center point and the current radius value as the radius, a circular area is divided as the cluster center area of the current cluster.
7. The method for generating a class activation map according to claim 5 or 6, characterized in that: The step of marking the cluster center areas corresponding to all clusters as representative feature areas in the target input image to obtain an image marked with the representative feature areas specifically includes: In the binary image feature map corresponding to each cluster, the corresponding representative feature areas are marked respectively; Performing whole-image mapping synthesis on the binary image feature maps corresponding to all clusters and having corresponding representative feature regions marked, to obtain a mapping synthesis map marking the representative feature regions corresponding to all clusters; According to the regional position information of the representative feature regions corresponding to all clusters in the mapping synthesis image, all the representative feature regions are marked in the target input image to obtain the image with the marked representative feature regions.
8. A class activation map generation device, characterized in that: include: A target input image acquisition module, used to acquire a target input image, wherein the target input image includes a medical image to be subjected to class activation map generation; An image feature extraction module is used to input the target input image into a pre-trained feature extraction model to perform image feature extraction and obtain an image feature extraction result, wherein the feature extraction model is a feature extraction model based on a CNN convolutional neural network; An image feature map generation module, used to generate an image feature map according to the image feature extraction result; A pixel clustering processing module, used to perform clustering processing on all pixels in the image feature map based on the vector value of each pixel in the image feature map using a preset clustering algorithm to obtain a clustering processing result; A representative feature acquisition module is used to obtain representative features corresponding to each cluster according to the clustering processing results; The class activation map generation module is used to generate a class activation map based on the image feature map according to the representative features corresponding to each cluster.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the class activation map generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the class activation map generation method according to any one of claims 1 to 7 are implemented.