Image recognition method based on class activation map algorithm and related device

Abstract

Embodiments of the present application relate to the technical field of artificial intelligence, and disclose an image recognition method based on a class activation map algorithm and related devices, the method comprising: obtaining an image to be recognized; inputting the image to be recognized into a class activation map algorithm based on multi-label gradient feedback, the class activation map algorithm based on multi-label gradient feedback performing feature convolution on the image to be recognized to obtain a multi-scale feature map, performing gradient backpropagation on feature maps of each label category when performing gradient backpropagation on a current category, determining an inter-category influence factor according to a gradient value set backpropagated by all label categories, determining an activation map weight of the current category according to the inter-category influence factor and an influence factor of different pixel positions of the current category, and obtaining a recognition result according to the activation map weight and the multi-scale feature map; wherein the current category is any category in a plurality of label categories; and outputting the recognition result. Embodiments of the present application improve the generation quality of a saliency map.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of artificial intelligence, in particular to an image recognition method and device based on a class activation map algorithm, computer equipment and a computer readable storage medium. BACKGROUND

[0002] At present, deep convolutional neural networks (DCNN) have shown excellent performance in various visual tasks, such as target detection, semantic segmentation, instance segmentation, etc. They can learn from a large amount of data and get satisfactory results. Although they have made significant progress in solving major visual tasks, they are still like black box structures, which makes it a difficult task to explain these models. If the decision-making process of the model cannot be understood and explained, it is difficult to safely deploy these models in sensitive industries such as medical and safety, because any wrong prediction and decision may cause the accuracy and precision of the deep network model to deviate significantly.

[0003] In order to explain the behavior of the model and improve its interpretability, researchers have proposed various methods. Early perturbation-based explainable methods monitor the impact of changes in different positions of the input image on the model output by perturbing the pixels at different positions of the input image, so as to evaluate the importance of the pixels. For example, some studies use black blocks to block different positions of the image to obtain a saliency map. However, this kind of method cannot show the reasoning process of the intermediate layer of the model, and the calculation cost is relatively high when blocking position by position. Under this background, by visualizing the saliency map of the intermediate layer of the convolutional neural network, the contribution degree of different positions in the feature map to a particular class, and other CAM algorithms (such as Grad-CAM, Grad-CAM++, Score-CAM, Zoom-CAM and Poly-CAM, etc.) proposed by subsequent researchers can explain the model decision-making process to some extent.

[0004] However, in multi-label tasks, current CAM algorithms only obtain channel weights based on the gradient feedback of a single class score. Therefore, when the input image contains multiple different categories of objects, the existing CAM algorithms do not fully consider the mutual influence of similar features between different categories of objects. To address this issue, this paper proposes a novel CAM algorithm—a Class Activation Map Method Based on Multilabel Gradient Feedback (Mgrad-CAM). This algorithm performs gradient backpropagation on scores from multiple different classes. When calculating channel weights, it integrates the influence factors between different classes, allowing Mgrad-CAM to retain the core feature regions belonging to the target class while reducing the interference of similar feature regions from other classes. Furthermore, we propose a recursive fusion method of intermediate layer saliency maps to address the problem of coarseness when using only a single layer's saliency map. This method employs a top-down recursive fusion approach, dynamically adjusting the contribution of features from each layer. This allows for a better combination of high-level semantic information and mid-level fine-grained features, more effectively capturing key features in complex scenes and avoiding the limitations of fixed weights. This dynamic fusion method is more flexible and can cope with the interference caused by the similarity of features of different objects in multi-label tasks, generating more delicate and high-resolution saliency maps. Summary of the Invention

[0005] In view of the above problems, embodiments of the present invention provide an image recognition method, apparatus, computer device, and computer-readable storage medium based on a class activation graph algorithm, which is used to solve the problem of low quality of saliency graphs caused by insufficient interactive interpretation of features and problems such as noise and gradient vanishing in the prior art.

[0006] According to one aspect of the present invention, an image recognition method based on a class activation graph algorithm is provided, the method comprising:

[0007] Acquire the image to be recognized;

[0008] The image to be identified is input into a class activation map algorithm based on multi-label gradient feedback. This algorithm performs feature convolution on the image to be identified to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category. Based on the set of gradient values ​​backpropagated from all label categories, an inter-category influence factor is determined. Based on the inter-category influence factor and the influence factor at different pixel positions of the current category, the activation map weight of the current category is determined. Finally, based on the activation map weight and the multi-scale feature map, the recognition result is obtained. Here, the current category is any one of multiple label categories; the multiple label categories are the label categories present in the image to be identified.

[0009] Output the recognition results.

[0010] In one optional approach, the class activation map algorithm based on multi-label gradient feedback performs feature convolution on the image to be identified to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category, and an inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. Based on the inter-category influence factor and the influence factor at different pixel positions of the current category, the activation map weight of the current category is determined, and the recognition result is obtained based on the activation map weight and the multi-scale feature map. This further includes:

[0011] The image to be identified is subjected to feature convolution to obtain a multi-scale feature map;

[0012] For each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category at the scale, and the inter-category influence factor corresponding to the current category is determined based on the set of gradient values ​​backpropagated for all categories. The activation map weight of the current category is determined based on the inter-category influence factor and the category influence factor of the current category.

[0013] Based on the activation map weights and the feature maps at the corresponding scales, the current saliency map at each scale corresponding to the current category is obtained;

[0014] Recursively fuse the current saliency maps at each scale corresponding to the current category to obtain the fused saliency map of the intermediate fusion layer corresponding to the current category;

[0015] The fusion saliency map corresponding to the current category is output to obtain the recognition result of the image to be recognized under the current category.

[0016] In an optional approach, for each feature map at each scale, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps at the scale for each label category, and the inter-category influence factor corresponding to the current category is determined based on the set of gradient values ​​backpropagated for all label categories. The activation map weight of the current category is then determined based on the inter-category influence factor and the category influence factor of the current category. This further includes:

[0017] For the feature map at each scale, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature map at the scale for each label category to obtain the backpropagation gradient set for all category labels at each scale.

[0018] The backpropagation gradient set for all label categories is standardized along the label category dimension to obtain the standardized gradient values ​​for all label categories.

[0019] For the current category, the inter-category influence factor is determined based on the standardized gradient values ​​corresponding to all label categories.

[0020] Calculate the activation graph weight corresponding to the current category based on the inter-category influence factors and the category influence factor of the current category.

[0021] In one alternative approach, the backpropagation gradient set for all label categories is standardized along the label category dimension to obtain standardized gradient values ​​corresponding to all label categories, including:

[0022] The standardized gradient values ​​for all label categories are calculated using the following standardized formula:

[0023] ;

[0024] in, The predicted score for category label c Activation graph The gradient backpropagation value is the backpropagation gradient value of the kth channel of the feature map corresponding to the label category and the lth layer of the network. For the current category C, this is the set of backpropagation gradients of the k-th channel of the feature map corresponding to each label category and the l-th layer of the network.

[0025] For the current category, the inter-category influence factor is determined based on the standardized gradient values ​​corresponding to all label categories, including:

[0026] The current category is calculated using the following formula. Inter-category influence factors :

[0027] ;

[0028] The step of calculating the activation graph weight corresponding to the current category based on the inter-category influence factors and the category influence factor of the current category includes:

[0029] The current category is calculated using the following formula. Corresponding activation graph weights :

[0030] , ,

[0031] in, It represents the Hadamardi (or Hadama) stack; For the current category The category influence factor in the k-th channel network layer l of the feature map; This indicates the activation pixel position (i,j) in the graph;

[0032] The step of obtaining the current saliency map for each scale corresponding to the current category based on the activation map weights and the feature maps at the corresponding scales includes:

[0033] The current category is calculated using the following formula. The current saliency map of the k-th channel network at layer l in the feature map. :

[0034] In one alternative approach, the recursive fusion of the original activation maps at each scale corresponding to the current category to obtain the fusion saliency map of the intermediate fusion layer corresponding to the current category includes:

[0035] Use bilinear interpolation to upsample the current saliency map of the higher levels of the current category to the size of the current saliency map of the lower levels;

[0036] The current saliency map of the upsampled high layer is fused with the current saliency map of the low layer to obtain the current fused saliency map;

[0037] The iterative process involves using bilinear interpolation to upsample the current saliency map of the higher-level layer of the current category to the size of the current saliency map of the lower-level layer, and then fusing the upsampled current saliency map of the higher-level layer with the current saliency map of the lower-level layer to obtain the current fused saliency map. This process continues until all current saliency maps are fused, recursively obtaining the final fused saliency map.

[0038] In one alternative approach, fusing the upsampled current saliency map of the higher layer with the current saliency map of the lower layer to obtain a current fused saliency map includes:

[0039] For the current category The The current saliency graph of the layer network and the first layer network The current saliency maps of the layer networks are fused using the following formula to obtain:

[0040] :

[0041] in, For the current category corresponding The current saliency graph of the layer; For the current category corresponding The current saliency graph of the layer; The network layer is higher than the layer. layer.

[0042] According to another aspect of the present invention, an image recognition device based on a class activation graph algorithm is provided, comprising:

[0043] The acquisition module is used to acquire the image to be recognized;

[0044] The class activation map algorithm module is used to input the image to be recognized into a class activation map algorithm based on multi-label gradient feedback. The class activation map algorithm based on multi-label gradient feedback performs feature convolution on the image to be recognized to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category, and the inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. Based on the inter-category influence factor and the influence factor of different pixel positions of the current category, the activation map weight of the current category is determined, and the recognition result is obtained based on the activation map weight and the multi-scale feature map. Here, the current category is any one of multiple label categories; the multiple label categories are the label categories present in the image to be recognized.

[0045] The output module is used to output the recognition results.

[0046] In an alternative embodiment, the class activation graph algorithm module further includes:

[0047] The input module is used to receive the image to be identified;

[0048] Convolutional layers are used to perform feature convolution on the image to be identified, resulting in multi-scale feature maps;

[0049] The multi-label gradient feedback module is used to perform gradient backpropagation on the feature maps of each label category when performing gradient backpropagation on the current category, and determine the inter-category influence factor based on the set of gradient values ​​backpropagated from all label categories. Based on the inter-category influence factor and the influence factor of different pixel positions of the current category, the activation map weight of the current category is determined, and the current saliency map at each scale corresponding to the current category is obtained based on the activation map weight and the multi-scale feature map. The multiple label categories are the label categories that exist in the image to be identified.

[0050] The recursive fusion intermediate layer saliency map module is used to recursively fuse the current saliency map at each scale corresponding to the current category to obtain the fused saliency map of the fusion intermediate layer corresponding to the current category.

[0051] The model output module is used to output the recognition result of the image to be recognized under the current category based on the fusion saliency map corresponding to the current category.

[0052] According to another aspect of the present invention, a computer device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0053] The memory is used to store at least one executable instruction that causes the processor to perform the operation of the image recognition method based on the class activation graph algorithm.

[0054] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the operation of the image recognition method based on the class activation graph algorithm.

[0055] This invention provides an embodiment of an image to be identified. The image is then input into a class activation map algorithm based on multi-label gradient feedback. This algorithm performs feature convolution on the image to obtain a multi-scale feature map. During gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category. An inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. The activation map weight for the current category is determined based on the inter-category influence factor and the influence factor at different pixel positions within the current category. Finally, the identification result is obtained based on the activation map weight and the multi-scale feature map. The current category can be any of multiple label categories. Outputting the identification result reduces interference from similar feature regions across multiple label categories.

[0056] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0057] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0058] Figure 1 A flowchart illustrating the image recognition method based on the class activation graph algorithm provided in an embodiment of the present invention is shown.

[0059] Figure 2 This invention provides a schematic diagram of the structure of an image recognition method based on a class activation graph algorithm according to another embodiment of the present invention.

[0060] Figure 3 This diagram illustrates the structure of an image recognition device based on a class activation graph algorithm provided in an embodiment of the present invention.

[0061] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0062] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0063] Below, we will first explain the terms involved:

[0064] Interpretive algorithms based on input gradients: Methods based on input gradients were an early and relatively direct class of interpretability algorithms. They measure feature importance by calculating the impact of input perturbations on the model output.

[0065] Layerwise Relevance Propagation (LRP) and its variants: LRP generates a saliency map by recursively calculating the relevance scores of each neuron in a layer to explain the contribution of input features to the output.

[0066] Interpretive algorithms based on activation graphs: Interpretive methods based on activation graphs reveal attention regions by visualizing intermediate or final layers of convolutional neural networks.

[0067] Figure 1A flowchart of an image recognition method based on a class activation graph algorithm provided in an embodiment of the present invention is shown. This method is executed by a computer device. The computer device can be a laptop computer, desktop computer, smart terminal, distributed device, wearable device, etc., and the embodiments of the present invention do not impose specific limitations. Figure 1 As shown, the method includes the following steps:

[0068] Step 110: Obtain the image to be recognized.

[0069] The image to be identified can be an image that may contain different categories of objects, such as human bodies, tables and chairs.

[0070] Step 120: Input the image to be recognized into a class activation map algorithm based on multi-label gradient feedback. The class activation map algorithm based on multi-label gradient feedback performs feature convolution on the image to be recognized to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category. The inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. The activation map weight of the current category is determined based on the inter-category influence factor and the influence factor of different pixel positions of the current category. The recognition result is obtained based on the activation map weight and the multi-scale feature map. The current category is any one of multiple label categories, and the multiple label categories are the label categories that exist in the image to be recognized.

[0071] Among them, such as Figure 2As shown, the class activation map algorithm module includes an input module, a convolutional layer, a multi-label gradient feedback module, a recursive fusion intermediate layer saliency map module, and a model output module. The input module receives the image to be recognized. The convolutional layer performs feature convolution on the image to be recognized to obtain multi-scale feature maps. The multi-label gradient feedback module performs gradient backpropagation on the feature maps of each label category when performing gradient backpropagation for the current category, determines the inter-category influence factor based on the set of gradient values ​​backpropagated from all label categories, determines the activation map weight of the current category based on the inter-category influence factor and the influence factor of different pixel positions of the current category, and obtains the current saliency map at each scale corresponding to the current category based on the activation map weight and the multi-scale feature maps. The recursive fusion intermediate layer saliency map module recursively fuses the current saliency maps at each scale corresponding to the current category to obtain the fused saliency map of the fusion intermediate layer corresponding to the current category. The model output module outputs the recognition result of the image to be recognized in the current category based on the fused saliency map corresponding to the current category. This algorithm takes a source image to be identified as input and generates a saliency map of the target category on the source image. First, the image to be identified is convolved to obtain a series of feature maps. Then, these feature maps undergo fully connected operations and standardization. During processing, gradient backpropagation is performed on multiple different categories to obtain the channel weights of the final feature map. Finally, a recursive fusion intermediate layer module is used to recursively fuse the saliency maps obtained from each network layer to output a higher-precision saliency map.

[0072] Specifically, it includes the following steps:

[0073] Step 1201: Perform feature convolution on the image to be identified to obtain a multi-scale feature map.

[0074] The image to be identified is convolved to obtain a multi-scale feature map. This multi-scale feature map is a series of feature maps at multiple scales.

[0075] Step 1202: For the feature map at each scale, when performing gradient backpropagation for the current category, perform gradient backpropagation for the feature map at the scale of each label category, and determine the inter-category influence factor corresponding to the current category based on the set of gradient values ​​backpropagated for all categories. Based on the inter-category influence factor and the category influence factor of the current category, determine the activation map weight of the current category.

[0076] In existing gradient-based CAM methods, gradient backpropagation is performed only for a specific category each time. The weights of the activation map corresponding to that category are then calculated based on the gradient information to obtain the final saliency map. This approach only considers the relative relationships of a specific category across different pixels in the activation map, neglecting the interactions between multiple categories of objects that may exist in the input image. For example, in an input image containing multiple categories of objects, assuming a selected label category... Its score Activation graph At pixel position ( i , j The gradient of a pixel is relatively large, and in previous gradient-based CAM methods, the pixel position ( i , j ) for label categories The contribution is relatively large, thus the saliency map is at pixel position ( i , j The values ​​are large at pixel positions i and j, but if pixel positions i and j have different values ​​for all other pixel categories, the values ​​are large. The gradients for most other categories are related to the label category. The gradients are similar, indicating that this pixel has a common feature and does not have a good class differentiation function. Therefore, the pixel position cannot be considered as ( i , j ) is a tag category The core attribution is insufficient. Therefore, in gradient backpropagation-based CAM methods, gradient backpropagation for a single class cannot obtain a complete and fair attribution of the model for that class, and the resulting saliency map is also inadequate. To address this issue and generate a saliency map that better reflects the core feature regions of the target class,

[0077] This invention proposes a multi-label gradient feedback module, Mgrad. Specifically, for each scale of the feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category at that scale, resulting in a set of backpropagated gradients for all category labels at each scale.

[0078] However, performing gradient backpropagation on all possible label categories would be computationally intensive and inconsistent with the model's internal inference process. Therefore, Mgrad chooses to backpropagate only the actual label categories present in the input image to be recognized. Assume the set of label categories present in the input image to be recognized is... , use Indicates category prediction score The gradient backpropagation results They have the same size. Then Mgrad applies to the set of label categories. backpropagation gradient set :

[0079] ;

[0080] in, l For the network's first l layer, k The first feature map represents the first feature map. k One channel; Indicates category label Predicted score Activation graph The gradient backpropagation value is the first value of the feature map corresponding to the label category. k The first channel, network l The backpropagation gradient value of the layer For each tag category The corresponding feature map of the 1st k The first channel, network l The set of backpropagation gradients for the layer.

[0081] Backpropagation gradient set for all label categories Standardize along the label category dimension to obtain the standardized gradient values ​​for all label categories. Specifically, the standardized gradient values ​​for all label categories are calculated using the following standardized formula. :

[0082] ;

[0083] For the current category, based on all tag categories The corresponding standardized gradient value is used to determine the inter-category influence factor for the current category. Specifically, the current category is calculated using the following formula. Inter-category influence factors :

[0084] ;

[0085] in, Current category Influence factors at different pixel locations Defined as:

[0086] ;

[0087] in, The dimensions are consistent.

[0088] Calculate the activation graph weight corresponding to the current category based on the inter-category influence factors and the current category's category influence factor. The current category is calculated using the following formula. Corresponding activation graph weights :

[0089] ,

[0090] ,

[0091] in, It represents the Hadamardi (or Hadama) stack; For the current category In the feature map k The first channel network l Layer category influence factor; This indicates the pixel position (i,j) in the active image.

[0092] Step 1203: Based on the activation map weights and the feature maps at the corresponding scales, obtain the current saliency map at each scale corresponding to the current category.

[0093] The current category is calculated using the following formula. In the feature map, the k-th channel network... l Current saliency graph of the layer :

[0094] .

[0095] When the input image contains only a single type of object. In this embodiment of the invention, Mgrad-CAM degenerates into the existing Grad-CAM.

[0096] By following the steps above, the current saliency map of the feature map at each scale corresponding to the current category can be calculated.

[0097] Step 1204: Recursively fuse the current saliency maps at each scale corresponding to the current category to obtain the fused saliency map of the fused intermediate layer corresponding to the current category.

[0098] For CNN models, higher layers often express semantic information while lower layers express fine-grained information. Therefore, to obtain a more accurate saliency map, it is necessary to fuse information from the intermediate layers of the model. However, gradients often contain significant noise when they reach the intermediate layers. The challenge of the Mgrad-CAM algorithm lies in how to eliminate the influence of intermediate layer gradient noise while preserving the fine-grained information of the intermediate layers. This invention proposes that eliminating intermediate layer gradient noise must rely on the semantic information of the higher layers of the model. That is, this invention focuses more on the detailed information in the salient regions of the saliency map obtained from the higher layers than in other regions. Therefore, this invention designs a recursive algorithm to continuously fuse the saliency maps of the intermediate layers.

[0099] Specifically, the fusion saliency map of the fusion intermediate layer corresponding to the current category is calculated using the following method:

[0100] Use bilinear interpolation to upsample the current saliency map of the higher levels of the current category to the size of the current saliency map of the lower levels;

[0101] The upsampled current saliency map of the higher level is fused with the current saliency map of the lower level to obtain the current fused saliency map. Specifically, for the current category... The The current saliency graph of the layer network and the first layer network The current saliency maps of the layer networks are fused using the following formula to obtain:

[0102] ;

[0103] in, For the current category corresponding The current saliency graph of the layer; For the current category corresponding The current saliency graph of the layer; The network layer is higher than the layer. layer; These represent the normalization function and the upsampling function, respectively.

[0104] The iterative process involves using bilinear interpolation to upsample the current saliency map of the higher-level layers of the current category to the size of the current saliency map of the lower-level layers, and then fusing the upsampled higher-level current saliency map with the lower-level current saliency map to obtain the current fused saliency map. This process continues until all current saliency maps have been fused, recursively yielding the final fused saliency map. Specifically, the recursive fusion process continues based on the aforementioned fusion formula. Layers and more network layers.

[0105] Step 1205: Output the fusion saliency map corresponding to the current category to obtain the recognition result of the image to be recognized in the current category.

[0106] The final saliency map obtained through multi-layer recursive fusion integrates saliency maps from different levels of the model, preserving high-resolution details while ensuring the accuracy of high-level linguistic information. This method ensures that, during saliency map generation, the influence of intermediate layer gradient noise is effectively eliminated while extracting fine-grained features from different levels, thus improving the accuracy of the saliency map. Compared to Feature Pyramid Networks (FPN), this embodiment focuses more on saliency map generation, emphasizing a layer-by-layer recursive approach to progressively enhance the detailed features of mid- and low-level layers while preserving high-level semantic information, thereby improving the overall accuracy of the saliency map. This method is particularly suitable for scenarios requiring fine-grained interpretation and denoising, enabling the saliency map to more accurately reflect the core regions of the target category.

[0107] Step 130: Output the recognition result.

[0108] After the aforementioned identification steps, this embodiment of the invention generates a saliency map of the label categories present in the image to be identified. This saliency map includes the targets marked in the image and their corresponding label categories.

[0109] This invention provides an embodiment of an image to be identified. The image is then input into a class activation map algorithm based on multi-label gradient feedback. This algorithm performs feature convolution on the image to obtain a multi-scale feature map. During gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category. An inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. The activation map weight for the current category is determined based on the inter-category influence factor and the influence factor at different pixel positions within the current category. Finally, the identification result is obtained based on the activation map weight and the multi-scale feature map. The current category can be any of multiple label categories. Outputting the identification result reduces interference from similar feature regions across multiple label categories.

[0110] Figure 3 A schematic diagram of the structure of an image recognition device based on a class activation graph algorithm provided in an embodiment of the present invention is shown. Figure 3 As shown, the device 300 includes:

[0111] The acquisition module 310 is used to acquire the image to be recognized;

[0112] The class activation map algorithm module 320 is used to input the image to be recognized into a class activation map algorithm based on multi-label gradient feedback. The class activation map algorithm based on multi-label gradient feedback performs feature convolution on the image to be recognized to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category, and the inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. Based on the inter-category influence factor and the influence factor of different pixel positions of the current category, the activation map weight of the current category is determined, and the recognition result is obtained based on the activation map weight and the multi-scale feature map. Wherein, the current category is any one of multiple label categories.

[0113] Output module 330 is used to output the recognition result.

[0114] like Figure 2 As shown, the class activation graph algorithm module further includes:

[0115] The input module is used to receive the image to be identified;

[0116] Convolutional layers are used to perform feature convolution on the image to be identified, resulting in multi-scale feature maps;

[0117] The multi-label gradient feedback module is used to perform gradient backpropagation on the feature maps of each label category when performing gradient backpropagation on the current category, and determine the inter-category influence factor based on the set of gradient values ​​backpropagated by all label categories. Based on the inter-category influence factor and the influence factor of different pixel positions of the current category, the activation map weight of the current category is determined, and the current saliency map at each scale corresponding to the current category is obtained based on the activation map weight and the multi-scale feature map.

[0118] The recursive fusion intermediate layer saliency map module is used to recursively fuse the current saliency map at each scale corresponding to the current category to obtain the fused saliency map of the fusion intermediate layer corresponding to the current category.

[0119] The model output module is used to output the recognition result of the image to be recognized under the current category based on the fusion saliency map corresponding to the current category.

[0120] In one optional approach, the class activation map algorithm based on multi-label gradient feedback performs feature convolution on the image to be identified to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category, and an inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. Based on the inter-category influence factor and the influence factor at different pixel positions of the current category, the activation map weight of the current category is determined, and the recognition result is obtained based on the activation map weight and the multi-scale feature map. This further includes:

[0121] The image to be identified is subjected to feature convolution to obtain a multi-scale feature map;

[0122] For each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category at the scale, and the inter-category influence factor corresponding to the current category is determined based on the set of gradient values ​​backpropagated for all categories. The activation map weight of the current category is determined based on the inter-category influence factor and the category influence factor of the current category.

[0123] Based on the activation map weights and the feature maps at the corresponding scales, the current saliency map at each scale corresponding to the current category is obtained;

[0124] Recursively fuse the current saliency maps at each scale corresponding to the current category to obtain the fused saliency map of the intermediate fusion layer corresponding to the current category;

[0125] The fusion saliency map corresponding to the current category is output to obtain the recognition result of the image to be recognized under the current category.

[0126] In an optional approach, for each feature map at each scale, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps at the scale for each label category, and the inter-category influence factor corresponding to the current category is determined based on the set of gradient values ​​backpropagated for all label categories. The activation map weight of the current category is then determined based on the inter-category influence factor and the category influence factor of the current category. This further includes:

[0127] For each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature map of each label category at the scale to obtain the backpropagation gradient set of all category labels at each scale.

[0128] The backpropagation gradient set for all label categories is standardized along the label category dimension to obtain the standardized gradient values ​​for all label categories.

[0129] For the current category, the inter-category influence factor is determined based on the standardized gradient values ​​corresponding to all label categories.

[0130] Calculate the activation graph weight corresponding to the current category based on the inter-category influence factors and the category influence factor of the current category.

[0131] In one alternative approach, the backpropagation gradient set for all label categories is standardized along the label category dimension to obtain standardized gradient values ​​corresponding to all label categories, including:

[0132] The standardized gradient values ​​for all label categories are calculated using the following standardized formula:

[0133] ;

[0134] in, Indicates category label Predicted score Activation graph The gradient backpropagation value is the backpropagation gradient value of the kth channel of the feature map corresponding to the label category and the lth layer of the network. For the current category The set of backpropagation gradients of the k-th channel of the feature map corresponding to each label category and the l-th layer of the network;

[0135] For the current category, the inter-category influence factor is determined based on the standardized gradient values ​​corresponding to all label categories, including:

[0136] The current category is calculated using the following formula. Inter-category influence factors :

[0137] ;

[0138] The step of calculating the activation graph weight corresponding to the current category based on the inter-category influence factors and the category influence factor of the current category includes:

[0139] The current category is calculated using the following formula. Corresponding activation graph weights :

[0140] , ,

[0141] in, It represents the Hadamardi (or Hadama) stack; For the current category The category influence factor in the k-th channel network layer l of the feature map; This indicates the activation pixel position (i,j) in the graph;

[0142] The step of obtaining the current saliency map for each scale corresponding to the current category based on the activation map weights and the feature maps at the corresponding scales includes:

[0143] The current category is calculated using the following formula. The current saliency map of the k-th channel network at layer l in the feature map. :

[0144] .

[0145] In one alternative approach, the recursive fusion of the original activation maps at each scale corresponding to the current category to obtain the fusion saliency map of the intermediate fusion layer corresponding to the current category includes:

[0146] Use bilinear interpolation to upsample the current saliency map of the higher levels of the current category to the size of the current saliency map of the lower levels;

[0147] The current saliency map of the upsampled high layer is fused with the current saliency map of the low layer to obtain the current fused saliency map;

[0148] The iterative process involves using bilinear interpolation to upsample the current saliency map of the higher-level layer of the current category to the size of the current saliency map of the lower-level layer, and then fusing the upsampled current saliency map of the higher-level layer with the current saliency map of the lower-level layer to obtain the current fused saliency map. This process continues until all current saliency maps are fused, recursively obtaining the final fused saliency map.

[0149] In one alternative approach, fusing the upsampled current saliency map of the higher layer with the current saliency map of the lower layer to obtain a current fused saliency map includes:

[0150] For the current category The The current saliency graph of the layer network and the first layer network The current saliency maps of the layer networks are fused using the following formula to obtain:

[0151] ;

[0152] in, For the current category corresponding The current saliency graph of the layer; For the current category corresponding The current saliency graph of the layer; The network layer is higher than the layer. layer.

[0153] The specific working process of the image recognition device based on the class activation graph algorithm in this embodiment of the invention is largely the same as the specific implementation steps of the above method embodiment, and will not be repeated here.

[0154] This invention provides an embodiment of an image to be identified. The image is then input into a class activation map algorithm based on multi-label gradient feedback. This algorithm performs feature convolution on the image to obtain a multi-scale feature map. During gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category. An inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. The activation map weight for the current category is determined based on the inter-category influence factor and the influence factor at different pixel positions within the current category. Finally, the identification result is obtained based on the activation map weight and the multi-scale feature map. The current category can be any of multiple label categories. Outputting the identification result reduces interference from similar feature regions across multiple label categories.

[0155] Figure 4 The diagram shows a structural schematic of a computer device provided in an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computer device.

[0156] like Figure 4 As shown, the computer device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0157] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other network elements such as clients or other servers. The processor 402 executes program 410, specifically performing the relevant steps described above in the embodiment of the image recognition method based on the class activation graph algorithm.

[0158] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0159] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0160] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0161] Specifically, program 410 can be called by processor 402 to cause the computer device to perform the following operations:

[0162] Acquire the image to be recognized;

[0163] The image to be identified is input into a class activation map algorithm based on multi-label gradient feedback. The class activation map algorithm performs feature convolution on the image to be identified to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category, and the inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. Based on the inter-category influence factor and the influence factor of different pixel positions of the current category, the activation map weight of the current category is determined, and the recognition result is obtained based on the activation map weight and the multi-scale feature map. Herein, the current category is any one of multiple label categories.

[0164] Output the recognition results.

[0165] In one optional approach, the class activation map algorithm based on multi-label gradient feedback performs feature convolution on the image to be identified to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category, and an inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. Based on the inter-category influence factor and the influence factor at different pixel positions of the current category, the activation map weight of the current category is determined, and the recognition result is obtained based on the activation map weight and the multi-scale feature map. This further includes:

[0166] The image to be identified is subjected to feature convolution to obtain a multi-scale feature map;

[0167] For each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category at the scale, and the inter-category influence factor corresponding to the current category is determined based on the set of gradient values ​​backpropagated for all categories. The activation map weight of the current category is determined based on the inter-category influence factor and the category influence factor of the current category.

[0168] Based on the activation map weights and the feature maps at the corresponding scales, the current saliency map at each scale corresponding to the current category is obtained;

[0169] Recursively fuse the current saliency maps at each scale corresponding to the current category to obtain the fused saliency map of the intermediate fusion layer corresponding to the current category;

[0170] The fusion saliency map corresponding to the current category is output to obtain the recognition result of the image to be recognized under the current category.

[0171] In an optional approach, for each feature map at each scale, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps at the scale for each label category, and the inter-category influence factor corresponding to the current category is determined based on the set of gradient values ​​backpropagated for all label categories. The activation map weight of the current category is then determined based on the inter-category influence factor and the category influence factor of the current category. This further includes:

[0172] For each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature map of each label category at the scale to obtain the backpropagation gradient set of all category labels at each scale.

[0173] The backpropagation gradient set for all label categories is standardized along the label category dimension to obtain the standardized gradient values ​​for all label categories.

[0174] For the current category, the inter-category influence factor is determined based on the standardized gradient values ​​corresponding to all label categories.

[0175] Calculate the activation graph weight corresponding to the current category based on the inter-category influence factors and the category influence factor of the current category.

[0176] In one alternative approach, the backpropagation gradient set for all label categories is standardized along the label category dimension to obtain standardized gradient values ​​corresponding to all label categories, including:

[0177] The standardized gradient values ​​for all label categories are calculated using the following standardized formula:

[0178] ;

[0179] in, Indicates category label Predicted score Activation graph The gradient backpropagation value is the backpropagation gradient value of the kth channel of the feature map corresponding to the label category and the lth layer of the network. For the current category The set of backpropagation gradients of the kth channel of the feature map corresponding to each label category and the lth layer of the network.

[0180] For the current category, the inter-category influence factor is determined based on the standardized gradient values ​​corresponding to all label categories, including:

[0181] The current category is calculated using the following formula. Inter-category influence factors :

[0182] ;

[0183] The step of calculating the activation graph weight corresponding to the current category based on the inter-category influence factors and the category influence factor of the current category includes:

[0184] The current category is calculated using the following formula. Corresponding activation graph weights :

[0185] ,

[0186] ,

[0187] in, It represents the Hadamardi (or Hadama) stack; For the current category The category influence factor in the k-th channel network layer l of the feature map; This indicates the activation pixel position (i,j) in the graph;

[0188] The step of obtaining the current saliency map for each scale corresponding to the current category based on the activation map weights and the feature maps at the corresponding scales includes:

[0189] The current category is calculated using the following formula. The current saliency map of the k-th channel network at layer l in the feature map. ;

[0190] .

[0191] In one alternative approach, the recursive fusion of the original activation maps at each scale corresponding to the current category to obtain the fusion saliency map of the intermediate fusion layer corresponding to the current category includes:

[0192] Use bilinear interpolation to upsample the current saliency map of the higher levels of the current category to the size of the current saliency map of the lower levels;

[0193] The current saliency map of the upsampled high layer is fused with the current saliency map of the low layer to obtain the current fused saliency map;

[0194] The iterative process involves using bilinear interpolation to upsample the current saliency map of the higher-level layer of the current category to the size of the current saliency map of the lower-level layer, and then fusing the upsampled current saliency map of the higher-level layer with the current saliency map of the lower-level layer to obtain the current fused saliency map. This process continues until all current saliency maps are fused, recursively obtaining the final fused saliency map.

[0195] In one alternative approach, fusing the upsampled current saliency map of the higher layer with the current saliency map of the lower layer to obtain a current fused saliency map includes:

[0196] For the current category The The current saliency graph of the layer network and the first layer network The current saliency maps of the layer networks are fused using the following formula to obtain:

[0197] ;

[0198] in, For the current category corresponding The current saliency graph of the layer; For the current category corresponding The current saliency graph of the layer; The network layer is higher than the layer. layer.

[0199] This invention provides an embodiment of an image to be identified. The image is then input into a class activation map algorithm based on multi-label gradient feedback. This algorithm performs feature convolution on the image to obtain a multi-scale feature map. During gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category. An inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. The activation map weight for the current category is determined based on the inter-category influence factor and the influence factor at different pixel positions within the current category. Finally, the identification result is obtained based on the activation map weight and the multi-scale feature map. The current category can be any of multiple label categories. Outputting the identification result reduces interference from similar feature regions across multiple label categories.

[0200] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on a computer device, causes the computer device to perform the image recognition method based on the class activation graph algorithm in any of the above method embodiments.

[0201] Executable instructions can be used to cause a computer device to perform the following operations:

[0202] Acquire the image to be recognized;

[0203] The image to be identified is input into a class activation map algorithm based on multi-label gradient feedback. The class activation map algorithm performs feature convolution on the image to be identified to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category, and the inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. Based on the inter-category influence factor and the influence factor of different pixel positions of the current category, the activation map weight of the current category is determined, and the recognition result is obtained based on the activation map weight and the multi-scale feature map. Herein, the current category is any one of multiple label categories.

[0204] Output the recognition results.

[0205] In one optional approach, the class activation map algorithm based on multi-label gradient feedback performs feature convolution on the image to be identified to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category, and an inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. Based on the inter-category influence factor and the influence factor at different pixel positions of the current category, the activation map weight of the current category is determined, and the recognition result is obtained based on the activation map weight and the multi-scale feature map. This further includes:

[0206] The image to be identified is subjected to feature convolution to obtain a multi-scale feature map;

[0207] For each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category at the scale, and the inter-category influence factor corresponding to the current category is determined based on the set of gradient values ​​backpropagated for all categories. The activation map weight of the current category is determined based on the inter-category influence factor and the category influence factor of the current category.

[0208] Based on the activation map weights and the feature maps at the corresponding scales, the current saliency map at each scale corresponding to the current category is obtained;

[0209] Recursively fuse the current saliency maps at each scale corresponding to the current category to obtain the fused saliency map of the intermediate fusion layer corresponding to the current category;

[0210] The fusion saliency map corresponding to the current category is output to obtain the recognition result of the image to be recognized under the current category.

[0211] In an optional approach, for each feature map at each scale, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps at the scale for each label category, and the inter-category influence factor corresponding to the current category is determined based on the set of gradient values ​​backpropagated for all label categories. The activation map weight of the current category is then determined based on the inter-category influence factor and the category influence factor of the current category. This further includes:

[0212] For each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature map of each label category at the scale to obtain the backpropagation gradient set of all category labels at each scale.

[0213] The backpropagation gradient set for all label categories is standardized along the label category dimension to obtain the standardized gradient values ​​for all label categories.

[0214] For the current category, the inter-category influence factor is determined based on the standardized gradient values ​​corresponding to all label categories.

[0215] Calculate the activation graph weight corresponding to the current category based on the inter-category influence factors and the category influence factor of the current category.

[0216] In one alternative approach, the backpropagation gradient set for all label categories is standardized along the label category dimension to obtain standardized gradient values ​​corresponding to all label categories, including:

[0217] The standardized gradient values ​​for all label categories are calculated using the following standardized formula:

[0218] ;

[0219] in, Indicates category label Predicted score Activation graph The gradient backpropagation value is the backpropagation gradient value of the kth channel of the feature map corresponding to the label category and the lth layer of the network. For the current category The set of backpropagation gradients of the k-th channel of the feature map corresponding to each label category and the l-th layer of the network;

[0220] For the current category, the inter-category influence factor is determined based on the standardized gradient values ​​corresponding to all label categories, including:

[0221] The current category is calculated using the following formula. Inter-category influence factors :

[0222] ;

[0223] The step of calculating the activation graph weight corresponding to the current category based on the inter-category influence factors and the category influence factor of the current category includes:

[0224] The current category is calculated using the following formula. Corresponding activation graph weights :

[0225] ,

[0226] ,

[0227] in, It represents the Hadamardi (or Hadama) stack; For the current category The category influence factor in the k-th channel network layer l of the feature map; This indicates the activation pixel position (i,j) in the graph;

[0228] The step of obtaining the current saliency map for each scale corresponding to the current category based on the activation map weights and the feature maps at the corresponding scales includes:

[0229] The current category is calculated using the following formula. The current saliency map of the k-th channel network at layer l in the feature map. :

[0230] .

[0231] In one alternative approach, the recursive fusion of the original activation maps at each scale corresponding to the current category to obtain the fusion saliency map of the intermediate fusion layer corresponding to the current category includes:

[0232] Use bilinear interpolation to upsample the current saliency map of the higher levels of the current category to the size of the current saliency map of the lower levels;

[0233] The current saliency map of the upsampled high layer is fused with the current saliency map of the low layer to obtain the current fused saliency map;

[0234] The iterative process involves using bilinear interpolation to upsample the current saliency map of the higher-level layer of the current category to the size of the current saliency map of the lower-level layer, and then fusing the upsampled current saliency map of the higher-level layer with the current saliency map of the lower-level layer to obtain the current fused saliency map. This process continues until all current saliency maps are fused, recursively obtaining the final fused saliency map.

[0235] In one alternative approach, fusing the upsampled current saliency map of the higher layer with the current saliency map of the lower layer to obtain a current fused saliency map includes:

[0236] For the current category The current saliency graph of the layer network and the first layer network The current saliency maps of the layer networks are fused using the following formula to obtain:

[0237] ;

[0238] in, For the current category corresponding The current saliency graph of the layer; For the current category corresponding The current saliency graph of the layer; The network layer is higher than the layer. layer.

[0239] This invention provides an embodiment of an image to be identified. The image is then input into a class activation map algorithm based on multi-label gradient feedback. This algorithm performs feature convolution on the image to obtain a multi-scale feature map. During gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category. An inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. The activation map weight for the current category is determined based on the inter-category influence factor and the influence factor at different pixel positions within the current category. Finally, the identification result is obtained based on the activation map weight and the multi-scale feature map. The current category can be any of multiple label categories. Outputting the identification result reduces interference from similar feature regions across multiple label categories.

[0240] This invention provides an image recognition device based on a class activation graph algorithm, used to execute the image recognition method based on the class activation graph algorithm described above.

[0241] This invention provides a computer program that can be called by a processor to enable a computer device to execute the image recognition method based on the class activation graph algorithm in any of the above method embodiments.

[0242] This invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed on a computer, cause the computer to perform the image recognition method based on the class activation graph algorithm in any of the above method embodiments.

[0243] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0244] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0245] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0246] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature serving the same, equivalent, or similar purpose.

[0247] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. An image recognition method based on a class activation graph algorithm, characterized in that, The method includes: Acquire the image to be recognized; The image to be identified is input into a class activation map algorithm based on multi-label gradient feedback. The algorithm performs feature convolution on the image to be identified to obtain a multi-scale feature map. During gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category. Based on the set of gradient values ​​backpropagated from all label categories, an inter-category influence factor is determined. Based on the inter-category influence factor and the influence factor at different pixel positions of the current category, the activation map weight of the current category is determined. Based on the activation map weight and the multi-scale feature map, the recognition result is obtained, including: performing feature convolution on the image to be identified to obtain a multi-scale feature map; for each scale of the feature map, during gradient backpropagation for the current category, the algorithm performs gradient backpropagation on the feature maps of each label category. The feature map at each scale is backpropagated using gradients, and the inter-category influence factor corresponding to the current category is determined based on the set of backpropagated gradient values ​​for all categories. The activation map weights for the current category are determined based on the inter-category influence factors and the category influence factor of the current category. The current saliency map at each scale corresponding to the current category is obtained based on the activation map weights and the feature map at the corresponding scale. The current saliency maps at each scale corresponding to the current category are recursively fused to obtain the fused saliency map of the intermediate fusion layer corresponding to the current category. The fused saliency map corresponding to the current category is then output to obtain the recognition result of the image to be recognized in the current category. Here, the current category is any one of multiple label categories; the multiple label categories are the label categories present in the image to be recognized. For each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category at that scale, and the inter-category influence factor corresponding to the current category is determined based on the set of backpropagated gradient values ​​for all label categories. The activation map weights for the current category are then determined based on the inter-category influence factor and the category influence factor of the current category. This further includes: for each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category at that scale, obtaining the backpropagated gradient set for all label categories at each scale; the backpropagated gradient set for all label categories is standardized along the label category dimension to obtain standardized gradient values ​​corresponding to all label categories; for the current category, the inter-category influence factor for the current category is determined based on the standardized gradient values ​​corresponding to all label categories; and the activation map weights for the current category are calculated based on the inter-category influence factor and the category influence factor of the current category. The backpropagation gradient set for all label categories is standardized along the label category dimension to obtain the standardized gradient values ​​corresponding to all label categories, including: The standardized gradient values ​​for all label categories are calculated using the following standardized formula: ； in, The predicted score for category label c Activation graph The gradient backpropagation value is the first value of the feature map corresponding to the label category. k The first channel, network l The layer's backpropagation gradient values; For each label category c, the feature map of the th k The first channel, network l The set of backpropagation gradients for each layer; For the current category, the inter-category influence factor is determined based on the standardized gradient values ​​corresponding to all label categories, including: The current category is calculated using the following formula. Inter-category influence factors : ； The step of calculating the activation graph weight corresponding to the current category based on the inter-category influence factors and the category influence factor of the current category includes: The current category is calculated using the following formula. Corresponding activation graph weights : ， ， Where ⊙ represents the Hadamarda complex; For the current category In the feature map k The first channel network l Layer category influence factor; This indicates the activation pixel position (i,j) in the graph; Output the recognition results.

2. The method according to claim 1, characterized in that, The step of obtaining the current saliency map for each scale corresponding to the current category based on the activation map weights and the feature maps at the corresponding scales includes: The current category is calculated using the following formula. In the feature map, the k-th channel network... l Current saliency graph of the layer : 。 3. The method according to claim 1, characterized in that, The recursive fusion of the original activation maps at each scale corresponding to the current category to obtain the fusion saliency map of the intermediate fusion layer corresponding to the current category includes: Use bilinear interpolation to upsample the current saliency map of the higher levels of the current category to the size of the current saliency map of the lower levels; The current saliency map of the upsampled high layer is fused with the current saliency map of the low layer to obtain the current fused saliency map; The iterative process involves using bilinear interpolation to upsample the current saliency map of the higher-level layer of the current category to the size of the current saliency map of the lower-level layer, and then fusing the upsampled current saliency map of the higher-level layer with the current saliency map of the lower-level layer to obtain the current fused saliency map. This process continues until all current saliency maps are fused, recursively obtaining the final fused saliency map.

4. The method according to claim 3, characterized in that, The step of fusing the current saliency map of the upsampled high-level layer with the current saliency map of the low-level layer to obtain the current fused saliency map includes: For the current category The The current saliency graph of the layer network and the first layer network The current saliency maps of the layer networks are fused using the following formula to obtain: ; in, For the current category corresponding The current saliency graph of the layer; For the current category corresponding The current saliency graph of the layer; The network layer is higher than the layer. layer.

5. An image recognition device based on a class activation graph algorithm, characterized in that, The device includes: The acquisition module is used to acquire the image to be recognized; The class activation map algorithm module is used to input the image to be recognized into a class activation map algorithm based on multi-label gradient feedback. The class activation map algorithm based on multi-label gradient feedback performs feature convolution on the image to be recognized to obtain a multi-scale feature map. When performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category, and the inter-category influence factor is determined based on the set of gradient values ​​backpropagated from all label categories. Based on the inter-category influence factor and the influence factor of different pixel positions of the current category, the activation map weight of the current category is determined, and the recognition result is obtained based on the activation map weight and the multi-scale feature map. This includes: performing feature convolution on the image to be recognized to obtain a multi-scale feature map; for each scale feature map, when performing gradient backpropagation for the current category, the algorithm performs gradient backpropagation on the feature maps of each label category. The feature maps at the specified scale of the category are backpropagated using gradients. Based on the set of backpropagated gradient values ​​from all categories, the inter-category influence factor corresponding to the current category is determined. Based on the inter-category influence factor and the category influence factor of the current category, the activation map weights of the current category are determined. Based on the activation map weights and the feature maps at the corresponding scales, the current saliency map at each scale corresponding to the current category is obtained. The current saliency maps at each scale corresponding to the current category are recursively fused to obtain the fused saliency map of the intermediate fusion layer corresponding to the current category. The fused saliency map corresponding to the current category is then output to obtain the recognition result of the image to be recognized in the current category. Here, the current category is any one of multiple label categories; the multiple label categories are the label categories present in the image to be recognized. For each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category at that scale, and the inter-category influence factor corresponding to the current category is determined based on the set of backpropagated gradient values ​​for all label categories. The activation map weights for the current category are then determined based on the inter-category influence factor and the category influence factor of the current category. This further includes: for each scale of feature map, when performing gradient backpropagation for the current category, gradient backpropagation is performed on the feature maps of each label category at that scale, obtaining the backpropagated gradient set for all label categories at each scale; the backpropagated gradient set for all label categories is standardized along the label category dimension to obtain standardized gradient values ​​corresponding to all label categories; for the current category, the inter-category influence factor for the current category is determined based on the standardized gradient values ​​corresponding to all label categories; and the activation map weights for the current category are calculated based on the inter-category influence factor and the category influence factor of the current category. The backpropagation gradient set for all label categories is standardized along the label category dimension to obtain the standardized gradient values ​​corresponding to all label categories, including: The standardized gradient values ​​for all label categories are calculated using the following standardized formula: ； in, Indicates category label Predicted score Activation graph The gradient backpropagation value is the first value of the feature map corresponding to the label category. k The first channel, network l The layer's backpropagation gradient values; For each label category c, the feature map of the th k The first channel, network l The set of backpropagation gradients for each layer; For the current category, the inter-category influence factor is determined based on the standardized gradient values ​​corresponding to all label categories, including: The current category is calculated using the following formula. Inter-category influence factors : ； The step of calculating the activation graph weight corresponding to the current category based on the inter-category influence factors and the category influence factor of the current category includes: The current category is calculated using the following formula. Corresponding activation graph weights ; ， ， Where ⊙ represents the Hadamarda complex; For the current category In the feature map k The first channel network l Layer category influence factor; This indicates the activation pixel position (i,j) in the graph; The output module is used to output the recognition results.

6. The apparatus according to claim 5, characterized in that, The class activation graph algorithm module further includes: The input module is used to receive the image to be identified; Convolutional layers are used to perform feature convolution on the image to be identified, resulting in multi-scale feature maps; The multi-label gradient feedback module is used to perform gradient backpropagation on the feature maps of each label category when performing gradient backpropagation on the current category, and determine the inter-category influence factor based on the set of gradient values ​​backpropagated by all label categories. Based on the inter-category influence factor and the influence factor of different pixel positions of the current category, the activation map weight of the current category is determined, and the current saliency map at each scale corresponding to the current category is obtained based on the activation map weight and the multi-scale feature map. The recursive fusion intermediate layer saliency map module is used to recursively fuse the current saliency map at each scale corresponding to the current category to obtain the fused saliency map of the fusion intermediate layer corresponding to the current category. The model output module is used to output the recognition result of the image to be recognized under the current category based on the fusion saliency map corresponding to the current category.

7. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the image recognition method based on the class activation graph algorithm as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the operation of the image recognition method based on the class activation graph algorithm as described in any one of claims 1-4.

Images