A bird classification method based on contrastive hierarchical correlation propagation theory

By comparing the confidence scores of hierarchical correlation propagation theory and deep neural networks, the problems of many iterations and high calculation costs in the existing technology are solved, and the key feature area display and better classification effect of bird classification are realized. in fine-grained image classification.

CN114724184BActive Publication Date: 2024-11-26CIVIL AVIATION UNIV OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210373006.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-11
Publication Date
2024-11-26
Estimated Expiration
2042-04-11

AI Technical Summary

Technical Problem

The prior art requires multiple iterations and high computational costs in bird classification, and there is a problem of the importance of gradient disappearance and difficulty in fully reflecting input features in deep networks.

Method used

Using a method based on the contrast hierarchical correlation propagation theory, the spatial-level correlation mask of the key feature areas of the image data set is obtained, and the channel-level weight is determined using the confidence score of the deep neural network, and linearly weighted to generate a target significant graph. to identify key feature areas of birds that affect model decisions.

Benefits of technology

A more obvious display of key feature areas of bird classification is achieved, the classification effect is better, the dependence on gradients is avoided, the model interpretability and classification speed is improved, and it is suitable for fine-grained image classification tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114724184B_ABST
    Figure CN114724184B_ABST
Patent Text Reader

Abstract

The present invention provides a bird classification method based on contrastive hierarchical correlation propagation theory. Obtain a data set of images to be classified as birds, and calculate the spatial-level correlation mask of the key feature area of ​​each image to be classified as birds in the bird image data set by contrastive hierarchical correlation propagation; according to the correlation mask, use the confidence score of a deep neural network to determine the channel-level weight of each image to be classified as birds; linearly weight the channel-level weight and the spatial-level correlation mask to generate a saliency map for marking different types of birds. The present invention achieves advanced performance in the three evaluation indicators of average decline, average growth, and energy-based, and can better discover important areas that affect model decisions, thereby classifying birds more accurately and efficiently. The present invention is applied to the task of fine-grained bird image classification, analyzes the misclassification behavior of the model, and guides the training process of the model to achieve more accurate bird classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of bird identification, and in particular to a bird classification method based on contrast hierarchical correlation propagation theory. Background Art

[0002] At present, with the development of deep learning technology, complex neural network systems have achieved outstanding achievements in the fields of image, sound and natural language processing, and are widely used in real-world tasks such as public security, medical auxiliary diagnosis, environmental protection and agricultural monitoring. It has a huge potential application space. Although neural network models have excellent performance in some scenarios, even surpassing humans. However, compared with traditional, easy-to-interpret machine learning algorithms built by manually designed feature sets and expert experience, representation learning algorithms represented by convolutional neural networks are highly dependent on the nonlinearity and debugging techniques of the model, and emphasize the generalization performance of the model when designing the algorithm, while abandoning the requirement of interpretability. The lack of interpretability of deep neural networks seriously restricts their further application in fields with high security requirements.

[0003] The current mainstream research methods on the interpretability of deep neural networks make use of the natural advantages of convolutional neural networks in image data processing, and highlight the important areas that affect the model's prediction results by visualizing the features that affect network decisions in images or videos. From a broad definition, these methods can be divided into perturbation-based methods, backpropagation-based methods, and class activation mapping-based methods. Perturbation-based algorithms can be seen as attribution processes that modify the input to detect the importance of the representation vector to the system output.

[0004] In the prior art of bird classification, typical perturbation-based methods can be divided into three types: simple perturbation, meaningful perturbation, and generative perturbation. However, if a fixed-size pixel block is used, it is necessary to block each area of ​​the input image in order from left to right and from top to bottom, so as to obtain the importance of the blocked area to the model decision. If the perturbation mask is generated in an optimized manner, important pixels can be effectively blocked so that the target category score reaches a local minimum. If a generative perturbation method is used, a more natural perturbation image is obtained visually. Although these methods can obtain better visual explanations, they require multiple iterations of the model, which takes a long time and has high computational costs. Back-propagation-based algorithms are mainly divided into gradient-based methods and methods based on hierarchical correlation propagation theory. In the prior art, ordinary gradient propagation methods and guided gradient propagation methods have also been proposed, but these two methods have problems such as gradient disappearance and difficulty in fully reflecting the importance of input features during deep network propagation. Summary of the invention

[0005] The present invention provides a bird classification method based on contrast hierarchical correlation propagation theory, which is used to accurately and efficiently find the key feature areas that affect bird classification, and solves the problems that the existing technologies need to iterate the model multiple times, which takes a long time and has high computational costs. The common gradient propagation method and the guided gradient propagation method have problems such as gradient disappearance and difficulty in fully reflecting the importance of input features during the deep network propagation process.

[0006] A bird classification method based on contrastive hierarchical correlation propagation theory, including:

[0007] Acquire a data set of images to be classified as birds, and calculate a spatial level correlation mask of a key feature area of ​​each image to be classified as birds in the data set of images by comparing hierarchical correlation propagation;

[0008] Determining the channel-level weight of each bird classification image using the confidence score of the deep neural network according to the correlation mask;

[0009] The channel-level weights are linearly weighted with the spatial-level activation mask to generate target saliency maps for marking different types of birds, and to determine the key feature areas of birds that affect the model's decision.

[0010] Preferably, the method further comprises:

[0011] The bird classification images in the bird classification image data set are input into a preset deep neural network to determine the prediction scores of birds of different target categories; wherein,

[0012] The prediction score is obtained by the following formula:

[0013] y c =f c (I,θ)

[0014] Among them, y c represents the prediction score of the deep neural network f(x) for the target category c, where c represents the type parameter of the bird; I represents the bird image to be classified; θ is the trained model parameter;

[0015] The predicted scores are used as a starting point for propagating correlations across levels.

[0016] Preferably, the comparison level correlation propagation calculation includes a network forward calculation stage and a correlation score backward transmission stage.

[0017] Preferably, the network forward calculation stage includes:

[0018] Establish a deep neural network based on multiple nested nonlinear functions; among them,

[0019] The nonlinear functions are nested to form a multi-layer correlation calculation network;

[0020] According to the deep neural network, the correlation output value of each bird classification image is calculated respectively; wherein,

[0021] The correlation output value is calculated by the following formula:

[0022]

[0023] in, represents the correlation output value of the i-th bird classification image at the l+1 layer; j represents the activation function; x (l) Represents the correlation output value of the bird classification image at the l+1 layer; represents the weight shared by the i-th bird classification image in the l+1 layer network; The deviation of the i-th bird classification image at the l+1 layer; l represents the number of layers, i=1, 2, 3...n; n represents the number of bird images in the bird classification image dataset.

[0024] Preferably, the relevance score reverse transfer stage includes:

[0025] Constructing multi-level correlation propagation rules according to the correlation calculation network;

[0026] The correlation output value is spatially decomposed according to the multi-level correlation propagation rule to determine the decomposition variables of each bird classification image during spatial decomposition;

[0027] Determine a correlation score based on the decomposition variables and the correlation output value; wherein,

[0028] The relevance score represents the contribution of the network decision to each position in each image to be classified as a bird in the comparison level relevance propagation calculation;

[0029] Determining the relevance score also includes normalizing the relevance score; wherein,

[0030] The standardization is to fix the value of the correlation score in the interval [0-1];

[0031] According to the contribution value, a correlation mask is generated; wherein,

[0032] The correlation mask is determined by the following formula:

[0033]

[0034]

[0035] Among them, Rc represents the category-related score generated for the target category c; Represents R c The spatial set of relevance scores of each position in the k-th bird image feature channel with respect to the target category c; min(R c ) represents the minimum category-related score generated for the target category c; max(R c ) represents the maximum category-related score generated for the target category c; A represents the spatial and correlation mask of the kth bird image feature channel with respect to the target category c; k Represents the image feature channel of the kth bird image to be classified.

[0036] Preferably, the method further comprises:

[0037] Determining, according to the correlation mask, a masking area masked by the correlation mask;

[0038] The masked area is replaced by blurred input image information to generate a mask image.

[0039] Preferably, the method further comprises:

[0040] Inputting the mask image into a deep neural network to determine the importance of each correlation mask;

[0041] The importance is used as the channel-level weight of each bird classification image; wherein,

[0042] The channel-level weight is shown in the following formula:

[0043]

[0044] in, represents the channel-level weights of k bird images to be classified with respect to target category c; f c (I′) represents the prediction score of the target category c when the bird mask image is input into the deep neural network; f c (I′) represents the prediction score of the target category c when the blurred bird image is input into the deep neural network;

[0045] Preferably, the method further comprises:

[0046] Generating a first saliency map according to the linear weighting;

[0047] Determining salient pixels of different color depths according to the first saliency map;

[0048] Classifying and marking different birds according to the significant pixels;

[0049] A target saliency map is generated according to the classification labels.

[0050] Preferably, the method further comprises:

[0051] Acquire historical bird classification data, and determine a second salient map in the target salient map that is prone to misclassification;

[0052] The second saliency map is cropped, and an upsampling operation is performed after the cropping; wherein,

[0053] The cropping step comprises: determining the length and width of the bird in the second salient image, setting a minimum bounding box according to the length and width, and cropping the second salient image according to the minimum bounding box;

[0054] The upsampled second salient image is re-contrasted and hierarchical correlation propagation is calculated to enhance the key areas of the bird classification image.

[0055] Preferably, the method further comprises:

[0056] Obtaining a mask area of ​​a target saliency map and the upsampling result;

[0057] Determining whether there is image noise according to the mask area and the upsampling result;

[0058] Determining an image quality of the target saliency map according to the image noise;

[0059] The image quality is used as the convergence result of each target saliency map, and each target saliency map is iteratively converged through the contrast level correlation propagation calculation, and when the convergence result remains unchanged, a final target saliency map is determined.

[0060] The beneficial effects of the present invention are as follows: the present invention gets rid of the dependence on gradients, and uses contrasting hierarchical correlation propagation and the confidence score of the network to implement a coarse-to-fine interpretation of the deep neural network from the spatial domain and the channel domain, thereby achieving a more obvious display of key feature areas for bird classification, and achieving better classification results. The method of the present invention is intuitive and easy to understand, and has achieved advanced performance in the three evaluation indicators of average decline, average growth, and energy-based. It can better discover important areas that affect model decisions, thereby classifying birds more quickly. When interpreting a deep neural network, there is no need to modify or retrain the structure of the model, which is more universal and convenient to use, and has passed the integrity check. It is a network debugging tool and data enhancement strategy, which is applied to fine-grained image classification tasks, analyzes the model's misclassification behavior, and guides the model's training process, achieving significant results and achieving more accurate bird classification results. The technical effects of the present invention are as shown in the attached figure. Figure 4 As shown, the present invention can perform classification more clearly through the saliency map.

[0061] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0062] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0064] Figure 1 It is a method flow chart of a bird classification method based on comparative hierarchical correlation propagation theory in an embodiment of the present invention;

[0065] Figure 2 Schematic diagram of calculation in the network forward calculation phase and the correlation score reverse transmission phase in an embodiment of the present invention;

[0066] Figure 3 is an overall process flow chart for obtaining a saliency map in an embodiment of the present invention;

[0067] Figure 4 This is a result diagram of bird key feature area positioning in an embodiment of the present invention;

[0068] Figure 5 This is an identification diagram of distant birds in an embodiment of the present invention;

[0069] Figure 6 This is a display process diagram of data enhancement in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0071] Deep learning algorithms represented by convolutional neural networks have achieved great success in bird classification, but these algorithms are highly dependent on the nonlinearity of the model and debugging technology. Black box properties are prevalent in the application process, which seriously limits their application in security-sensitive tasks. To this end, the present invention proposes a bird classification method based on the contrastive hierarchical correlation propagation theory. The present invention is based on a coarse-to-fine class activation mapping algorithm (CF-CAM) for diagnosing the erroneous decision-making behavior of deep neural networks. The algorithm re-establishes the relationship between feature maps and model decisions, uses contrastive hierarchical correlation propagation to obtain the contribution of each position in the feature map to the network decision to generate a spatial-level correlation mask, finds the important area that affects the model decision, and then linearly weights the input image after the fuzzification operation and re-inputs it into the network to obtain the importance score of the feature map, and implements a coarse-to-fine interpretation of the deep neural network from the spatial domain and the channel domain. The CF-CAM of the present invention has better target positioning capabilities than existing methods. Applying CF-CAM as a network debugging tool and data enhancement strategy to the bird image classification task can effectively improve the accuracy of network recognition. Compared with the prior art, the effectiveness and superiority of the CF-CAM method of the present invention.

[0072] Embodiment 1:

[0073] As attached Figure 1 and attached Figure 3 As shown, a bird classification method based on contrast hierarchical correlation propagation theory includes:

[0074] Acquire a data set of images to be classified as birds, and calculate a spatial level correlation mask of a key feature area of ​​each image to be classified as birds in the data set of images by comparing hierarchical correlation propagation;

[0075] Determining the channel-level weight of each bird classification image using the confidence score of the deep neural network according to the correlation mask;

[0076] The channel-level weights and the spatial-level activation masks are linearly weighted to generate target saliency maps for marking different types of birds, accurately and efficiently displaying and determining key feature areas of birds that affect model decisions.

[0077] The principle of the above technical solution is: the principle of the present invention is that in the process of bird classification, bird images are classified through a novel and effective fine-grained class activation mapping CF-CAM, that is, a comparative hierarchical correlation propagation calculation, channel-level weights and linear weighting method, to generate more obvious and significant images of the bird images that need to be classified, and accurately and efficiently find the key feature areas that affect bird classification.

[0078] The beneficial effects of the above technical solution are: the present invention gets rid of the dependence on gradients, and uses contrasting hierarchical correlation propagation and the confidence score of the network to realize a coarse-to-fine interpretation of the deep neural network from the spatial domain and the channel domain, achieving a more obvious bird classification display and better classification effect. The method of the present invention is intuitive and easy to understand, and has achieved advanced performance in the three evaluation indicators of average decline, average growth and energy-based. It can better discover important areas that affect model decisions, thereby classifying birds more quickly. When interpreting the deep neural network, there is no need to modify and retrain the structure of the model. It is more universal and convenient to use, and has passed the integrity check. It is a network debugging tool and data enhancement strategy, which is applied to fine-grained image classification tasks, analyzes the misclassification behavior of the model, and guides the training process of the model. It has achieved significant results and achieved clearer bird classification. The technical effects of the present invention are as shown in the attached figure. Figure 4 As shown, the present invention can more accurately find the key feature areas of birds through the saliency map and achieve better classification performance.

[0079] Embodiment 2:

[0080] Preferably, the method further comprises:

[0081] The bird classification images in the bird classification image data set are input into a preset deep neural network to determine the prediction scores of birds of different target categories; wherein,

[0082] The prediction score is obtained by the following formula:

[0083] y c =f c (I,θ)

[0084] Among them, y c represents the prediction score of the deep neural network f(x) for the target category c, where c represents the type parameter of the bird; I represents the bird image to be classified; θ is the trained model parameter;

[0085] The predicted scores are used as a starting point for propagating correlations across levels.

[0086] The principle of the above technical solution is that in the process of bird classification, the present invention adopts the comparative hierarchical correlation propagation theory. Therefore, when classifying, prediction is performed through deep neural network training. This prediction is a prediction of different bird types, that is, through existing pictures, in the initial prediction, it is determined which bird types there are, and each type is reflected in the form of a score.

[0087] The beneficial effect of the above technical solution is that, compared with the prior art, the present invention is more continuous in the final classification decision.

[0088] Embodiment 3:

[0089] Preferably, the comparison level correlation propagation calculation includes a network forward calculation phase and a correlation score reverse transmission phase, as shown in the attached Figure 2 shown.

[0090] The principle of the above technical solution is that the present invention has two major steps, and also uses the contrast hierarchical correlation propagation theory to obtain the contribution of each position in the feature map to the network decision to generate a spatial-level correlation mask. Then, the correlation mask upsampled to the size of the input image is weighted with the input image, and after meaningful perturbation, it is re-input into the network to obtain the importance score of the feature map as the channel-level weight of the feature map. Finally, the generated channel-level weight is weighted and summed with the spatial-level correlation mask to generate the final saliency map, highlighting the key feature areas of the bird.

[0091] The beneficial effect of the above technical solution is that compared with the deep neural network classification method in the prior art, the present invention has a more reliable explanation for the classification of deep neural networks, and can accurately and efficiently determine the important areas that affect the model decision, that is, the key factors for bird classification.

[0092] Embodiment 4:

[0093] Preferably, the network forward calculation stage includes:

[0094] Establish a deep neural network based on multiple nested nonlinear functions; among them,

[0095] The nonlinear functions are nested to form a multi-layer correlation calculation network;

[0096] According to the deep neural network, the correlation output value of each bird classification image is calculated respectively; wherein,

[0097] The correlation output value is calculated by the following formula:

[0098]

[0099] in, represents the correlation output value of the i-th bird classification image at the l+1 layer; j represents the activation function; x (l) Represents the correlation output value of the bird classification image at the l+1 layer; represents the weight shared by the i-th bird classification image in the l+1 layer network; The deviation of the i-th bird classification image at the l+1 layer; l represents the number of layers, i=1, 2, 3...n; n represents the number of bird images in the bird classification image dataset.

[0100] The principle of the above technical solution is that in the process of network forward calculation, the present invention uses each bird image to be classified as an input vector of different levels of the neural network based on the neural network, activates it under a preset activation function, and then determines the output value of each neuron through the neuron deviation of each layer. The deep neural network is composed of multiple nonlinear functions, has more continuity, can realize layer-by-layer deviation detection, and then outputs the corresponding correlation output value.

[0101] In the present invention, the output value f(x) of the network can be decomposed from the output value to each variable in the input space according to the specified hierarchical correlation propagation rule, thereby measuring the correlation score R between each variable and the network output value. The commonly used hierarchical correlation propagation rule is z + Rules and β Rules: The definition is as follows:

[0102]

[0103]

[0104] in, and Respectively represent the positive and negative weights connecting the neurons in the lth layer and the l+1th layer; [U, V] represents the value range of the neuron activation value. In the design process of the CF-CAM algorithm, the present invention uses the comparative hierarchical correlation propagation to obtain the contribution of each position in the feature map to the network decision, and this contribution value is the correlation score.

[0105] The beneficial effect of the above technical solution is that the deep neural network of the present invention is composed of a number of such nested nonlinear functions, so it has a high degree of nonlinearity and excellent detection performance.

[0106] Embodiment 5:

[0107] Preferably, the relevance score reverse transfer stage includes:

[0108] Constructing multi-level correlation propagation rules according to the correlation calculation network;

[0109] The correlation output value is spatially decomposed according to the multi-level correlation propagation rule to determine the decomposition variables of each bird classification image during spatial decomposition;

[0110] Determine a correlation score based on the decomposition variables and the correlation output value; wherein,

[0111] The relevance score represents the contribution of the network decision to each position in each image to be classified as a bird in the comparison level relevance propagation calculation;

[0112] Determining the relevance score also includes normalizing the relevance score; wherein,

[0113] The standardization is to fix the value of the correlation score in the interval [0-1];

[0114] According to the contribution value, a correlation mask is generated; wherein,

[0115] The correlation mask is determined by the following formula:

[0116]

[0117]

[0118] Among them, R c represents the category-related score generated for the target category c; Represents R c The spatial set of correlation scores of each position of the feature channel of the kth bird image to be classified with respect to the target category c; min(R c ) represents the minimum category-related score generated for the target category c; max(R c ) represents the maximum category-related score generated for the target category c; A represents the spatial and correlation mask of the kth bird image feature channel with respect to the target category c; k Represents the image feature channel of the kth bird image to be classified.

[0119] The principle of the above technical solution is that in the stage of back-propagation of the correlation score, the present invention performs back-propagation to the target layer layer by layer to obtain the correlation score of each position in all feature maps of the target layer to the model decision. In order to avoid the lack of continuity of the generated category correlation mask due to the different contributions of each position in the feature map to the model decision, before generating the category correlation mask, the category correlation score of the feature map is first standardized so that its value is between the interval [0,1] to generate a smooth category correlation mask.

[0120] The beneficial effect of the above technical solution is that, compared with the prior art, the correlation mask generated by the present invention does not lack continuity and has more standard and smoother characteristics.

[0121] Embodiment 6:

[0122] Preferably, the method further comprises:

[0123] Determining, according to the correlation mask, a masking area masked by the correlation mask;

[0124] The masked area is replaced by blurred input image information to generate a mask image.

[0125] The principle of the above technical solution is that the present invention multiplies the generated spatial level correlation mask with the corresponding feature map element by element according to the channel dimension to retain the pixels in the image that are related to the model decision. However, because there is a sharp boundary between the retained pixel area and the unreserved pixel area, it is easy to produce adversarial effects in the network prediction process. The present invention replaces the area shielded by the correlation mask with blurred input image information to generate a mask image.

[0126] The beneficial effect of the above technical solution is that the image mask generated by the present invention eliminates the sharp boundary between the retained pixel area and the unreserved pixel area, and will not produce adversarial effects in the prediction process.

[0127] Embodiment 7:

[0128] Preferably, the method further comprises:

[0129] Inputting the mask image into a deep neural network, and determining the importance of each correlation mask using the confidence score of the deep neural network;

[0130] The importance is used as the channel-level weight of each bird classification image; wherein,

[0131] The channel-level weight is shown in the following formula:

[0132]

[0133] in, represents the channel-level weights of k bird images to be classified with respect to target category c; f c (I′) represents the prediction score of the target category c when the bird mask image is input into the deep neural network; f c (I′) represents the predicted score of the target category c when the blurred bird image is input into the deep neural network; the principle of the above technical solution is that in the process of determining the channel-level weight, the main principle of the present invention is that as the number of convolutional neural network layers increases, the network extracts richer semantic information in the image, and only by performing channel-level weighted adjustment on the final feature map can the class activation map most relevant to the specified category be generated. Therefore, the present invention first uses the contrast hierarchical correlation propagation theory to obtain the contribution of each position in the feature map to the network decision to generate a spatial-level correlation mask, and then weights the input image with the correlation mask upsampled to the size of the input image, and re-inputs it into the network after meaningful perturbation to obtain the importance score of the feature map as the channel-level weight of the feature map, and visualizes the important features of each channel in the form of a heat map.

[0134] The beneficial effect of the above technical solution is that the category correlation scores obtained by back-propagation of contrastive hierarchical correlation propagation theory are used as spatial-level masks, which gets rid of the dependence on the global average pooling layer and makes the class activation mapping-based method more general.

[0135] Embodiment 8:

[0136] Preferably, the method further comprises:

[0137] Generating a first saliency map according to the linear weighting;

[0138] Determining salient pixels of different color depths according to the first saliency map;

[0139] Classifying and marking different birds according to the significant pixels;

[0140] A target saliency map is generated according to the classification labels.

[0141] The principle of the above technical solution is that after the present invention generates a saliency map, the saliency map can only display the birds more visually through different degrees of brightness. This is the role of saliency pixels, but there is no distinction. Therefore, the present invention adopts a classification marking method to mark different birds through classification marking to achieve bird distinction.

[0142] The beneficial effect of the above technical solution is that compared with the prior art, the result obtained by the present invention is clearer and different birds can be distinguished more clearly.

[0143] Embodiment 9:

[0144] Preferably, the method further comprises:

[0145] Acquire historical bird classification data, and determine a second salient map in the target salient map that is prone to misclassification;

[0146] The second saliency map is cropped, and an upsampling operation is performed after the cropping; wherein,

[0147] The cropping step comprises: determining the length and width of the bird in the second salient image, setting a minimum bounding box according to the length and width, and cropping the second salient image according to the minimum bounding box;

[0148] The upsampled second salient image is re-contrasted and hierarchical correlation propagation is calculated to enhance the key areas of the bird classification image.

[0149] The principle of the above technical solution is that the present invention is also a data enhancement method. The present invention generates a saliency map for the middle layer of the network, crops the bird images that are easily misclassified, and then performs an upsampling operation on the cropped image, restores it to the input image size, and then re-inputs it into the network to highlight the key areas in the image that affect the model decision. Among them, in order to improve the tolerance of cropping, this paper does not directly crop after calculating the mask area through the saliency map, but after calculating the minimum bounding box, appropriately retains the surrounding part of the area indicated by the saliency map.

[0150] The beneficial effect of the above technical solution is that the present invention can effectively extract the fine-grained features of birds photographed at a long distance, such as the attached Figure 5 As shown, effective focus on these important areas enables accurate distinction of birds.

[0151] Embodiment 10:

[0152] Preferably, the method further comprises:

[0153] Obtaining a mask area of ​​a target saliency map and the upsampling result;

[0154] Determining whether there is image noise according to the mask area and the upsampling result;

[0155] Determining an image quality of the target saliency map according to the image noise;

[0156] The image quality is used as the convergence result of each target saliency map, and each target saliency map is iteratively converged through the contrast level correlation propagation calculation, and when the convergence result remains unchanged, a final target saliency map is determined.

[0157] The principle of the above technical solution is that in the process of data enhancement of saliency map, we include input image, saliency map, mask area determined by saliency map and upsampling result. Since noise reflects the performance of the network to a certain extent, the quality of the saliency map generated by the present invention is used as an indication of whether the network has converged. When the generated saliency map no longer changes, it means that the network may have converged, and the network training is stopped to obtain the most accurate saliency map, as shown in the attached figure. Figure 6 shown.

[0158] The beneficial effect of the above technical solution is that: through the convergence value, the present invention can obtain a bird recognition result with a higher accuracy in the process of continuous iterative calculation.

[0159] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A bird classification method based on contrastive hierarchical correlation propagation theory, characterized in that: include: Acquire a data set of images to be classified as birds, and calculate a spatial level correlation mask of a key feature area of ​​each image to be classified as birds in the data set of images by comparing hierarchical correlation propagation; Determining the channel-level weight of each bird classification image using the confidence score of the deep neural network according to the correlation mask; Linearly weighting the channel-level weights and the spatial-level activation masks to generate target saliency maps for marking different types of birds and determine key feature areas of birds that affect model decisions; Wherein, the comparison level correlation propagation calculation includes a network forward calculation stage and a correlation score backward transmission stage; The network forward calculation stage includes: Establish a deep neural network based on multiple nested nonlinear functions; among them, The nonlinear functions are nested to form a multi-layer correlation calculation network; According to the deep neural network, the correlation output value of each bird classification image is calculated respectively; wherein, The correlation output value is calculated by the following formula: in, Indicates The bird classification image is in The correlation output value of the layer; represents the activation function; Indicates that the bird classification image is in The correlation output value of the layer; Indicates The bird classification image is in The weights shared by the layer network; No. The bird classification images are in Deviation of layers; Indicates the number of layers, ; Represents the number of bird images in the bird classification image dataset; The relevance score reverse transfer stage includes: Constructing multi-level correlation propagation rules according to the correlation calculation network; Decomposing the correlation output value in space according to the multi-level correlation propagation rule, determining the decomposition variables of each bird classification image during the spatial decomposition, and determining the pixel position that affects the model decision; Determine a correlation score based on the decomposition variables and the correlation output value; wherein, The relevance score represents the contribution of the network decision to each position in each bird classification image in the comparison hierarchical relevance propagation theory calculation; Determining the relevance score also includes normalizing the relevance score; wherein, The standardization is to fix the value of the correlation score within an interval; According to the contribution value, a correlation mask is generated; wherein, The correlation mask is determined by the following formula: in, Represents the target category Generated category relevance scores; express Middle Each position in the feature channel of the bird image to be classified is related to the target category The spatial set of correlation scores of ; Represents the target category The minimum class relevance score generated; Represents the target category The maximum class relevance score generated; Indicates The feature channels of the bird images to be classified are related to the target category Spatial and correlation masks of; Indicates The image feature channels of the bird images to be classified.

2. A bird classification method based on contrastive hierarchical correlation propagation theory as claimed in claim 1, characterized in that: The method further comprises: The bird classification images in the bird classification image data set are input into a preset deep neural network to determine the prediction scores of birds of different target categories; wherein, The prediction score is obtained by the following formula: in, Representing a deep neural network For target category The predicted score of Represents the type parameters of the bird; represents the bird image to be classified; are the trained model parameters; The predicted scores are used as the starting point for the theoretical calculation of propagation of comparative hierarchical relevance.

3. A bird classification method based on contrastive hierarchical correlation propagation theory as claimed in claim 1, characterized in that: The method further comprises: Determining, according to the correlation mask, a masking area masked by the correlation mask; The masked area is replaced by blurred input image information to generate a mask image.

4. A bird classification method based on contrastive hierarchical correlation propagation theory as claimed in claim 3, the method further comprising: Inputting the mask image into a deep neural network, and determining the importance of each correlation mask in combination with the confidence score of the deep neural network; The importance is used as the channel-level weight of each bird classification image; wherein, The channel-level weight is shown in the following formula: in, express Bird images to be classified about the target category The channel-level weights of Represents the bird mask image input to the deep neural network for the target category The predicted score of Represents the target category of the blurred bird image input into the deep neural network The predicted score of .

5. A bird classification method based on contrastive hierarchical correlation propagation theory as claimed in claim 1, characterized in that: The method further comprises: Generating a first saliency map according to the linear weighting; Determining salient pixels of different color depths according to the first saliency map; Classifying and marking different birds according to the significant pixels; A target saliency map is generated according to the classification labels.

6. A bird classification method based on contrastive hierarchical correlation propagation theory as claimed in claim 1, characterized in that: The method further comprises: Acquire historical bird classification data, and determine a second salient map in the target salient map that is prone to misclassification; The second saliency map is cropped, and an upsampling operation is performed after the cropping; wherein, The cropping step comprises: determining the length and width of the bird in the second salient image, setting a minimum bounding box according to the length and width, and cropping the second salient image according to the minimum bounding box; The upsampled second salient image is re-contrasted and hierarchical correlation propagation is calculated to enhance the key feature areas of the bird classification image.

7. A bird classification method based on contrastive hierarchical correlation propagation theory as claimed in claim 6, characterized in that: The method further comprises: Obtaining a mask region of a bird target saliency map and the upsampling result; Determining whether there is image noise according to the mask area and the upsampling result; Determining an image quality of the target saliency map according to the image noise; The image quality is used as the convergence result of each target saliency map, and each target saliency map is iteratively converged through the contrast level correlation propagation calculation, and when the convergence result remains unchanged, a final target saliency map is determined.