Fine-grained image classification method based on multi-level information fusion and electronic equipment

Abstract

The embodiment of the invention provides a multi-level information fusion fine-grained image classification method and electronic equipment, and the method comprises the steps: obtaining a to-be-classified image, inputting the image into a trained fine-grained classification model, and obtaining a classification result; the fine-grained classification model comprises a backbone network of at least five stages, and each stage comprises a plurality of stacked convolution blocks; all stages of the backbone network are connected in sequence; the model processing process comprises the steps of processing a feature map output at the nth stage through an attention mechanism to obtain a weighted feature map, and performing clustering processing and classification prediction on the feature map and the weighted feature map to obtain a prediction probability output at the nth stage; the prediction probabilities of the at least two stages are selected to be combined, a final classification result is obtained, and n is smaller than or equal to the stage maximum value of the backbone network. According to the method, the problem that the requirements of high resolution and high complexity of fine-grained image classification processing cannot be met in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a multi-level information fusion fine-grained image classification method, a multi-level information fusion fine-grained classification model training method, an electronic device and a computer-readable medium. Background Art

[0002] Fine-grained image classification is an important computer vision task that aims to finely classify objects with high similarity in detail features and small differences between subspecies. Compared with ordinary image classification, this method has its own particularity and high complexity. For example, in ordinary image classification, a model may only need to distinguish whether an object is a cow or a horse, while in fine-grained image classification, it is necessary to identify the specific breed of cow or horse.

[0003] The research on fine-grained image classification is driven by a large number of computer vision technologies, including image preprocessing, feature extraction, deep learning, etc. Among them, the rapid development of deep learning is particularly noteworthy. Image preprocessing is the most basic but crucial step. First, the image needs to be standardized by cleaning, cutting, scaling, rotating, etc. to make the input data easier to process. Secondly, image noise is removed by conversion, enhancement or special data styles to help the model better understand and learn image features. In the feature extraction stage, we use various algorithms (such as SIFT, SURF, HOG, etc.) to extract key features of the image, such as color, texture, shape, etc. However, these traditional methods cannot meet the high resolution and high complexity requirements of fine-grained image classification. Summary of the invention

[0004] The embodiment of the present invention provides a multi-level information fusion fine-grained image classification method and electronic device to solve the problem that the prior art cannot meet the high resolution and high complexity requirements of fine-grained image classification processing.

[0005] The embodiment of the present invention discloses a multi-level information fusion fine-grained image classification method, which is characterized by comprising:

[0006] Obtain an image to be classified, input the image into a trained fine-grained classification model, and obtain a classification result output by the fine-grained classification model;

[0007] The fine-grained classification model includes: a backbone network of at least five stages, and each stage includes a plurality of stacked convolutional blocks; the stages of the backbone network are connected in sequence;

[0008] The model processing process includes: processing the feature map output by the nth stage through the attention mechanism to obtain a weighted feature map, clustering the feature map and the weighted feature map and performing classification prediction to obtain the predicted probability of the nth stage output; starting from the high to low stage order, selecting the predicted probabilities of at least two stages to combine and obtain the final classification result, where n is less than or equal to the highest stage value of the backbone network.

[0009] Optionally, the prediction probabilities of at least two stages are selected in descending order to be combined to obtain a final classification result, including:

[0010] The predicted probabilities of all stages are averaged to obtain the predicted probability of the final classification result.

[0011] Optionally, the method further comprises:

[0012] Get the training dataset;

[0013] Using the training data set to train the pre-built fine-grained classification model to obtain the trained fine-grained classification model;

[0014] Wherein, in the training stage, the fine-grained classification model further includes: an image enhancement processing structure for obtaining an enhanced image of the original image connected to the feature map output of each stage;

[0015] The image enhancement processing structure is used for clustering and upsampling the feature map outputted by the nth stage with the original image after each image in the training data set is inputted into the fine-grained classification model, and performing mask processing after upsampling to the size of the original image to obtain an enhanced image; the label of the enhanced image is consistent with the label of the original image, and the enhanced image is an image that hides some significant features in the original image;

[0016] The enhanced image is used as a training picture and input into the fine-grained classification model for training.

[0017] Optionally, a pre-built fine-grained classification model is trained using a training data set to obtain a trained fine-grained classification model, including:

[0018] Each stage of the backbone network performs convolution on the input image and outputs the feature maps of each stage;

[0019] The feature map outputted at the nth stage is processed by the attention mechanism to obtain a weighted feature map. The feature map and the weighted feature map are clustered and classified to obtain the predicted probability and supervision loss value L of the nth stage output. n ; Start from the high to low stage order and select at least two stages of prediction probabilities to combine and get the final classification result and supervision loss value L n+1 ,

[0020] The loss value L corresponding to the final classification result n+1 , the loss value L of all stage outputs n Sum all of them to get the loss function of the model;

[0021] Based on the loss function of the model, the stochastic gradient descent method is used to optimize the model parameters, and the trained fine-grained classification model is obtained after preset rounds.

[0022] Optionally, the feature map outputted at the nth stage is processed by an attention mechanism to obtain a weighted feature map, and the feature map and the weighted feature map are clustered and classified and predicted, including:

[0023] The n-th stage outputs the feature map F n First, clustering operation is performed according to the cosine distance, and all feature maps are grouped according to the clusters obtained by the clustering operation. Every 3 feature maps represent a local area, which is expressed as follows:

[0024]

[0025] J(X) represents the clustering objective function, and its parameters are updated iteratively during training; K represents the number of clusters formed by clustering; u k is the center vector of a cluster; x ki is the i-th sample of the k-th cluster;

[0026] Then perform an average pooling operation on the three feature maps of each cluster to obtain a local area feature map, f k =pool(f k1 ,f k2 ,f k3 );

[0027] f k represents the local area feature map formed by pooling the three feature maps of the kth cluster, f k1 、f k2 、f k3 Three feature maps representing the kth class;

[0028] Concatenate all local region feature maps of stage n to obtain the component-level feature map A n Then A n With F n After fusion, we get F n ', which means as follows:

[0029]

[0030] ⊙ represents the Hadamard product operation;

[0031] The feature maps output from other stages are processed in the same way to obtain features with richer semantic information;

[0032] All stages of F n 'Perform a global average pooling flattening, and then send them to the classifier for classification prediction. At the same time, F n 'After flattening, they are concatenated and sent to a classifier to make predictions on the image.

[0033] Optionally, based on the loss function of the model, a stochastic gradient descent method is used to optimize the model parameters, and a trained fine-grained classification model is obtained after a preset number of rounds, including:

[0034] When the backbone network consists of 5 stages, and the prediction result of the 5th stage is determined by the supervised loss value For supervision, the prediction result of the fourth stage is determined by the supervision loss value The prediction result of the third stage is supervised by the supervised loss value Supervision is performed, and the prediction results after the concatenation of the 5th stage, the 4th stage, and the 3rd stage are determined by the supervised loss value For supervision, the overall loss function L is:

[0035]

[0036] Select stochastic gradient descent SGD to optimize the model parameters. The initial learning rate is set to 1e-3, the momentum is set to 0.9, the learning rate is multiplied by 0.1 every 2 epochs, and the weight decay coefficient is set to 1e-4.

[0037] The entire training cycle is divided into 4 stages. At the beginning, only The loss supervises the model. After P1 epochs, the The loss supervises the model and after P2 epochs, the The model is supervised and then after P3 epochs, the Under the supervision of all losses, the model learns P4 epochs to obtain the trained fine-grained classification model; P1, P2, P3, and P4 are all natural numbers greater than or equal to 30.

[0038] The embodiment of the present invention also discloses a multi-level information fusion fine-grained classification model training method, including:

[0039] Get the training dataset;

[0040] Using the training data set to train the pre-built fine-grained classification model to obtain the trained fine-grained classification model;

[0041] The fine-grained classification model includes: a backbone network of at least five stages, and each stage includes a plurality of stacked convolution blocks; the stages of the backbone network are connected in sequence; an image enhancement processing structure for obtaining an enhanced image of the original image connected to the feature map output of each stage;

[0042] The image enhancement processing structure is used to cluster and upsample the feature map output by the nth stage with the original image after each image in the training data set is input into the fine-grained classification model, and then perform mask processing after upsampling to obtain an enhanced image; the label of the enhanced image is consistent with the label of the original image, and the enhanced image is an image that hides some significant features in the original image; n is less than or equal to the highest value of the stage of the backbone network, and the enhanced image is input as a training picture into the fine-grained classification model for training.

[0043] Optionally, a pre-built fine-grained classification model is trained using a training data set to obtain a trained fine-grained classification model, including:

[0044] Each stage of the backbone network performs convolution on the input image and outputs the feature maps of each stage;

[0045] The feature map outputted at the nth stage is processed by the attention mechanism to obtain a weighted feature map. The feature map and the weighted feature map are clustered and classified to obtain the predicted probability and supervision loss value L of the nth stage output. n ; Start from the high to low stage order and select at least two stages of prediction probabilities to combine and get the final classification result and supervision loss value L n+1 ,

[0046] The loss value L corresponding to the final classification result n+1 , the loss value L of all stage outputs n Sum all of them to get the loss function of the model;

[0047] Based on the loss function of the model, the stochastic gradient descent method is used to optimize the model parameters, and the trained fine-grained classification model is obtained after preset rounds.

[0048] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0049] The memory is used to store computer programs;

[0050] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0051] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0052] The embodiments of the present invention include the following advantages:

[0053] The multi-level information fusion fine-grained image classification method of the embodiment of the present invention extracts global information in the model training stage and supplements the global information at the same time to improve the prediction accuracy of the model. In the use of the model, the semantic information of different depths of the network is fused to finally generate a feature representation containing richer semantic information, and the final classification score prediction is made using the information of each layer, which overcomes the problem that the existing model only uses the deeper and more global information for the final classification and ignores some shallow detailed information.

[0054] The training method of the embodiment of the present invention constructs a target local image through the output features of the two stages of the backbone network to obtain more training data, and then uses the above features for classification prediction to make the training more accurate. That is, an information enhancement strategy is adopted to extract the various parts of the model's attention using multi-layer convolution features, and upsample to the original image size, and finally undergo a feature masking process to obtain a number of masked target part images. These images can be used as input images to continue to be input into the network for training, and the model performance is improved by increasing the number of samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart of a multi-level information fusion fine-grained image classification method provided in an embodiment of the present invention;

[0056] Figure 2 It is a flowchart of the steps of a multi-level information fusion fine-grained image classification method provided in an embodiment of the present invention;

[0057] Figure 3 is a flow chart of a multi-level information fusion fine-grained classification model training method provided in an embodiment of the present invention;

[0058] Figure 4 Schematic diagram of the processing process of bilinear attention pooling BAP provided in an embodiment of the present invention;

[0059] Figure 5 It is a structural block diagram of a multi-level information fusion fine-grained image classification device provided in an embodiment of the present invention;

[0060] Figure 6 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0061] Figure 7is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] In order to solve the problem that the existing technology cannot meet the high resolution and high complexity requirements of fine-grained image classification, deep learning technology has emerged. In particular, CNN has performed well in practice. CNN can automatically learn and extract multi-level information and identify tiny details that are crucial to classification. Existing CNN structures, such as AlexNet, VGG, GoogLeNet, ResNet, etc., have achieved remarkable results in fine-grained image classification. With the research and development of large language models, another deep learning architecture Transformer has also been increasingly used in the field of image recognition. The Transformer architecture, due to its inherent attention mechanism, often enables the model to obtain better training effects, but due to the large amount of calculation of small sample learning tasks, the Transformer architecture will further increase the amount of calculation, making the training cost too large. Therefore, the present invention will continue to use the CNN (convolutional neural network) architecture.

[0064] The attention mechanism in convolutional neural networks is a method to improve model performance by focusing on different parts of the input data, combining the advantages of convolutional neural networks and Transformers to simultaneously achieve better generalization and larger model capacity. This mechanism is designed to address the problem of traditional convolutional neural networks lacking attention to local key information when dealing with complex scenes and tasks, thereby improving the performance and efficiency of the model. The attention mechanism in convolutional neural networks allows the model to prioritize certain features or areas when processing images, thereby improving the model's ability to extract key information and make accurate predictions.

[0065] The embodiment of the present invention provides a fine-grained classification model for processing fine-grained image classification based on the pytorch framework. Through an information enhancement strategy, more information is obtained from an original image. The strategy includes processing features in two dimensions. On the one hand, semantic information at different depths of the network is integrated to finally generate a feature representation containing richer semantic information, and the information at each layer is used to make the final classification score prediction, thereby overcoming the problem that the network only uses the deeper and more global information for the final classification while ignoring some shallow detail information. On the other hand, since the various levels of the convolutional neural network in the backbone network of the fine-grained classification model have different focus points on the target image, the embodiment of the present invention uses multi-layer convolutional features to extract the various parts that the model focuses on, and upsamples them to the original image size, and finally undergoes a feature mask processing to obtain a number of masked target part images, which can be used as input images to continue to be input into the network for training, thereby improving the model performance by increasing the number of samples.

[0066] Reference Figure 1 , shows a flowchart of a multi-level information fusion fine-grained image classification method provided in an embodiment of the present invention, which may specifically include the following steps:

[0067] 101. Obtain an image to be classified, input the image into a trained fine-grained classification model, and obtain a classification result output by the fine-grained classification model.

[0068] The fine-grained classification model includes: a backbone network of at least five stages, and each stage includes a plurality of stacked convolutional blocks; the stages of the backbone network are connected in sequence.

[0069] The model processing process includes: processing the feature map output by the nth stage through the attention mechanism to obtain a weighted feature map, clustering the feature map and the weighted feature map (corresponding to Figure 4 The processing of bilinear attention pooling (BAP) and classification prediction shown in FIG. 1 is performed to obtain the predicted probability of the output of the nth stage; the predicted probabilities of at least two stages are selected and combined in a stage order from high to low to obtain the final classification result, where n is less than or equal to the highest value of the stage of the backbone network.

[0070] For example, if there are prediction probabilities corresponding to prediction branches in three stages, the sum of the prediction branches in the three stages also obtains the prediction probability of a total prediction branch, that is, the prediction probabilities of four prediction branches are obtained, and the average value of the four prediction branches is finally taken as the final prediction result, which is expressed as follows:

[0071]

[0072] Combination Figure 3 For example, in the method use / testing stage, Y1 can be the output prediction score of stage 3; Y2 is the output prediction score of stage 4; Y3 is the output prediction score of stage 5; and Y4 is the comprehensive prediction score after integrating the three features. At this time, Y belongs to the final classification result.

[0073] In this embodiment, selecting the prediction probabilities of at least two stages in order from high to low to combine and obtain the final classification result may include: averaging the prediction probabilities of all stages to obtain the prediction probability of the final classification result.

[0074] The multi-level information fusion fine-grained image classification method of this embodiment extracts global information in the model training stage and supplements the global information to improve the prediction accuracy of the model. In the use of the model, the semantic information of different depths of the network is fused to finally generate a feature representation containing richer semantic information, and the information of each layer is used to make the final classification score prediction, which overcomes the problem that the model only uses the deeper and more global information for the final classification and ignores some shallow detailed information.

[0075] It is understandable that in this embodiment, Figure 1 Before that, it is necessary to train a fine-grained classification model. The specific steps are as follows: Figure 2 As shown:

[0076] 100. Obtain a training data set; use the training data set to train a pre-built fine-grained classification model to obtain a trained fine-grained classification model.

[0077] The training data set of this embodiment may be a preprocessed training data set. For example, after a picture is input, preprocessing operations such as random cropping, horizontal flipping, and normalization are performed to achieve preprocessing.

[0078] Wherein, in the training stage, the fine-grained classification model further includes: an image enhancement processing structure for obtaining an enhanced image of the original image connected to the feature map output of each stage;

[0079] The image enhancement processing structure is used for clustering and upsampling the feature map output by the nth stage with the original image after each image in the training data set is input into the fine-grained classification model, and performing mask processing after upsampling to obtain an enhanced image; the label of the enhanced image is consistent with the label of the original image (that is, the true category of the species in the image), and the enhanced image is an image that hides some significant features in the original image;

[0080] The enhanced image is used as a training picture and input into the fine-grained classification model for training.

[0081] For example, using a training data set to train a pre-built fine-grained classification model to obtain the trained fine-grained classification model may include:

[0082] Each stage of the backbone network performs convolution on the input image and outputs the feature maps of each stage;

[0083] The feature map output by the nth stage is processed by the attention mechanism to obtain a weighted feature map, and the feature map and the weighted feature map are clustered (such as Figure 4 As shown) and classification prediction, the predicted probability and supervision loss value L of the nth stage output are obtained n ; Start from the high to low stage order and select at least two stages of prediction probabilities to combine and get the final classification result and supervision loss value L n+1 ;

[0084] The loss value L corresponding to the final classification result n+1 , the loss value L of all stage outputs n Sum all of them to get the loss function of the model;

[0085] Based on the loss function of the model, the stochastic gradient descent method is used to optimize the model parameters, and the trained fine-grained classification model is obtained after preset rounds.

[0086] In this embodiment, clustering processing and classification prediction of feature graphs and weighted feature graphs include:

[0087] A clustering operation is performed according to the cosine distance, and all feature maps are grouped according to the clusters obtained by the clustering operation. Every three feature maps represent a local area. These three feature maps are averaged and pooled to obtain a local area feature map. All local area feature maps are then concatenated to obtain a component-level feature map. The component-level feature map is then fused with the feature map to obtain a local feature map. Each local feature map is input into the classifier for prediction. The prediction results of all local feature maps are combined to obtain the prediction probability output at the nth stage.

[0088] The training method of this embodiment constructs a target local image through the output features of at least two stages of the backbone network to obtain more training data, and then uses the above features for classification prediction to make the training more accurate. That is, an information enhancement strategy is adopted to extract the various parts of the model's attention using multi-layer convolution features, and upsample to the original image size, and finally undergo a feature masking process to obtain a number of masked target part images, which can be used as input images to continue to be input into the network for training, thereby improving the model performance by increasing the number of samples.

[0089] Combination Figures 1 to 4As shown, the backbone network of the fine-grained classification model of this embodiment includes 5 stages, each of which is composed of a stack of convolutional blocks, and the stages are connected in sequence. In this embodiment, the feature maps output by the third stage (Stage3), the fourth stage (Stage4) and the fifth stage (Stage5) are selected for enhancement processing, and subsequent attention mechanism processing is performed.

[0090] The specific process of the training method in this embodiment is described as follows.

[0091] Step S1: Select a data set and preprocess the data in the data set.

[0092] In this embodiment, the CUB-200-2011 data set can be selected and data segmentation can be performed to divide the data set into a training set (i.e., a training data set), a validation set, and a test set. The data is then preprocessed by random cropping, horizontal flipping, normalization, and the like.

[0093] Step S2: Build a fine-grained classification model.

[0094] In this embodiment, ResNet50 can be selected as the backbone network of the model to extract image features. The ResNet50 network can be roughly divided into 5 stages. Each stage is composed of a number of convolutional blocks stacked together.

[0095] Step S3: Use the training set to train the fine-grained classification model.

[0096] To better understand the above training process, let's combine Figure 3 A detailed description is given, which includes the following sub-steps.

[0097] Sub-step S3.1: Each stage of the backbone network performs convolution processing on the input image and outputs the feature map of each stage;

[0098] Sub-step S3.2: After each image in the training set is input into the fine-grained classification model, the feature maps output from the 3rd, 4th and 5th stages are clustered and upsampled with the original image, and after upsampling, mask processing is performed to obtain an enhanced image; the label of the enhanced image is consistent with the label of the original image, and the enhanced image is an image that hides some significant features in the original image.

[0099] That is to say, since each level of the convolutional neural network focuses on the target image differently, the original image generates a target local image under the guidance of each component-level feature map, and the generated target local image is upsampled to the original image size; for example, the feature maps output by the backbone network stage5, stage4, and stage3 are fused with the original input image respectively, and then upsampled to the original image size after fusion.

[0100] Then, the target local image Xn is processed by masking to hide the more prominent features, and the final data-enhanced image Yn is obtained. The data-enhanced image Yn is re-input into the model, and the label of the data-enhanced image is consistent with the label of the original image, inducing the model to discover more potential features. That is, Yn = mask*Xn.

[0101] Sub-step S3.3: The feature maps output by stage5, stage4, and stage3 are processed through the attention mechanism to obtain weighted feature maps, and the feature maps and weighted feature maps are clustered and classified for prediction.

[0102] like Figure 2 and Figure 3 As shown, the feature maps output by the backbone network stage5, stage4, and stage3 are processed by the attention mechanism respectively, including:

[0103] Stage5 outputs feature map F5 (such as Figure 4 The Feature Maps shown in the figure are first clustered according to the cosine distance, and all feature maps are grouped according to the clusters (Attention Maps) obtained by the clustering operation. Every three feature maps represent a local area, as shown below:

[0104]

[0105] J(X) represents the clustering objective function, and its parameters are updated iteratively during training; K represents the number of clusters formed by clustering; u k is the center vector of a cluster; x ki is the i-th sample of the k-th cluster.

[0106] Then, the three feature maps of each cluster are averaged and pooled to obtain a local area feature map, which is expressed as follows: k =pool(f k1 ,f k2 ,f k3 )

[0107] f k represents the local area feature map formed by pooling the three feature maps of the kth cluster, f k1 、f k2 、f k3 Three feature maps representing the k-th class.

[0108] Concatenate all local area feature maps in stage 5 to obtain the component-level feature map A 5 Then A 5 With F 5After fusion, we get F 5 ', which means as follows:

[0109]

[0110] ⊙ represents the Hadamard product operation.

[0111] The feature F output from stage4 and stage3 4 and F 3 The same process is performed to obtain a feature representation F with richer semantic information. 4 ' and F 3 '.

[0112] F 5 '、F 4 ' and F 3 'Perform a global average pooling flattening, and then send them to the classifier for classification prediction. At the same time, in F 5 '、F 4 ' and F 3 'After flattening, they are concatenated and sent to a classifier to make predictions on the image.

[0113] Sub-step S3.4: Obtain the prediction probabilities and supervision loss values ​​Loss3, Loss2, and Loss1 of the outputs of the three stages stage5, stage4, and stage3; select the prediction probabilities of at least two stages in descending order to combine and obtain the final classification result and supervision loss value Loss4.

[0114] Sum the loss value Loss4 corresponding to the final classification result, and the loss values ​​Loss3, Loss2, and Loss1 output from all stages to get the loss function of the model;

[0115] Based on the loss function of the model, the stochastic gradient descent method is used to optimize the model parameters, and the trained fine-grained classification model is obtained after preset rounds.

[0116] During the training phase, Figure 3 The losses of the four branches shown in are accumulated and back-propagated to update the network parameters.

[0117] In the test phase, the average of the four prediction branches is taken as the final prediction result, which is expressed as follows:

[0118]

[0119] Among them, Y1 is the output prediction score of stage3; Y2 is the output prediction score of stage4; Y3 is the output prediction score of stage5; and Y4 is the comprehensive prediction score after integrating the features of the three parts.

[0120] In this example, the cross-entropy loss function is used to measure the difference between the distribution learned by the model and the true distribution. The prediction result of stage 5 is given by For supervision, the prediction results of stage4 are given by For supervision, the prediction results of stage3 are given by For supervision, the prediction results after stage5, stage4 and stage3 are concatenated are given by To supervise. Therefore, the overall loss function of the network is designed as:

[0121]

[0122] in represents the loss value of stage i, and It represents the deviation between the predicted value and the true value after the concatenation of stage5, stage4 and stage3.

[0123] Sub-step S3.5: Select stochastic gradient descent SGD to optimize the model parameters, where the initial learning rate is set to 1e-3, the momentum is set to 0.9, the learning rate is multiplied by 0.1 every 2 epochs, and the weight decay coefficient is set to 1e-4.

[0124] In order to give full play to the multi-layer information fusion concept, this embodiment divides the entire training cycle into four stages. The loss supervises the model. After 50 epochs, the The loss supervises the model and after 50 epochs, the The model is supervised and then after 50 epochs, the The model learns for another 50 epochs under the supervision of all losses. This shallow-to-deep training strategy can help the model learn more detailed features in the early stage, forcing the model to use these unnoticed detailed features for classification prediction. In this embodiment, 50 epochs are used as an example for explanation. In actual applications, the number of epochs can be the same or different according to needs.

[0125] Most traditional fine-grained image classification models use the deep features of the model to predict image classification, while ignoring the role of shallow detail features. In this embodiment, the model is improved and optimized based on this prominent problem. Not only does it avoid this problem from the model design aspect, but it also adopts a more scientific and reasonable training strategy to maximize the performance of the model. The model of this embodiment not only uses the global information extracted from the deep layer, but also refers to the shallow detail information of the network to supplement the global information and improve the prediction accuracy of the model. At the same time, during the training process, a training strategy from shallow to deep is adopted to force the network to learn shallow detail information, and use these detail information for classification prediction. Finally, overall optimization is performed to improve the model's ability to integrate shallow and deep information, and it can take into account both global information and local detail information in classification prediction, thereby improving classification accuracy.

[0126] In the above embodiment, three stages of feature map output are used as an example. In other embodiments, the enhancement strategy and attention clustering processing are not limited to three stages of feature map output. Two stages, four stages or five stages can be selected according to actual needs. It should be noted that it is recommended to start from the last stage in the selection. Figure 3 As shown, such as the fifth stage, the fourth stage; or the fifth stage, the fourth stage, the third stage, the second stage, etc.

[0127] In addition, the embodiment of the present invention also provides a multi-level information fusion fine-grained classification model training method, which may include the following steps:

[0128] Step 1: Get the training data set;

[0129] Step 2: Use the training data set to train the pre-built fine-grained classification model to obtain the trained fine-grained classification model;

[0130] The fine-grained classification model includes: a backbone network of at least five stages, and each stage includes a plurality of stacked convolution blocks; the stages of the backbone network are connected in sequence; an image enhancement processing structure for obtaining an enhanced image of the original image connected to the feature map output of each stage;

[0131] The image enhancement processing structure is used to cluster and upsample the feature map output by the nth stage with the original image after each image in the training data set is input into the fine-grained classification model, and then perform mask processing after upsampling to obtain an enhanced image; the label of the enhanced image is consistent with the label of the original image, and the enhanced image is an image that hides some significant features in the original image; n is less than or equal to the highest value of the stage of the backbone network, and the enhanced image is input as a training picture into the fine-grained classification model for training.

[0132] In this embodiment, the pre-built fine-grained classification model is trained using a training data set to obtain the trained fine-grained classification model, including:

[0133] Each stage of the backbone network performs convolution on the input image and outputs the feature maps of each stage;

[0134] The feature map outputted at the nth stage is processed by the attention mechanism to obtain a weighted feature map. The feature map and the weighted feature map are clustered and classified to obtain the predicted probability and supervision loss value L of the nth stage output. n ; Start from the high to low stage order and select at least two stages of prediction probabilities to combine and get the final classification result and supervision loss value L n+1 ;

[0135] The loss value L corresponding to the final classification result n+1 , the loss value L of all stage outputs n Sum all of them to get the loss function of the model;

[0136] Based on the loss function of the model, the stochastic gradient descent method is used to optimize the model parameters, and the trained fine-grained classification model is obtained after preset rounds.

[0137] For example, when the backbone network consists of 5 stages, and the prediction result of the 5th stage is determined by the supervised loss value For supervision, the prediction result of the fourth stage is determined by the supervision loss value The prediction result of the third stage is supervised by the supervised loss value Supervision is performed, and the prediction results after the concatenation of the 5th stage, the 4th stage, and the 3rd stage are determined by the supervised loss value For supervision, the overall loss function L is:

[0138]

[0139] Select stochastic gradient descent SGD to optimize the model parameters. The initial learning rate is set to 1e-3, the momentum is set to 0.9, the learning rate is multiplied by 0.1 every 2 epochs, and the weight decay coefficient is set to 1e-4.

[0140] The entire training cycle is divided into 4 stages. At the beginning, only The loss supervises the model. After 50 epochs, the The loss supervises the model and after 50 epochs, the The model is supervised and then after 50 epochs, the The model is then learned for another 50 epochs under the supervision of all Losses to obtain a trained fine-grained classification model.

[0141] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0142] Reference Figure 5 , shows a structural block diagram of a multi-level information fusion fine-grained image classification device provided in an embodiment of the present invention, which may specifically include the following modules:

[0143] An image acquisition module, used to acquire images to be classified;

[0144] A classification prediction module is used to input the image into the trained fine-grained classification model to obtain the classification result output by the fine-grained classification model;

[0145] The fine-grained classification model includes: a backbone network of at least five stages, and each stage includes a plurality of stacked convolutional blocks; the stages of the backbone network are connected in sequence;

[0146] The model processing process includes: processing the feature map output by the nth stage through the attention mechanism to obtain a weighted feature map, clustering the feature map and the weighted feature map and performing classification prediction to obtain the predicted probability of the nth stage output; starting from the high to low stage order, the predicted probabilities of at least two stages are selected and combined to obtain the final classification result, where n is less than or equal to the highest stage value of the backbone network.

[0147] The classification module is also used to average the prediction probabilities of all stages to obtain the prediction probability of the final classification result.

[0148] The device of this embodiment further includes a training module, which can be specifically used to: obtain a training data set; use the training data set to train a pre-built fine-grained classification model to obtain a trained fine-grained classification model;

[0149] Wherein, in the training stage, the fine-grained classification model further includes: an image enhancement processing structure for obtaining an enhanced image of the original image connected to the feature map output of each stage;

[0150] The image enhancement processing structure is used for clustering and upsampling the feature map output by the nth stage with the original image after each image in the training data set is input into the fine-grained classification model, and performing mask processing after upsampling to the size of the original image to obtain an enhanced image; the label of the enhanced image is consistent with the label of the original image, and the enhanced image is an image that hides some significant features in the original image;

[0151] The enhanced image is used as a training picture and input into the fine-grained classification model for training.

[0152] Specifically, each stage of the backbone network performs convolution on the input image and outputs the feature maps of each stage;

[0153] The feature map outputted at the nth stage is processed by the attention mechanism to obtain a weighted feature map. The feature map and the weighted feature map are clustered and classified to obtain the predicted probability and supervision loss value L of the nth stage output. n ; Start from the high to low stage order and select at least two stages of prediction probabilities to combine and get the final classification result and supervision loss value L n+1 ,

[0154] The loss value L corresponding to the final classification result n+1 , the loss value L of all stage outputs n Sum all of them to get the loss function of the model;

[0155] Based on the loss function of the model, the stochastic gradient descent method is used to optimize the model parameters, and the trained fine-grained classification model is obtained after preset rounds.

[0156] In particular, the feature map outputted at the nth stage is processed by the attention mechanism to obtain a weighted feature map, and clustering and classifying the feature map and the weighted feature map may include:

[0157] The n-th stage outputs the feature map F n (like Figure 4 The Feature Maps in the first step are clustered according to the cosine distance, and all feature maps are clustered according to the clustering operation (such as Figure 4 The Attention Maps shown in the figure are grouped, and each 3 feature maps represent a local area (such as Figure 4 Part Feature Maps shown in the figure are as follows:

[0158]

[0159] J(X) represents the clustering objective function, and its parameters are updated iteratively during training; K represents the number of clusters formed by clustering; u kis the center vector of a cluster; x ki is the i-th sample of the k-th cluster;

[0160] Then perform an average pooling operation on the three feature maps of each cluster to obtain a local area feature map, f k =pool(f k1 ,f k2 ,f k3 );

[0161] f k represents the local area feature map formed by pooling the three feature maps of the kth cluster, f k1 、f k2 、f k3 Three feature maps representing the kth class;

[0162] All local area feature maps of the nth stage are concatenated (i.e. Figure 4 Concatenate in ), and obtain the component-level feature map A n Then A n With F n After fusion, we get F n ', which means as follows:

[0163]

[0164] ⊙ represents the Hadamard product operation;

[0165] The feature maps output from other stages are processed in the same way to obtain features with richer semantic information;

[0166] All stages of F n 'Perform a global average pooling flattening, and then send them to the classifier for classification prediction. At the same time, F n 'After flattening, they are concatenated and sent to a classifier to predict the image.

[0167] Furthermore, based on the loss function of the model, the model parameters are optimized using the stochastic gradient descent method, and a fine-grained classification model is obtained after training for a preset number of rounds, including:

[0168] When the backbone network consists of 5 stages, and the prediction result of the 5th stage is determined by the supervised loss value For supervision, the prediction result of the fourth stage is determined by the supervision loss value The prediction result of the third stage is supervised by the supervised loss value Supervision is performed, and the prediction results after the concatenation of the 5th stage, the 4th stage, and the 3rd stage are determined by the supervised loss value For supervision, the overall loss function L is:

[0169]

[0170] In this embodiment, the output features of the last three convolutional blocks of the network are used to construct a target local image to obtain more training data; at the same time, the features are used for classification prediction to form the final loss function.

[0171] Select stochastic gradient descent SGD to optimize the model parameters. The initial learning rate is set to 1e-3, the momentum is set to 0.9, the learning rate is multiplied by 0.1 every 2 epochs, and the weight decay coefficient is set to 1e-4.

[0172] The entire training cycle is divided into 4 stages. At the beginning, only The loss supervises the model. After 50 epochs, the The loss supervises the model and after 50 epochs, the The model is supervised and then after 50 epochs, the The model is then learned for another 50 epochs under the supervision of all Losses to obtain a trained fine-grained classification model.

[0173] The device of this embodiment constructs the target local image through the output features of the two stages of the backbone network, obtains more training data, and then uses the above features for classification prediction, making the training more accurate. That is, an information enhancement strategy is adopted to extract the various parts of the model's attention using multi-layer convolution features, and upsample to the original image size, and finally undergo a feature masking process to obtain a number of masked target part images, which can be used as input images to continue to be input into the network for training, thereby improving the model performance by increasing the number of samples.

[0174] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0175] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0176] Memory 1303, used for storing computer programs;

[0177] The processor 1301 is used to implement the following steps when executing the program stored in the memory 1303:

[0178] Obtain an image to be classified, input the image into a trained fine-grained classification model, and obtain a classification result output by the fine-grained classification model;

[0179] The fine-grained classification model includes: a backbone network of at least five stages, and each stage includes a plurality of stacked convolutional blocks; the stages of the backbone network are connected in sequence;

[0180] The model processing process includes: processing the feature map output by the nth stage through the attention mechanism to obtain a weighted feature map, clustering the feature map and the weighted feature map and performing classification prediction to obtain the predicted probability of the nth stage output; starting from the high to low stage order, the predicted probabilities of at least two stages are selected and combined to obtain the final classification result, where n is less than or equal to the highest stage value of the backbone network.

[0181] And, the following steps may also be included:

[0182] Get the training dataset;

[0183] Using the training data set to train the pre-built fine-grained classification model to obtain the trained fine-grained classification model;

[0184] Wherein, in the training stage, the fine-grained classification model further includes: an image enhancement processing structure for obtaining an enhanced image of the original image connected to the feature map output of each stage;

[0185] The image enhancement processing structure is used for clustering and upsampling the feature map output by the nth stage with the original image after each image in the training data set is input into the fine-grained classification model, and performing mask processing after upsampling to the size of the original image to obtain an enhanced image; the label of the enhanced image is consistent with the label of the original image, and the enhanced image is an image that hides some significant features in the original image;

[0186] The enhanced image is used as a training picture and input into the fine-grained classification model for training.

[0187] Specifically, a pre-built fine-grained classification model is trained using a training data set to obtain a trained fine-grained classification model, including:

[0188] Each stage of the backbone network performs convolution on the input image and outputs the feature maps of each stage;

[0189] The feature map outputted at the nth stage is processed by the attention mechanism to obtain a weighted feature map. The feature map and the weighted feature map are clustered and classified to obtain the predicted probability and supervision loss value L of the nth stage output. n ; Start from the high to low stage order and select at least two stages of prediction probabilities to combine and get the final classification result and supervision loss value Ln+1 ,

[0190] The loss value L corresponding to the final classification result n+1 , the loss value L of all stage outputs n Sum all of them to get the loss function of the model;

[0191] Based on the loss function of the model, the model parameters are optimized using the stochastic gradient descent method, and the fine-grained classification model is obtained after training for a preset number of rounds.

[0192] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0193] The communication interface is used for communication between the above terminal and other devices.

[0194] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0195] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0196] like Figure 7As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1401 is also provided, in which instructions are stored. When the computer-readable storage medium 1401 is run on a computer, the computer executes the fine-grained image classification method with multi-level information fusion described in the above embodiment.

[0197] In another embodiment provided by the present invention, a computer program product containing instructions is also provided. When the computer program product is run on a computer, the computer executes the multi-level information fusion fine-grained image classification method described in the above embodiment.

[0198] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0199] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0200] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0201] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A multi-level information fusion fine-grained image classification method, characterized in that: include: Obtain an image to be classified, input the image into a trained fine-grained classification model, and obtain a classification result output by the fine-grained classification model; The fine-grained classification model includes: a backbone network of at least five stages, and each stage includes a plurality of stacked convolutional blocks; the stages of the backbone network are connected in sequence; The model processing process includes: processing the feature map output by the nth stage through the attention mechanism to obtain a weighted feature map, clustering the feature map and the weighted feature map and performing classification prediction to obtain the predicted probability of the nth stage output; starting from the high to low stage order, selecting the predicted probabilities of at least two stages to combine and obtain the final classification result, where n is less than or equal to the highest stage value of the backbone network.

2. The method according to claim 1, characterized in that Starting from the highest to the lowest stage, the predicted probabilities of at least two stages are selected and combined to obtain the final classification results, including: The predicted probabilities of all stages are averaged to obtain the predicted probability of the final classification result.

3. The method according to claim 1, characterized in that The method further comprises: Get the training dataset; Using the training data set to train the pre-built fine-grained classification model to obtain the trained fine-grained classification model; Wherein, in the training stage, the fine-grained classification model further includes: an image enhancement processing structure for obtaining an enhanced image of the original image connected to the feature map output of each stage; The image enhancement processing structure is used for clustering and upsampling the feature map outputted by the nth stage with the original image after each image in the training data set is inputted into the fine-grained classification model, and performing mask processing after upsampling to the size of the original image to obtain an enhanced image; the label of the enhanced image is consistent with the label of the original image, and the enhanced image is an image that hides some significant features in the original image; The enhanced image is used as a training picture and input into the fine-grained classification model for training.

4. The method according to claim 3, characterized in that The pre-built fine-grained classification model is trained using the training data set to obtain the trained fine-grained classification model, including: Each stage of the backbone network performs convolution on the input image and outputs the feature maps of each stage; The feature map outputted at the nth stage is processed by the attention mechanism to obtain a weighted feature map. The feature map and the weighted feature map are clustered and classified to obtain the predicted probability and supervision loss value L of the nth stage output. n ; Start from the high to low stage order and select at least two stages of prediction probabilities to combine and get the final classification result and supervision loss value L n+1 , The loss value L corresponding to the final classification result n+1 , the loss value L of all stage outputs n Sum all of them to get the loss function of the model; Based on the loss function of the model, the stochastic gradient descent method is used to optimize the model parameters, and the trained fine-grained classification model is obtained after preset rounds.

5. The method according to claim 4, characterized in that The feature map outputted from the nth stage is processed by the attention mechanism to obtain a weighted feature map, and the feature map and the weighted feature map are clustered and classified for prediction, including: The n-th stage outputs the feature map F n First, clustering operation is performed according to the cosine distance, and all feature maps are grouped according to the clusters obtained by the clustering operation. Every 3 feature maps represent a local area, which is expressed as follows: J(X) represents the clustering objective function, and its parameters are updated iteratively during training; K represents the number of clusters formed by clustering; u k is the center vector of a cluster; x ki is the i-th sample of the k-th cluster; Then perform an average pooling operation on the three feature maps of each cluster to obtain a local area feature map, f k =pool(f k1 ,f k2 ,f k3 ); f k represents the local area feature map formed by pooling the three feature maps of the kth cluster, f k1 、f k2 、f k3 Three feature maps representing the kth class; Concatenate all local region feature maps of stage n to obtain the component-level feature map A n Then A n With F n After fusion, we get F n ', which means as follows: ⊙ represents the Hadamard product operation; The feature maps output from other stages are processed in the same way to obtain features with richer semantic information; All stages of F n 'Perform a global average pooling flattening, and then send them to the classifier for classification prediction. At the same time, F n 'After flattening, they are concatenated and sent to a classifier to make predictions on the image.

6. The method according to claim 4, characterized in that Based on the loss function of the model, the stochastic gradient descent method is used to optimize the model parameters. After a preset number of rounds, a fine-grained classification model is obtained after training, including: When the backbone network consists of 5 stages, and the prediction result of the 5th stage is determined by the supervised loss value For supervision, the prediction result of the fourth stage is determined by the supervision loss value The prediction result of the third stage is supervised by the supervised loss value Supervision is performed, and the prediction results after the concatenation of the 5th stage, the 4th stage, and the 3rd stage are determined by the supervised loss value For supervision, the overall loss function L is: Select stochastic gradient descent SGD to optimize the model parameters. The initial learning rate is set to 1e-3, the momentum is set to 0.9, the learning rate is multiplied by 0.1 every 2 epochs, and the weight decay coefficient is set to 1e-4. The entire training cycle is divided into 4 stages. At the beginning, only The loss supervises the model. After P1 epochs, the The loss supervises the model and after P2 epochs, the The model is supervised and then after P3 epochs, the Under the supervision of all losses, the model learns P4 epochs to obtain the trained fine-grained classification model; P1, P2, P3, and P4 are all natural numbers greater than or equal to 30.

7. A multi-level information fusion fine-grained classification model training method, characterized in that: include: Get the training dataset; Using the training data set to train the pre-built fine-grained classification model to obtain the trained fine-grained classification model; The fine-grained classification model includes: a backbone network of at least five stages, and each stage includes a plurality of stacked convolution blocks; the stages of the backbone network are connected in sequence; an image enhancement processing structure for obtaining an enhanced image of the original image connected to the feature map output of each stage; The image enhancement processing structure is used to cluster and upsample the feature map output by the nth stage with the original image after each image in the training data set is input into the fine-grained classification model, and then perform mask processing to obtain an enhanced image after upsampling to the original image size; the label of the enhanced image is consistent with the label of the original image, and the enhanced image is an image that hides some significant features in the original image; n is less than or equal to the highest value of the stage of the backbone network, and the enhanced image is input as a training picture into the fine-grained classification model for training.

8. The method according to claim 7, characterized in that The pre-built fine-grained classification model is trained using the training data set to obtain the trained fine-grained classification model, including: Each stage of the backbone network performs convolution on the input image and outputs the feature maps of each stage; The feature map outputted at the nth stage is processed by the attention mechanism to obtain a weighted feature map. The feature map and the weighted feature map are clustered and classified to obtain the predicted probability and supervision loss value L of the nth stage output. n ; Start from the high to low stage order and select at least two stages of prediction probabilities to combine and get the final classification result and supervision loss value L n+1 , The loss value L corresponding to the final classification result n+1 , the loss value L of all stage outputs n Sum all of them to get the loss function of the model; Based on the loss function of the model, the stochastic gradient descent method is used to optimize the model parameters, and the trained fine-grained classification model is obtained after preset rounds.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 8 when executing the program stored in the memory.

10. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 8.

Images