A fine-grained image classification method based on multi-layer coordinated convolutional neural network

Abstract

The present invention relates to the fields of deep learning and image classification, and more specifically to a fine-grained image classification method based on a multi-layer coordinated convolutional neural network. This method locates key regions and extracts features for fine-grained image classification tasks. It utilizes multi-scale image cropping and padding to train different convolutional layers, thereby fusing shallow and deep network features. It also disrupts the integrity of the image by disrupting local regions and records the original positions through position encoding to reduce noise introduced by the disrupted image. Compared to existing classification methods, the multi-layer coordinated fine-grained classification method of the present invention achieves higher accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and image classification, and in particular to a fine-grained image classification method based on a multi-layer coordinated convolutional neural network. Background Art

[0002] Traditional classification tasks generally refer to coarse classification, such as between cats and dogs. Because their distinguishing features are large, classification is relatively easy. Fine-grained classification, on the other hand, is a subtask of image classification, focusing on subcategories of the same category. For example, Huskies and Alaskan Malamutes are very similar in appearance due to their close ancestry, with differences limited to a few local areas, such as eye color and the shape of the fur on their foreheads. Fine-grained recognition focuses on local features. A single image has numerous local features, and learning useful features from these numerous local features is a challenge. Especially when there are relatively few images, it is easy to learn incorrect features, resulting in overfitting to the training set. In recent years, much work has focused on attention mechanisms, which can help networks focus more on key local areas and improve classification accuracy.

[0003] There are two main approaches to fine-grained classification. One is based on strongly supervised classification models. Strong supervision, in addition to using category labels, also requires additional information such as manually annotated object bounding boxes and local part bounding boxes. For example, Wei X S et al. used manually annotated part annotation points and category labels in the local localization module of the Mask-CNN algorithm during training. Part R-CNN and Pose Normalized CNN also use additional manually annotated information, such as object and part annotation points, in the discriminative domain extraction process. Extensive manual annotation is very expensive, so classification methods based on weakly supervised learning are the mainstream trend. The other is based on weakly supervised classification models, which rely solely on category labels and do not use additional part annotation information. For example, Ge et al. proposed a classification model in 2019. This model uses CAM to identify key regions of an image through a classification model. It then iteratively generates more suitable object proposals (multiple) through CRF correction and object detection methods. A supplementary model selection algorithm is then used to select the most suitable proposal from these proposals. An LSTM is then used for feature extraction and classification. Labels are limited to the image's category labels, making it a weakly supervised algorithm. In order to learn key regions and features, in addition to using a standard basic classification network, Chen et al. proposed a destruction and construction learning (DCL) method to improve the accuracy of fine-grained image recognition. Although the features extracted by existing methods have certain identification capabilities, it is challenging to effectively extract features of key regions when only category labels are available. Summary of the Invention

[0004] To address the challenges of the existing technology, the present invention provides a fine-grained image classification method based on a multi-layer coordinated convolutional neural network. This method leverages the characteristics of each layer of the convolutional network and coordinates features across multiple layers to improve final classification accuracy. The technical solution of the present invention is as follows:

[0005] Obtain an image data set and preprocess the images to be classified in the image data set;

[0006] A convolutional neural network is used to extract the image features of the image to be classified and input them into the positioning subnet. The positioning subnet is used to obtain the positioning key area and obtain the key area subgraph of the image to be classified.

[0007] Performing multi-scale cutting and filling on the key area sub-images, and randomly exchanging each image block to obtain multiple groups of cutting and filling sub-images of different scales;

[0008] Position-encode the image blocks in each group of cut and fill sub-images, and connect the corresponding position-encoded feature maps with the cut and fill sub-images according to the channels;

[0009] The cut and filled sub-images of different scales are input into the first classification model of the preset training sub-network in turn to obtain the probability value of the corresponding category;

[0010] The probability values of the corresponding categories of the cut and filled sub-images of different scales are input into the second classification model of the preset training sub-network. After weighted averaging, the fine-grained classification results of the image to be classified are obtained.

[0011] Furthermore, the method of using the positioning subnet to obtain the positioning key area and obtain the key area subgraph of the image to be classified includes:

[0012] The extracted image features are summed according to the channels, and the resulting summed feature map is bilinearly upsampled to obtain a saliency map of the same size as the image to be classified;

[0013] The saliency map is selected according to the adaptive threshold to obtain a mask matrix, and the mask matrix is mapped to the corresponding image to be classified to obtain the focus part;

[0014] The attention part is upsampled by bilinear interpolation to obtain an attention image of the same size as the image to be classified, that is, a key area subimage of the image to be classified.

[0015] Furthermore, the selection of the saliency map according to the adaptive threshold includes calculating the adaptive threshold based on the average value of the saliency map and the hyperparameter, and determining the mask matrix based on the size relationship between the adaptive threshold and the corresponding matrix element in the saliency map, that is, when the adaptive threshold is greater than the corresponding matrix element in the saliency map, the corresponding matrix element in the mask matrix is 1, otherwise it is 0.

[0016] Furthermore, the step of performing multi-scale cutting and filling on the key region sub-image and randomly exchanging image blocks to obtain multiple groups of cutting and filling sub-images of different scales includes:

[0017] Cut the key area sub-images into different N×N sub-images;

[0018] Fill each sub-image at each cutting scale with 0 to obtain the filled sub-image IP n , the filled sub-image set is I pad {IP n |0≤n <N 2};

[0019] According to the spatial position of the image to be classified, the zero-filled sub-images at each cutting scale are randomly spliced into a new image to be classified;

[0020] Downsample the new image to be classified at each cutting scale to obtain a cutting and filling sub-image of the same size as the original image to be classified:

[0021] Among them, IP n represents the image after filling the nth sub-image, n represents the sub-image number of the key area sub-image, that is, the sub-image of the focus image, and N represents the side length of the image block; I pad {·} represents the set of padded sub-images.

[0022] Furthermore, the position encoding of the image blocks in each group of cut and filled sub-images includes inputting the sequence of image blocks into the sine-cosine position coding function PSC respectively to obtain the position encoding of the corresponding image blocks; and quantizing the position encoding according to the filling granularity and cutting granularity of the cut and filled sub-image to obtain a position encoding feature map.

[0023] Furthermore, the method sequentially inputs the cut and filled sub-images of different scales into the preset first classification model to obtain the probability value of the corresponding category, including inputting each cut and filled sub-image into different residual convolution modules to obtain weighted feature maps of different levels; downsampling all weighted feature maps except the last level, and splicing and fusing the downsampled weighted feature maps with the weighted feature maps of the next level; and using the weighted feature map of the first level and other spliced and fused feature maps as multi-level feature maps, respectively passing through a linear layer and a softmax classifier to obtain the probability value of the corresponding category.

[0024] Furthermore, the probability values of corresponding categories of cut and filled sub-images of different scales are input into a preset second classification model, and after weighted averaging, a fine-grained classification result of the image to be classified is obtained, including inputting the key area sub-image into the residual convolution module, and fusing the feature maps of the key area sub-image with the multi-level feature maps to form a multi-level feature vector, performing hierarchical bilinear pooling on the above feature vectors in pairs to obtain a multi-layer feature fusion feature vector; and passing the feature vectors through a linear layer and a softmax classifier respectively to obtain the probability value of the corresponding category.

[0025] The preset first classification model and the preset second classification model are obtained after training respectively, and the training process includes calculating the loss value of the output category value and the true label through the cross entropy loss function respectively, calculating its gradient, and performing back propagation, and continuously updating the parameters through the gradient descent method.

[0026] The advantages and beneficial effects of the present invention are as follows:

[0027] 1. In the present invention, the positioning subnet is used to output the bounding box of the target, and the training subnet is used to train the network parameters. The positioning subnet and the training subnet share network parameters and their input image size is consistent with the convolutional feature extractor, which can reduce image processing time.

[0028] 3. When training the entire model, the present invention can use different scale images for training different convolutional layers, thereby improving the robustness of the network model. At the same time, resnet50 is adapted separately for each scale image, which can save time and cost;

[0029] 4. The image dataset used in the training process of the present invention only requires classification labels, without manually labeled bounding boxes, which can effectively reduce computing consumption.

[0030] 5. During the training process of the entire model of the present invention, from the original data input to the result output, from the input end to the output end, the intermediate neural network is self-contained, eliminating the need for separate training of intermediate parameters and preventing module connection dependency problems. Therefore, the present invention is an end-to-end model.

[0031] 6. The present invention is based on multi-scale cutting and filling. It disrupts the pixels of the image, destroys the integrity, adds position coding, retains the original pixel position information, and finally fuses the features of multi-layer convolutional networks. Compared with the existing method based on feature fusion, the accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the present invention;

[0033] Figure 2 It is a network structure diagram of the present invention;

[0034] Figure 3 This is the feature fusion diagram of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] Example 1

[0037] The purpose of this invention is to extract local detail features and overall semantic features of fine-grained images in classification tasks by utilizing different layers of convolutional neural networks, different receptive fields, and training different scale cut-and-fill images, and finally to fuse different features to improve classification accuracy.

[0038] like Figure 1 As shown, the present invention provides a fine-grained classification method based on multi-layer coordination of convolutional neural networks, including the following:

[0039] S1. Obtain an image dataset and preprocess the images to be classified in the image dataset;

[0040] In an embodiment of the present invention, the image dataset can be either known training set data and test set data, or an unknown image dataset to be classified. If it is known training set data, the training set images need to be labeled with corresponding category labels. For example, the category label of training set image A is dog-husky. If it is an unknown image to be classified or a test set image, the category of the image needs to be predicted through a subsequent neural network model.

[0041] In an embodiment of the present invention, preprocessing the image to be classified may include conventional image preprocessing methods such as cropping, spatial transformation, denoising, image enhancement, etc. The present invention does not make specific limitations on this, and those skilled in the art can select the corresponding method according to actual conditions.

[0042] S2. Use a convolutional neural network to extract the image features of the image to be classified, and use the positioning subnet to obtain the positioning key area to obtain the key area subgraph of the image to be classified;

[0043] In an embodiment of the present invention, the original image to be classified is input into a convolutional neural network, and feature extraction can be performed in the last convolutional layer or the last two convolutional layers, wherein the convolutional neural network can be a conventional convolutional neural network, a recurrent convolutional neural network, a residual convolutional neural network, etc. The present invention preferably uses the Resnet50 model to extract image features of the image to be classified.

[0044] In the embodiment of the present invention, the method of using the positioning subnet to obtain the positioning key area and obtain the key area subgraph of the image to be classified includes:

[0045] S21, summing the extracted image features according to the channels, and performing bilinear upsampling on the obtained summed feature map to obtain a saliency map of the same size as the image to be classified;

[0046] In the embodiment of the present invention, since the convolutional neural network can extract the image features of the image to be classified, the image features can be input into the LCU (Location, Cutting and Upsample) model in this embodiment. The model locates, crops and upsamples the image to the original image size, forcing the network to pay more attention to the local area. The model can be used as the positioning subnet in this embodiment. It can take the intersection of the last two layers of feature maps after extraction and sum them according to the channel, so as to make the saliency map more accurate. The summation formula is expressed as:

[0047]

[0048] The formula for obtaining the saliency map is:

[0049] S(x,y)=g(M(x,y))

[0050] Among them, D and Z represent the number of feature map channels of the last two layers, F z (x,y) represents the zth feature map of the second-to-last layer, G n (x,y) represents the nth feature map of the last layer, M(x,y) is F z (x,y) and G n (x,y) is the channel-wise summation value, g(·) is the bilinear interpolation method for M(x,y), S(x,y) is represented as a saliency map, x represents the row of the two-dimensional matrix, and y represents the column of the two-dimensional matrix.

[0051] S22, selecting the saliency map according to the adaptive threshold to obtain a mask matrix, and mapping the mask matrix to the corresponding image to be classified to obtain the focus part;

[0052] In the embodiment of the present invention, a hyperparameter α can be set first, and the adaptive threshold θ can be calculated based on the average value of the saliency map and the hyperparameter. The formula of the adaptive threshold θ is as follows:

[0053] θ=(1-α)·avg(S(x,y))

[0054] Where α represents the hyperparameter of attention to the key region, and avg(S(x,y)) represents the average value on the saliency map S(x,y).

[0055] In the embodiment of the present invention, the above-mentioned attention refers to the degree of attention paid to the key area. A larger value can be understood as a greater degree of magnification of the detected object, so 0<α<1.

[0056] In this embodiment, the adaptive threshold θ is used to select the saliency map S(x,y) to obtain the mask matrix Mask(i,j), and the mask matrix is mapped to the original image to be classified to obtain the focus part. The formula for obtaining the mask matrix Mask(i,j) is:

[0057]

[0058] Where i represents the row coordinate of the mask matrix, j represents the column coordinate of the mask matrix, and when the adaptive threshold θ is greater than the corresponding matrix element S(i, j) in the saliency map, the corresponding matrix element Mask(i, j) in the mask matrix is set to 1, otherwise it is set to 0.

[0059] S23. Perform bilinear interpolation upsampling on the focus portion to obtain a focus image of the same size as the image to be classified, that is, a key region subimage of the image to be classified.

[0060] In the embodiment of the present invention, bilinear interpolation upsampling is performed on the portion of interest determined by the mask matrix to obtain an image of interest of the same size as the input image to be classified; the image of interest is cut into N*N sub-images, and the formula for cutting the image of interest is:

[0061]

[0062] Where i represents the row coordinate of the mask matrix, j represents the column coordinate of the mask matrix, ψ(I) represents the attention image after the attention part is upsampled, N is a constant, and f c (·) represents a cutting method, that is, cutting the image ψ(I) into N*N sub-images, I sub Represented as a set of cut sub-images, where IS n Represented as the nth sub-image.

[0063] By using the localization subnet to obtain the key positioning area and obtain a saliency map, and using the saliency map to crop the target, the network can reduce the impact of environmental factors other than the target itself. At the same time, because fine-grained classification often distinguishes points in very detailed parts of the image, the upsampling process forces the network to focus on the details, thereby achieving the effect of improving the resolution accuracy.

[0064] S3, performing multi-scale cutting and filling on the key area sub-image, and randomly exchanging each image block to obtain multiple groups of cutting and filling sub-images of different scales;

[0065] In an embodiment of the present invention, it is necessary to cut and fill the key area sub-image, i.e., the key image, at different scales. At each scale, it is necessary to randomly exchange the image blocks corresponding to the scale to obtain cut and filled sub-images of different scales.

[0066] Specifically, in some embodiments of the present invention, taking four scales as an example, that is, in addition to the scale {1} of the original image to be classified, three other scales are required, assuming {2, 4, 8}. Then the cutting and filling process of the present invention may include:

[0067] S31: Cut the image of interest into N*N sub-images; where N={2, 4, 8};

[0068] S32: Fill each sub-image at each cutting scale with 0 to obtain the filled sub-image IP n , the filled sub-image set is I pad {IP n |0≤n <N 2};

[0069] For example, taking N=4, the image of interest can be cut into 4×4=16 sub-images. Each sub-image is actually an image block. These 16 sub-images are filled with 0, that is, 0 is filled around each sub-image to separate different image blocks, making it easier to focus on the fine-grained information of the image blocks.

[0070] S33: randomly splicing the zero-filled sub-images at each cutting scale into a new image to be classified according to the spatial position of the image to be classified;

[0071] In this embodiment, the zero-padded sub-images at each of the above-mentioned cutting scales can be randomly spliced into a new image to be classified according to the spatial position of the original image to be classified. Taking N=4 as an example, the 16 image blocks are randomly scrambled and the image to be classified is re-spliced according to the spatial position (4×4) of the original image to be classified. It can be found that since the sub-images are padded with zeros, each image block is expanded, and the combined size of the 16 image blocks will be larger than the original image to be classified.

[0072] S34: down-sampling the new image to be classified at each cutting scale to obtain a cutting and filling sub-image with the same size as the original image to be classified;

[0073] The re-joined new image to be classified is downsampled to remove redundant combined sizes so that the new image to be classified is the same size as the original image to be classified.

[0074] In the embodiment of the present invention, the image of interest is cut into N*N sub-images, and the formula for cutting the image of interest is:

[0075]

[0076] Where i represents the row coordinate of the mask matrix, j represents the column coordinate of the mask matrix, ψ(I) represents the attention image after the attention part is upsampled, N is a constant, and f c (·) represents a cutting method, that is, cutting the image ψ(I) into N*N sub-images, I sub Represented as a set of cut sub-images, where IS n Represented as the nth sub-image.

[0077] In the embodiment of the present invention, the sub-image set I sub Each sub-image of is padded with 0 of size P, and then the padded images are spliced according to the original spatial position and downsampled to the same size as the original image; the formula for zero padding of sub-images is:

[0078] IP n =f p (IS n ,P)

[0079] The formula for image stitching is:

[0080]

[0081] Among them, f p (·) represents the sub-image IS n Fill with 0 of size P, I pad {IP n |0≤n <N 2 ,N={2,4,8,…}} represents the set of filled sub-images, IP n represents the image after the nth sub-image is filled, f s (·) indicates that all padded sub-images are concatenated into a new image of the same size as the original image, which is named the padded image.

[0082] S4. Position-encode the image blocks in each group of cut and fill sub-images, and connect the corresponding position-encoded feature maps with the cut and fill sub-images according to the channels.

[0083] In an embodiment of the present invention, position encoding of image blocks in each group of cut and filled sub-images includes inputting a sequence of image blocks into a sine-cosine position coding function PSC to obtain position encoding of corresponding image blocks; and quantizing the position encoding according to the filling granularity and the cutting granularity of the cut and filled sub-image to obtain a position encoding feature map.

[0084] The steps of embedding position encoding include:

[0085] S41, inputting the patch sequence into the sine-cosine position coding function PSC respectively to obtain the position coding of the corresponding patch;

[0086] P em =PSC(idx)

[0087] The PSC can be expressed as:

[0088]

[0089] Where P represents the position code, dim represents the current input image dimension, ω represents the angular frequency, and t represents the position sequence. The angular frequency ω can be expressed as:

[0090]

[0091] S42, the generated position encoding feature map; wherein the position encoding feature map RE can be expressed as:

[0092]

[0093] Among them, RE xy [i][j] represents the position code of the pixel coordinate (i, j) in the (x, y)th image block; x is the horizontal coordinate of the image block in the cut-fill sub-image, y is the vertical coordinate of the image block in the cut-fill sub-image; (x, y) represents the arrangement position of the image block in the cut-fill sub-image, i is the horizontal coordinate of the pixel point of the image block, j is the vertical coordinate of the pixel point of the image block, and (i, j) represents the arrangement position of the pixel point in the image block; P xy Represents the position encoding of the (x,y)th image block, cut is the cutting granularity, pad is the filling granularity, where x, y satisfy: {0≤x,y≤cut-1,x,y∈N *}, N is the side length of each image patch, which can be expressed as:

[0094]

[0095] Where width is the side length of the image.

[0096] S43. Connect the position encoding map RE according to the channel and the cut and filled sub-map to obtain a feature map with position encoding:

[0097] R=concat c (RE,Rm)

[0098] Among them, concat c Represents feature connection, m represents scale, m = 1, 2, 4, 8….

[0099] The present invention disrupts the image pixel blocks and uses zero-interval pixel blocks to destroy the integrity of the image, reduce the connection between pixel blocks, and force the network to pay more attention to local areas rather than the whole. However, disrupting the pixel blocks will cause distortion of the original image, so pixel-level position coding is embedded to ensure that the original position information of the image is always maintained during the transmission process.

[0100] S5. Inputting the cut and filled sub-images of different scales into the first classification model of the preset training sub-network in turn to obtain the probability value of the corresponding category;

[0101] In an embodiment of the present invention, the first classification model is mainly used to classify cut and filled sub-images of different scales. The first classification model is mainly composed of some residual convolution modules. The cut and filled sub-images in the present invention can be respectively input into different residual convolution modules to obtain weighted feature maps of different levels; all weighted feature maps except the last level are downsampled, and the downsampled weighted feature maps are spliced and fused with the weighted feature maps of the next level; and the weighted feature maps of the first level and other spliced and fused feature maps are used as multi-level feature maps, and are respectively passed through linear layers and softmax classifiers to obtain probability values of corresponding categories.

[0102] In a preferred embodiment of the present invention, the first classification model of this embodiment adopts the resnet50_conv_x model, wherein resnet50_conv_x includes resnet50_conv_3 and resnet50_conv_4, which respectively represent the first three convolution blocks and the first four convolution blocks of the resnet50 network.

[0103] Taking the first classification model in the above preferred embodiment as an example, in this embodiment, the process of inputting cut and filled sub-images of different scales into the preset first classification model to obtain the probability value of the corresponding category may include:

[0104] S51, input the cut and filled sub-image R8 in S3 into resnet50_conv_3 to obtain the weighted feature map F1, input the cut and filled sub-image R4 in S3 into resnet50_conv_4 to obtain the weighted feature map F2, and input the cut and filled sub-image R2 in S3 into resnet50_conv_4 to obtain the weighted feature map F3;

[0105] S52, reduce the size of the feature maps F1 and F2 by half through a convolution operation with a convolution kernel of 3*3 and a step size of 2*2 to obtain downsampled feature maps F1_ds and F2_ds;

[0106] S53, leaving the F1 feature unchanged, concatenate and fuse the feature maps F1_ds and F2, as well as F2_ds and F3 through the channel to obtain fused feature maps F1_F2 and F2_F3;

[0107] S54. Pass the feature vectors F1, F1_F2, and F2_F3 in S43 through the linear layer and the softmax classifier respectively to obtain the probability values of the corresponding categories.

[0108] In order to meet the requirements of the value range here, it is necessary to use the Relu activation function formula in the last layer of the fully connected network:

[0109] Relu=max(0,x)

[0110] In an embodiment of the present invention, the preset first classification model can be a first classification model that has been trained or a first classification model that is being trained. The preset first classification model can be obtained after training, and its training process includes calculating the loss value of the output category value and the true label through the cross entropy loss function, calculating its gradient, and performing back propagation, and continuously updating the parameters through the gradient descent method.

[0111] Among them, the gradient descent algorithm should be used for training, and the loss function uses the logistic regression loss function formula:

[0112]

[0113] n is the number of images in a batch, y is the correct label, is the output of the network, and R is the ReLU function.

[0114] S6. Input the probability values of the corresponding categories of the cut and filled sub-images of different scales into the second classification model of the preset training sub-network, and obtain the fine-grained classification results of the image to be classified through weighted averaging.

[0115] In an embodiment of the present invention, the key area sub-image is input into the residual convolution module, and the feature maps of the key area sub-image are respectively fused with the multi-level feature maps to form a multi-level feature vector, and the above feature vectors are layered bilinear pooled in pairs to obtain a multi-layer feature fusion feature vector; the feature vectors are respectively passed through a linear layer and a softmax classifier to obtain the probability value of the corresponding category.

[0116] In a preferred embodiment of the present invention, the second classification model of this embodiment adopts the resnet50_conv_x model, wherein resnet50_conv_x includes resnet50_conv_5, which represents the first five convolution blocks of the resnet50 network.

[0117] Taking the second classification model in the above preferred embodiment as an example, in this embodiment, the probability values of the corresponding categories of the cut and filled sub-images of different scales are input into the preset second classification model. After weighted averaging, the process of obtaining the fine-grained classification results of the image to be classified may include:

[0118] S61, input the original image R in S3 into resnet50_conv_5, and obtain multiple sets of feature vectors F through feature fusion a ,F b ,F c ;

[0119] S62, set the parameters α, β, and γ to be F a 、F b 、F c The feature weight parameter is . The above feature vectors are layered bilinear pooled in pairs to obtain the multi-layer feature fusion feature vector F zbp ;

[0120] The hierarchical bilinear pooling structure is:

[0121] Z HBP =HBP(F a ,F b ,F c ,…)

[0122] Among them, HBP (F a ,F b ,F c ) is calculated as follows:

[0123] HBP(F a ,F b ,F c )=P T concat(α*U T F a *V T F b ,β*U T F a *S T F c ,γ*V T F b *S T F c )

[0124] Where P is the classification matrix, U, V, S are the convolution features F a ,F b ,F c The projection matrix of F a 、F b 、F c The feature weight parameter.

[0125] S63, the eigenvector F zbp The probability values of the corresponding categories are obtained through the linear layer and softmax classifier respectively.

[0126] In order to meet the requirements of the value range here, it is necessary to use the Relu activation function formula in the last layer of the fully connected network:

[0127] Relu=max(0,x)

[0128] In an embodiment of the present invention, the preset first classification model can be a first classification model that has been trained or a first classification model that is being trained. The preset first classification model can be obtained after training, and its training process includes calculating the loss value of the output category value and the true label through the cross entropy loss function, calculating its gradient, and performing back propagation, and continuously updating the parameters through the gradient descent method.

[0129] Among them, the gradient descent algorithm should be used for training, and the loss function uses the logistic regression loss function formula:

[0130]

[0131] n is the number of images in a batch, y is the correct label, is the output of the network, and R is the ReLU function.

[0132] In a preferred embodiment of the present invention, the first classification model and the second classification model of the present invention can adopt the same resnet50_conv_x model, and the two classification models can share parameters. It can be found that the model accuracy of the first classification model affects the model accuracy of the second classification model, that is, the classification results of cut and filled sub-images of different scales will affect the fine-grained classification results of the image to be classified. Therefore, the present invention can also use scale-related weights for back propagation, which can not only learn fine-grained information related to scale, but also greatly reduce the problems faced in image classification such as uneven image data quality and too little data.

[0133] Figure 2 This is the network structure diagram of the present invention, as shown in Figure 2As shown, in an embodiment of the present invention, the first step is to input the original image into the network to obtain a feature map. The second step is to input the feature map into the LCU module, locate the key areas of the image, cut and upsample. The third step is to crop and fill the image at multiple scales and perform position encoding. The fourth step is to embed the position codes in the cropped and filled images respectively. The fifth step is to input the multi-scale correspondences into the convolution layer for training to obtain the loss value, and back propagate to update the network parameters. The sixth step is to fuse the feature vectors of different layers, input them into the softmax layer to obtain the predicted value, compare them with the true category label to calculate the loss value, and back propagate to continue to update the network parameters. Finally, in the testing process, you only need to input the original image into the trained network to obtain the feature vector, and input it into the softmax layer to obtain the final predicted category.

[0134] Figure 3 is the feature fusion diagram of the present invention, such as Figure 3 As shown, the present invention first downsamples the feature map with a channel of 1024 and a length and width of 56 output by stage3 to 1024×28×28 through a convolution layer with a convolution kernel of 3×3 and a step size of 2, and uses it as the output P1 of stage3. At the same time, the feature vector output by P1 and stage4 is fused according to the channel to obtain a fused feature vector of size 2048×28×28. At the same time, the convolution layer with a convolution kernel of 3×3 and a step size of 2 is used to downsample the fused feature map to 2048×14×14, and uses this vector as the output P2 of stage4. Finally, the feature vector output by P2 and stage5 is fused according to the channel to obtain a fused feature vector of size 3072×14×14. At the same time, the convolution layer with a convolution kernel of 3×3 and a step size of 2 is used to downsample the fused feature map to 2048×7×7, and uses this vector as the output P3 of stage5.

[0135] In the description of the present invention, it should be understood that the terms "coaxial", "bottom", "one end", "top", "middle", "the other end", "upper", "one side", "top", "inside", "outside", "front", "center", "both ends", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0136] In the present invention, unless otherwise clearly stipulated and limited, the terms "installation", "setting", "connection", "fixation", "rotation" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0137] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A fine-grained image classification method based on a multi-layer coordinated convolutional neural network, characterized in that: The method steps include: Obtain an image data set and preprocess the images to be classified in the image data set; A convolutional neural network is used to extract the image features of the image to be classified, and the positioning subnet is used to obtain the positioning key area to obtain the key area subgraph of the image to be classified; Performing multi-scale cutting and filling on the key area sub-images, and randomly exchanging each image block to obtain multiple groups of cutting and filling sub-images of different scales; Position encoding is performed on the image blocks in each group of cut and fill sub-images, and the corresponding position encoding feature maps are connected to the cut and fill sub-images according to the channels. Specifically, the following steps are performed: Input the sequence of image blocks into the sine and cosine position coding function PSC respectively to obtain the position coding of the corresponding image blocks: According to the filling granularity and cutting granularity of the cut and fill sub-image, the position code is quantized to obtain a position code feature map; Connect the position encoding map RE according to the channel and cut filling sub-map to obtain the feature map with position encoding: Among them, the obtained position encoding feature map RE is expressed as: Among them, RE xy [i][j] represents the position code of the pixel coordinate (i, j) in the (x, y)th image block; x is the horizontal coordinate of the image block in the cut-fill sub-image, y is the vertical coordinate of the image block in the cut-fill sub-image; i is the horizontal coordinate of the pixel point of the image block, j is the vertical coordinate of the pixel point of the image block, P xy Represents the position code of the (x,y)th image block, cut is the cutting granularity, pad is the filling granularity, and {0≤x,y≤cut-1,x,y∈N * }, N is the side length of each image block, expressed as: Width is the side length of the image; The cut and filled sub-images of different scales are input into the first classification model of the preset training sub-network in turn to obtain the probability value of the corresponding category; The probability values of the corresponding categories of the cut and filled sub-images of different scales are input into the second classification model of the preset training sub-network. After weighted averaging, the fine-grained classification results of the image to be classified are obtained.

2. The fine-grained image classification method based on a multi-layer coordinated convolutional neural network according to claim 1 is characterized in that: The method of using the positioning subnet to obtain the positioning key area to obtain the key area subgraph of the image to be classified includes: The extracted image features are summed according to the channels, and the resulting summed feature map is bilinearly upsampled to obtain a saliency map of the same size as the image to be classified; The saliency map is selected according to the adaptive threshold to obtain a mask matrix, and the mask matrix is mapped to the corresponding image to be classified to obtain the focus part; The attention part is upsampled by bilinear interpolation to obtain an attention image of the same size as the image to be classified, that is, a key area subimage of the image to be classified.

3. The fine-grained image classification method based on a multi-layer coordinated convolutional neural network according to claim 2, characterized in that: The selection of the saliency map according to the adaptive threshold includes calculating the adaptive threshold based on the average value of the saliency map and the hyperparameter, and determining the mask matrix based on the size relationship between the adaptive threshold and the corresponding matrix element in the saliency map, that is, when the adaptive threshold is greater than the corresponding matrix element in the saliency map, the corresponding matrix element in the mask matrix is set to 1, otherwise it is set to 0.

4. The fine-grained image classification method based on a multi-layer coordinated convolutional neural network according to claim 3 is characterized in that: The calculation formula of the adaptive threshold is expressed as: θ=(1-α)·avg(S(x,y)) Where θ represents the adaptive threshold; α represents the hyperparameter of attention to the key area, avg(S(x,y)) represents the average value on the saliency map S(x,y), and (x,y) represents the matrix element.

5. The fine-grained image classification method based on a multi-layer coordinated convolutional neural network according to claim 1, characterized in that: The step of performing multi-scale cutting and filling on the key region sub-images and randomly exchanging image blocks to obtain multiple groups of cutting and filling sub-images of different scales includes: Cut the key area sub-images into different N×N sub-images; Fill each sub-image at each cutting scale with 0 to obtain the filled sub-image IP n , the filled sub-image set is I pad {IP n |0≤n <N 2 }; According to the spatial position of the image to be classified, the zero-filled sub-images at each cutting scale are randomly spliced into a new image to be classified; Downsample the new image to be classified at each cutting scale to obtain a cutting and filling sub-image of the same size as the original image to be classified; Among them, IP n represents the image after filling the nth sub-image, n represents the sub-image number of the key area sub-image, that is, the sub-image of the focus image, and N represents the side length of the image block; I pad {·} represents the set of padded sub-images.

6. The fine-grained image classification method based on a multi-layer coordinated convolutional neural network according to claim 1, characterized in that: The method sequentially inputs the cut and filled sub-images of different scales into the first classification model of the preset training subnet to obtain the probability value of the corresponding category, including inputting each cut and filled sub-image into different residual convolution modules to obtain weighted feature maps of different levels; downsampling all weighted feature maps except the last level, splicing and fusing the downsampled weighted feature maps with the weighted feature maps of the next level; and using the weighted feature map of the first level and the other spliced and fused feature maps as multi-level feature maps, respectively passing through a linear layer and a softmax classifier to obtain the probability value of the corresponding category.

7. The fine-grained image classification method based on a multi-layer coordinated convolutional neural network according to claim 6, characterized in that: The method includes inputting the probability values of the corresponding categories of the cut and filled sub-images of different scales into the second classification model of the preset training subnet, and obtaining the fine-grained classification results of the image to be classified after weighted averaging, which includes inputting the key area sub-image into the residual convolution module, and fusing the feature maps of the key area sub-image with the multi-level feature maps to form a multi-level feature vector, and performing hierarchical bilinear pooling on the above feature vectors to obtain a multi-layer feature fusion feature vector; The feature vector is passed through the linear layer and the softmax classifier respectively to obtain the probability value of the corresponding category.

8. The fine-grained image classification method based on a multi-layer coordinated convolutional neural network according to claim 1, characterized in that: The preset first classification model and the preset second classification model are obtained after training respectively, and the training process includes calculating the loss value of the output category value and the true label through the cross entropy loss function respectively, calculating its gradient, and performing back propagation, and continuously updating the parameters through the gradient descent method.

Images