Fine-grained image classification method based on spatially symmetric irregular local region feature extraction

Abstract

The present invention discloses a fine-grained image classification method based on spatially symmetric irregular local region feature extraction, comprising: inputting an image into a deep convolutional network to obtain a feature map; inputting the feature map into a symmetric local region extraction module to extract local region features; utilizing an attention mechanism to capture the contextual relationship between features; utilizing a bidirectional recurrent neural network to fuse spatial structure information with feature information; clustering all features using a number of cluster centers, and combining all cluster centers for classification to obtain target categories. This method overcomes the problem of spatial information loss between features during local feature extraction. At the same time, the proposed irregular region feature extraction template can effectively extract the key irregular local region features of the target, retaining the integrity of local information and reducing irrelevant information to the greatest extent, thereby improving the accuracy of fine-grained image classification.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a fine-grained image classification method based on spatially symmetric irregular local region feature extraction. Background Art

[0002] Fine-grained image classification is a significant development in image classification. Compared to general-purpose image classification, it offers greater practical application value, such as in smart retail, intelligent transportation, and biodiversity monitoring. However, it is also more challenging. General-purpose image classification distinguishes objects of different categories, which vary significantly and can be distinguished based on features such as appearance, shape, and color. Therefore, the model emphasizes global features. Fine-grained image classification, on the other hand, builds on general-purpose classification by further subdividing categories. Since the task is to distinguish between different subcategories of the same type of object, it utilizes more subtle local features. Compared to general-purpose image classification, the difficulty lies in the fact that fine-grained images have smaller inter-class differences and larger intra-class differences.

[0003] In recent years, many methods for fine-grained image classification have been based on positioning. Using additional annotation information, they first train the model to recognize key parts of an object, then extract features from these key areas, and then perform fine-grained classification based on the features of these key areas. In practical applications, these additional labels often require experts to spend a considerable amount of time and are expensive to label. Failure to use additional labeled boxes makes it difficult for the model to find discriminative parts, and the local features found may not be useful, making classification more difficult. Furthermore, since most rectangular candidate boxes often extract incomplete local information or contain a large amount of background information, the extracted discriminative parts are inaccurate for classification. Summary of the Invention

[0004] The purpose of the present invention is to provide a fine-grained image classification method based on spatially symmetric irregular local region feature extraction.

[0005] The purpose of the present invention can be achieved by adopting the following technical solutions:

[0006] A fine-grained image classification method based on spatially symmetric irregular local region feature extraction includes the following steps:

[0007] Input the image into the deep convolutional network to obtain the feature map of the image;

[0008] The feature map is input into the symmetric local region extraction module for local region feature extraction. The module extracts features by using an irregular region feature extraction template to obtain a local feature set.

[0009] Use the attention mechanism to capture the contextual relationship between features, highlight important local features, and suppress other local features;

[0010] A bidirectional recurrent network is used to fuse spatial structure information with feature information to obtain a feature set with richer semantic information.

[0011] All features are represented by cluster centers in a clustering manner to remove redundant information, and all cluster centers are combined for classification to obtain the target category.

[0012] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the fine-grained image classification method based on spatially symmetric irregular local area feature extraction is implemented.

[0013] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned fine-grained image classification method based on spatially symmetric irregular local area feature extraction.

[0014] Compared with the existing technology, the present invention has the following significant advantages: (1) by extracting local area features through spatial symmetry, the spatial logical relationship between these local features can be retained to the greatest extent while extracting local features, and a bidirectional recurrent neural network is subsequently used to fuse spatial information and feature information; (2) by using regular rectangular candidate boxes, geometric transformation is performed on them to obtain irregular candidate boxes of equal area to simulate the shapes of those irregular local areas as much as possible, so that the useful information extracted for the target local features is as complete as possible, while the background information is minimized; (3) for the extracted feature set, a clustering method is used to represent all features with several cluster centers, which effectively removes redundant features and obtains a better image representation.

[0015] The fine-grained image classification method based on spatially symmetric irregular local region feature extraction provided by the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The flowchart of the fine-grained image classification method based on spatially symmetric irregular local region feature extraction according to the present invention.

[0017] Figure 2 FIG. 4 is a diagram showing the working principle of the synthetic local area extraction module according to the present invention.

[0018] Figure 3 It is a regularized local area feature extraction template according to the present invention.

[0019] Figure 4A template is provided for extracting features of irregular local regions according to the present invention. DETAILED DESCRIPTION

[0020] The present invention proposes a fine-grained image classification method based on spatially symmetric irregular local area feature extraction, designs a symmetric local area feature extraction module, preserves the spatial information between features while extracting local features, and proposes an irregular local area feature extraction template, which can fully extract complete local information and include less background information. After using the attention mechanism to capture the contextual relationship between features, a bidirectional recurrent neural network is used to fuse the spatial relationship information between features with the feature information. Finally, clustering is used to remove redundant information to obtain a relatively complete and effective image representation for classification, which has good application prospects.

[0021] In order to make the technical solution of the present invention more clear and specific to those skilled in the art, the present invention is described in further detail below with reference to embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0022] Combine Figure 1 , a fine-grained image classification method based on spatially symmetric irregular local region feature extraction, the specific steps are as follows:

[0023] Step 1: Use a deep convolutional neural network to calculate the feature map of the input image:

[0024] (1.1) For any given size image, image preprocessing is performed: the image is scaled to the specified size using bilinear interpolation, and then each pixel is normalized;

[0025] (1.2) The preprocessed image is input into the deep convolutional network, and features are extracted through multiple convolution operations to obtain the feature map F of the image.

[0026] Step 2: Input the feature map into the symmetric local region extraction module to extract local region features, such as Figures 2 to 4 As shown in the figure, the module extracts features by using the irregular region feature extraction template to obtain a local feature set:

[0027] (2.1) For a given regular rectangular candidate box, perform geometric transformation on it to obtain a set of candidate boxes with equal area but irregular shape:

[0028] R_list=[R1,R2,......,Rn]

[0029] I_list=[I1,I2,......,In]

[0030] Ii=T(Ri)

[0031] S(Ii)=S(Ri)

[0032] Where Ri represents a regular candidate box, Ii represents an irregular candidate box, T() represents shifting Ri left or right or up or down by row or column, S() represents the area of ​​the candidate box, and i = 1, 2, ..., n.

[0033] (2.2) Use these irregular candidate boxes to extract local features at different positions in the feature map, and then symmetric these features about the center according to their spatial positions on the original image. As the number of features increases, the range of these features gradually increases:

[0034] f_list=[f1, f2, f3,..., fn]

[0035]

[0036] A(f1)∩…∩A(f n-i )+ΔA=A(f1)∩…∩A(f n-i+1 )

[0037] Where fi represents the i-th local feature, A(fi) represents the candidate box of the i-th local feature, represents the central symmetric relationship, and ΔA represents the increment of the area adjacent to the known set but not belonging to the set.

[0038] Step 3: Use the attention mechanism to capture the contextual relationship between features, highlight important local features, and suppress other local features:

[0039] (3.1) The local features of various shapes in the above set are scaled to the specified size using the bilinear interpolation function:

[0040] f=Bilinear(f)

[0041] (3.2) Use the attention mechanism to capture contextual relationships. Specifically, the formula for self-attention is:

[0042] First calculate the similarity between features t and t'. Here, we use the method of first fully connecting the two features and then adding them together to obtain the similarity vector g. t,t' :

[0043] g t,t' =tanh(W g f t +W g' f t' +b g )

[0044] Then the obtained vector g t,t' Calculate the attention value α in the input full connectiont,t' :

[0045] α t,t' =σ(W a g t,t' +b a )

[0046] Use these attention values ​​to perform a weighted summation on the entire feature set to obtain a new feature set I of equal size:

[0047]

[0048] Where W represents the matrix to be learned, b represents the bias to be learned, and σ represents the activation function.

[0049] Step 4: Use a bidirectional recurrent neural network to fuse the spatial structure information with the feature information to obtain a feature set with richer semantic information:

[0050] The newly generated feature h t By the characteristics (I1, I2, ..., I t ) are generated together, specifically:

[0051] According to the input I t and the previous network status h t-1 To calculate the input information ratio i t :

[0052] i t =σ(W i ·[h t-1 ,I t ]+b i )

[0053] According to the input I t and the previous network status h t-1 To calculate the backup information for this update

[0054]

[0055] According to the input I t ,i t , and the previous network variable C t-1 , to update the network variable C t :

[0056]

[0057] According to the input I t and the previous network status h t-1 To calculate the information discard ratio o t :

[0058] o t =σ(W o [h t-1 , I t ]+b o )

[0059] By o t and network variable C t To update the network status h t :

[0060] h t =o t *tanh(C t )

[0061] Where W represents the matrix to be learned, b represents the bias to be learned, and σ represents the activation function.

[0062] Step 5: Cluster all features using cluster centers to represent them, thereby removing redundant information, and combine all cluster centers for classification to obtain the target category:

[0063] (5.1) Calculate the distance between each feature and the cluster center, and assign the feature to each cluster center according to the distance ratio:

[0064]

[0065] Among them, h i represents the eigenvector, c k represents the cluster center;

[0066] (5.2) According to the obtained ratio, each feature is assigned to each cluster center according to the ratio:

[0067]

[0068] (5.3) Use these cluster centers to represent the entire image, and then use the multi-layer perceptron to classify it:

[0069] predict = MLP(V)

[0070] classid=argmax(predict)

[0071] Where classid represents the image category, MLP represents a multi-layer perceptron composed of full connections, and argmax represents the index of the maximum value.

[0072] The present invention designs a symmetric local area extraction module to overcome the problem of spatial information loss between features during local feature extraction. At the same time, the proposed irregular area feature extraction template can effectively extract the irregular key local area features of the target, retain the integrity of local information and reduce the content of irrelevant information to the greatest extent, and can improve the accuracy of fine-grained image classification.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A fine-grained image classification method based on spatially symmetric irregular local region feature extraction, characterized in that: The following steps are involved: Input the image into the deep convolutional network to obtain the feature map of the image; The feature map is input into the symmetric local region extraction module for local region feature extraction. The module extracts features by using the irregular region feature extraction template to obtain a local feature set, as follows: For a given regular rectangular candidate box, perform geometric transformation on it to obtain a set of candidate boxes with equal area but irregular shape: R_list=[R1,R2,......,Rn] I_list=[I1,I2,......,In] Ii=T(Ri) S(Ii)=S(Ri) Where Ri represents a regular candidate box, Ii represents an irregular candidate box, T() represents shifting Ri left or right or up or down by row or column, S() represents the area of ​​the candidate box; i = 1, 2, ..., n; Use these irregular candidate boxes to extract local features at different positions in the feature map, and then symmetric these features about the center according to their spatial positions on the original image. As the number of features increases, the range of these features gradually increases: f_list=[f1,f2,f3,……,f n ] A(f1)∩…∩A(f n-i )+ΔA=A(f1)∩…∩A(f n-i+1 ) Where fi represents the i-th local feature, A(fi) represents the candidate box of the i-th local feature, represents the central symmetric relationship, ΔA represents the increment of the area adjacent to the known set but not belonging to the set; The attention mechanism is used to capture the contextual relationship between features, highlight important local features, and suppress other local features. The details are as follows: The local features of various shapes in the above set are scaled to the specified size using the bilinear interpolation function: f=Bilinear(f) Use the attention mechanism to capture contextual relationships, specifically: First calculate the similarity between features t and t', and use the method of first fully connecting the two features and then adding them together to obtain the similarity vector g t,t' : g t,t' =tanh(W g f t +W g' f t' +b g ) Then the obtained vector g t,t' Calculate the attention value α in the input full connection t,t' : a t,t' =σ(W a g t,t' +b a ) The corresponding attention value is used to perform weighted summation on the entire feature set to obtain a new feature set I of equal size: Where W g 、W g' 、W a Represent the matrices that need to be learned, b g 、b a They represent the bias to be learned, and σ represents the activation function; The bidirectional recurrent network is used to fuse spatial structure information with feature information to obtain a feature set with richer semantic information, as follows: The newly generated feature h t By the characteristics (I1, I2, ..., I t ) is generated together with all the information; According to the input I t and the previous network status h t-1 To calculate the input information ratio i t : i t =σ(W i ·[h t-1 ,I t ]+b i ) According to the input I t and the previous network status h t-1 To calculate the backup information for this update According to the input I t ,i t , and the previous network variable C t-1 , to update the network variable C t : According to the input I t and the previous network status h t-1 To calculate the information discard ratio o t : the t =σ(W o [h t-1 ,I t ]+b o ) By o t and network variable C t To update the network status h t : h t =o t *tanh(C t ) Where W i 、W c 、W o Represent the matrix to be learned, b i 、b c 、b o Represent the bias to be learned, σ represents the activation function; All features are represented by cluster centers in a clustering manner to remove redundant information, and all cluster centers are combined for classification to obtain the target category.

2. The fine-grained image classification method based on spatially symmetric irregular local region feature extraction according to claim 1 is characterized in that: The image is input into the deep convolutional network to obtain the feature map of the image, as follows: For any given size image, image preprocessing is first performed: bilinear interpolation is used to scale the image to the specified size, and then each pixel is normalized; The preprocessed image is input into the deep convolutional network, and feature extraction is performed after multiple convolution operations to obtain the feature map of the image.

3. The fine-grained image classification method based on spatially symmetric irregular local region feature extraction according to claim 1 is characterized in that: The clustering method is to represent all features with cluster centers, thereby removing redundant information, and classifying all cluster centers to obtain the target category, as follows: Calculate the distance between each feature and the cluster center, and assign the feature to each cluster center according to the distance ratio: Among them, h i represents the eigenvector, c k represents the cluster center; According to the obtained ratio, each feature is assigned to each cluster center according to the ratio: Use these cluster centers to represent the entire image, and then use the multi-layer perceptron for classification: predict = MLP(V) classid=argmax(predict) Where classid represents the image category, MLP represents a multi-layer perceptron composed of full connections, and argmax represents the index of the maximum value.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the fine-grained image classification method based on spatially symmetric irregular local area feature extraction as described in any one of claims 1 to 3 is implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the fine-grained image classification method based on spatially symmetric irregular local region feature extraction as described in any one of claims 1 to 3 is implemented.

Images