A small sample fine-grained image classification method based on spatial-frequency domain hierarchical learning

Abstract

The present invention relates to the field of image processing technology, and more particularly to a small-sample, fine-grained image classification method based on spatial-frequency domain hierarchical learning. The method comprises the following steps: obtaining an image of a plant disease to be identified; and inputting the image into a preset disease classification model to obtain a plant disease identification result. The disease classification model is configured to hierarchically fuse spatial and frequency domain features in the plant disease image to identify the plant disease. This method can alleviate the problem of insufficient generalization capability caused by excessive fine-grained learning in existing methods. It is particularly effective for disease data characterized by subtle, varying, and dispersed discriminative features. This method can be used to detect plant disease identification in actual production activities, reducing costs.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a small sample fine-grained image classification method based on space-frequency domain hierarchical learning. Background Art

[0002] Few-shot fine-grained image classification (FS-FGIC) refers to image classification tasks where the number of samples per category is very limited and the differences between categories are very subtle, making the classification task more challenging. FS-FGIC is often modeled using meta-learning, which leverages category prototypes in an embedding space to achieve rapid adaptation. This approach is particularly well-suited for scenarios where samples are scarce because it leverages prior knowledge and experience to accelerate the learning process for new tasks, thereby achieving higher accuracy and generalization in fine-grained image classification.

[0003] In fine-grained image classification tasks, due to the availability of sufficient data for learning, it is generally not a problem for the model to focus on learning high-level fine-grained features. However, in the FS-FGIC task, due to insufficient data, excessive focus on learning fine-grained features may cause the feature space to deviate from the original data distribution, thereby affecting the model's generalization ability during the testing phase. It is also worth noting that the FS-FGIC task is usually trained based on the CUB dataset. In the images of the CUB dataset, the super-class discriminant regions of most images are generally large and concentrated, while there are large inter-class differences between sub-classes. In contrast, in actual plant disease classification applications, the discriminative features of different types of leaf diseases are often of different sizes and are scattered, and the inter-class variance between sub-classes is further reduced. These characteristics make current plant disease recognition more difficult.

[0004] The shortcomings of the current FS-FGIC model are summarized as follows: (1) Due to insufficient data, excessive focus on learning fine-grained features may cause the feature space to deviate from the original data distribution, thereby affecting the generalization ability of the model during the testing phase; (2) It cannot effectively identify and extract subtle, uneven, and scattered discriminative features; (3) These difficult-to-identify discriminative features often have smaller inter-class variance and larger intra-class variance, increasing the difficulty of recognition. Therefore, to address the above technical issues and to obtain more accurate plant disease recognition results, the present invention proposes a small-sample fine-grained image classification method based on spatial-frequency domain hierarchical learning. Summary of the Invention

[0005] The purpose of this invention is to provide a small sample fine-grained image classification method based on spatial-frequency domain hierarchical learning, which can alleviate the problem of insufficient generalization ability caused by excessive fine-grained learning in existing methods and can be applied to real plant disease identification scenarios.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A small-sample fine-grained image classification method based on spatial-frequency domain hierarchical learning, including:

[0008] Acquire an image of a plant disease to be identified;

[0009] The plant disease image to be identified is input into a preset disease classification model to obtain a plant disease identification result, wherein the disease classification model is used to hierarchically fuse spatial domain features and frequency domain features in the plant disease image to identify plant diseases.

[0010] Optionally, the disease classification model includes a dual-frequency domain interaction enhancement module, a spatial-frequency domain hierarchical learning module, and a support set-query set interactive learning module. The dual-frequency domain interaction enhancement module is used to extract features from the input image and obtain the spatial domain features and frequency domain features of the input image; the spatial-frequency domain hierarchical learning module is used to enhance the spatial domain features and frequency domain features based on hierarchical learning to obtain enhanced spatial domain features and frequency domain features; the support set-query set interactive learning module is used to divide the enhanced spatial domain features and frequency domain features into support sets and query sets, and interactively learn the divided support sets and query sets, optimize the disease classification model based on the interactive learning results, and output the final classification results.

[0011] Optionally, the dual-frequency domain interaction enhancement module performs feature extraction on the input image, and obtaining spatial domain features and frequency domain features of the input image includes:

[0012] Inputting the RGB image into the neural network model to obtain the spatial domain features;

[0013] Performing color space conversion on the RGB image to obtain a YCbCr image, and performing block and DCT transformation on the YCbCr image to obtain a DCT block;

[0014] Reorganizing the DCT blocks to construct DCT frequency bands, and splitting the DCT frequency bands into high-frequency features and low-frequency features;

[0015] A self-attention operation is used to enhance the high-frequency features and the low-frequency features respectively, and then a splicing operation is performed to obtain a total feature. The self-attention operation is then used to enhance the total feature and then a Reshape operation is performed to obtain a frequency domain feature equal in size to the spatial domain feature.

[0016] Optionally, reorganizing the DCT blocks to construct DCT frequency bands includes:

[0017] The DCT blocks of the same frequency domain components are spliced, and the DCT blocks of different frequency domain components are sorted and reorganized from low frequency to high frequency to obtain the DCT frequency band.

[0018] Optionally, the spatial-frequency hierarchical learning module enhances the spatial features and the frequency features based on hierarchical learning, and obtaining the enhanced spatial features and the frequency features includes:

[0019] The spatial domain features and the frequency domain features are respectively scaled to obtain spatial domain features and frequency domain features at several levels;

[0020] Adaptive learning between spatial domain features and frequency domain features is performed at each level, and the adaptively learned spatial domain features and frequency domain features are adaptively fused according to the level from large to small to obtain the spatial domain features and frequency domain features after hierarchical learning;

[0021] Self-attention operations are performed on the spatial domain features and frequency domain features after hierarchical learning to obtain enhanced spatial domain features and frequency domain features.

[0022] Optionally, performing adaptive learning between spatial domain features and frequency domain features at each level includes:

[0023] The spatial domain features and frequency domain features of the same level are spliced to obtain spliced features, and the spliced features are subjected to average pooling and softmax function to obtain descriptors of each level;

[0024] Multiplying the descriptor with the spatial features at the corresponding level and the first learnable matrix to obtain the adaptively learned spatial features;

[0025] The descriptor is multiplied by the frequency domain features on the corresponding level and the second learnable matrix to obtain the frequency domain features after adaptive learning.

[0026] Optionally, the support set-query set interactive learning module divides the enhanced spatial domain features and frequency domain features into a support set and a query set, and interactively learns the divided support set and query set, including:

[0027] The enhanced spatial domain features and frequency domain features are divided into support set and query set respectively to obtain spatial domain support set, spatial domain query set, frequency domain support set, and frequency domain query set;

[0028] The spatial domain support set and the spatial domain query set are interactively learned with the frequency domain support set and the frequency domain query set respectively to obtain an interactive learning score, wherein the interactive learning score is obtained by calculating the Euclidean distance between the two parties performing the interactive learning.

[0029] Optionally, optimizing the disease classification model based on the interactive learning result includes: performing back-propagation optimization on the disease classification model based on the loss calculated based on the obtained interactive learning score.

[0030] The beneficial effects of the present invention are:

[0031] This invention uses frequency and spatial domain features to improve existing small-sample, fine-grained networks. This approach alleviates the problem of insufficient generalization caused by excessive fine-grained learning in existing methods. It enables the network model to address the difficulty of traditional networks in accurately classifying images with small, scattered discriminant regions, while only slightly increasing computational parameters. This significantly improves inference accuracy. This approach is particularly effective for plant disease data, which is characterized by subtle, variable, and scattered discriminant features. It can solve plant disease identification problems in actual production activities and reduce costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 A schematic diagram of the dual-frequency domain interaction enhancement module structure and workflow according to an embodiment of the present invention;

[0034] Figure 2 Schematic diagram of the spatial-frequency domain hierarchical learning module structure and workflow of an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of the support set-query set interactive learning module structure and workflow of an embodiment of the present invention;

[0036] Figure 4 Reconstructing an image effect diagram for the support set-query set interactive learning module of an embodiment of the present invention;

[0037] Figure 5 This is a heat map of the recognition results of the disease classification model of an embodiment of the present invention on a plant disease dataset. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] This embodiment provides a small-sample fine-grained image classification method based on spatial-frequency domain hierarchical learning, including:

[0041] Obtain a plant disease image to be identified, input the plant disease image to be identified into a preset disease classification model, and obtain a plant disease identification result, wherein the disease classification model is used to hierarchically fuse the spatial domain features and frequency domain features in the plant disease image to identify plant diseases.

[0042] The disease classification model includes a dual-frequency domain interactive enhancement module, a spatial-frequency domain hierarchical learning module, and a support set-query set interactive learning module. The dual-frequency domain interactive enhancement module is used to extract features from the input image and obtain the spatial and frequency domain features of the input image; the spatial-frequency domain hierarchical learning module is used to enhance the spatial and frequency domain features based on hierarchical learning to obtain enhanced spatial and frequency domain features; the support set-query set interactive learning module is used to divide the enhanced spatial and frequency domain features into support sets and query sets, and interactively learn the divided support sets and query sets. Based on the interactive learning results, the disease classification model is optimized and the final classification results are output.

[0043] Specifically, this embodiment uses frequency and spatial domain features to improve existing small-sample, fine-grained networks. This alleviates the problem of insufficient generalization caused by excessive fine-grained learning in existing methods. This allows the network model to address the difficulty of traditional networks in accurately classifying images with small, scattered discriminative regions, while only slightly increasing computational parameters. This significantly improves inference accuracy. This is particularly effective for disease data characterized by subtle, varying, and scattered discriminative features, enabling plant disease identification in real-world production processes and reducing costs.

[0044] Furthermore, the dual-frequency domain interaction enhancement module extracts features from the input image, and obtains spatial domain features and frequency domain features of the input image, including:

[0045] Input the RGB image into the neural network model to obtain spatial features;

[0046] Perform color space conversion on the RGB image to obtain a YCbCr image, and then perform DCT transformation on the YCbCr image to obtain a DCT block;

[0047] Reorganizing the DCT blocks to construct DCT frequency bands, and splitting the DCT frequency bands into high-frequency features and low-frequency features, wherein reorganizing the DCT blocks to construct the DCT frequency bands includes: splicing DCT blocks of the same frequency domain components, and reorganizing DCT blocks of different frequency domain components in order from low frequency to high frequency to obtain DCT frequency bands;

[0048] The self-attention operation is used to enhance the high-frequency features and low-frequency features respectively, and then the splicing operation is performed to obtain the total features. The self-attention operation is then used to enhance the total features and then the Reshape operation is performed to obtain the frequency domain features that are equal in size to the spatial domain features.

[0049] Specifically, the structure and workflow of the dual-frequency domain interaction enhancement module of this embodiment are as follows: Figure 1 As shown, data processing is performed to obtain the spatial domain feature representation and frequency domain feature representation of the image, including:

[0050] Step 1.1. Input the image into the Resnet12 / Conv4 network model to obtain the spatial feature F processed by the CNN neural network s , where the input must be divided into a support set and a query set, and the image color space must be RGB.

[0051] Specifically, Resnet12 contains four residual blocks, each with three convolutional layers. "12" indicates a total of 12 convolutional layers. After the first residual block, the image output becomes 64×42×42 (2×2 max pooling). After the second residual block, the output becomes 160×21×21. The third residual block outputs 320×10×10, and the fourth residual block outputs 640×5×5. After 5×5 average pooling, the output is 640×1×1. DropBlock is used to prevent overfitting, and the image is then flattened to 640 dimensions according to the channel. Conv4 means that the network contains 4 repeated convolution blocks. The four convolution blocks are identical. The image shape changes as follows: after the image passes through the first convolution block, 3×84×84 becomes 64×42×42 (2×2 maximum pooling); after passing through the second convolution block, it becomes 64×21×21; the third convolution block outputs 64×10×10; and the fourth convolution block outputs 64×5×5.

[0052] Step 1.2. Convert the original RGB image to YCbCr. YCbCr images contain brightness, blue, and red. The human eye is more sensitive to brightness than color, so using YCbCr representation is more convenient for subsequent processing.

[0053] Specifically, the calculation process of RGB→YCbCr is as follows:

[0054] Y=0.257*R+0.564*G+0.098*B+16

[0055] Cb=-0.148*R-0.291*G+0.439*B+128

[0056] Cr=0.439*R-0.368*G-0.071*B+128

[0057] Step 1.3. Divide the obtained YCbCr image into blocks of 8×8 pixels;

[0058] Step 1.4. Use a two-bit discrete cosine transform to convert the spatial domain information of the resulting 8×8 block to the frequency domain. The low-frequency information of the transformed block is concentrated in the upper left corner, and the high-frequency information is concentrated in the lower right corner.

[0059] Specifically, the DCT transform formula is as follows:

[0060] Let f(x,y) be an M×N digital image matrix:

[0061]

[0062] Where, x,u=0,1,2,…,M-1; y,v=0,1,2,…,N-1.

[0063] Step 1.5. Reassemble the DCT blocks by concatenating the same frequency domain components of each DCT block. Then, reassemble the blocks with different frequency domain components in the order of low frequency → high frequency to obtain DCT bands.

[0064] Step 1.6. Split the DCT band into two from the middle to obtain the high-frequency feature F h and low-frequency features F l ;

[0065] Step 1.7. Use self-attention operation to separately analyze high-frequency features F h and low-frequency features F l Perform enhancement processing;

[0066] Specifically, the self-attention calculation formula is as follows:

[0067] First, for F hand F l Make three copies, Q h , K h 、V h , Q l , K l and V l :

[0068]

[0069] Step 1.8. Process the high-frequency features F h and low-frequency features F l Splice them together to get the total feature F ALL ;

[0070] Step 1.9. Use the attention operation again on the total feature F f Perform enhancement processing;

[0071] Step 1.10. The processed total feature F ALL Perform the Reshape operation to obtain the spatial feature F s Frequency domain features F of equal size f .

[0072] Furthermore, the spatial-frequency hierarchical learning module enhances the spatial and frequency domain features based on hierarchical learning. The enhanced spatial and frequency domain features include:

[0073] The spatial domain features and frequency domain features are scaled and classified respectively to obtain several levels of spatial domain features and frequency domain features;

[0074] Adaptive learning between spatial domain features and frequency domain features is performed at each level, and the adaptively learned spatial domain features and frequency domain features are adaptively fused according to the level from large to small to obtain the spatial domain features and frequency domain features after hierarchical learning;

[0075] Self-attention operations are performed on the spatial domain features and frequency domain features after hierarchical learning to obtain enhanced spatial domain features and frequency domain features.

[0076] Among them, the adaptive learning between spatial domain features and frequency domain features at each level includes:

[0077] The spatial domain features and frequency domain features of the same level are spliced to obtain spliced features, and the average pooling and softmax function are used on the spliced features to obtain the descriptors of each level;

[0078] Multiply the descriptor with the spatial features at the corresponding level and the first learnable matrix to obtain the adaptively learned spatial features;

[0079] Multiply the descriptor with the frequency domain features at the corresponding level and the second learnable matrix to obtain the frequency domain features after adaptive learning.

[0080] Specifically, the structure and workflow of the spatial-frequency domain hierarchical learning module of this embodiment are as follows: Figure 2 As shown in Figure 2, spatial and frequency domain features are learned hierarchically to make them more sensitive to subtle, small, and scattered discriminative regions, including:

[0081] Step 2.1. Obtain the spatial feature F s And the frequency domain feature F f Finally, the two features are scaled and divided into four levels respectively. as well as The scale sizes are 1, 1 / 2, 1 / 4 and 1 / 8 respectively;

[0082] Step 2.2. Between the various levels Perform adaptive feature learning in spatial and frequency domains respectively;

[0083] Step 2.2.1. and Splice it together;

[0084] Step 2.2.2. Apply average pooling to the concatenated tensor and apply the softmax function to obtain the descriptor μ;

[0085] Step 2.2.3. Compare the descriptor μ with And the learnable matrix W s Multiplying together, we get

[0086] Step 2.2.4. Compare the descriptor μ with And the learnable matrix W f Multiplying together, we get

[0087] Step 2.2.5. and The fused features are obtained by splicing them together.

[0088] Step 2.2.6. The remaining fusions between the layers are calculated according to this process;

[0089] Step 2.3. Adaptively fuse features from large to small across all levels of the spatial domain to obtain the spatial features after hierarchical learning.

[0090] Step 2.4. Perform self-attention operation on the spatial features to strengthen the spatial feature representation.

[0091] Step 2.5. Adaptively fuse features from large to small across all levels in the frequency domain to obtain the frequency domain features after hierarchical learning.

[0092] Step 2.6. Perform self-attention operation on the frequency domain features to strengthen the frequency domain feature representation.

[0093] Step 2.7. Obtain the final enhanced feature representation and

[0094] Furthermore, the support set-query set interactive learning module divides the enhanced spatial domain features and frequency domain features into support sets and query sets, and interactively learns the divided support sets and query sets, including:

[0095] The enhanced spatial domain features and frequency domain features are divided into support set and query set respectively to obtain spatial domain support set, spatial domain query set, frequency domain support set, and frequency domain query set;

[0096] The spatial domain support set and the spatial domain query set are interactively learned with the frequency domain support set and the frequency domain query set respectively to obtain an interactive learning score, wherein the interactive learning score is obtained by calculating the Euclidean distance between the two parties performing interactive learning.

[0097] Optimizing the disease classification model based on the interactive learning results includes: calculating the loss based on the obtained interactive learning score and performing back propagation optimization on the disease classification model.

[0098] Specifically, the structure and workflow of the support set-query set interactive learning module of this embodiment are as follows: Figure 3 As shown, and Divide the support set and query set, and conduct interactive learning to increase the variance between classes and reduce the variance within classes. The learning and reconstruction effect is shown in Figure 4 The steps include:

[0099] Step 3.1. and Divide the support set and query set to get S sp 、S fr , Q sp and Q fr .

[0100] Step 3.2. Perform S on the four sets sp →Q sp 、S sp →Q tr 、S fr →Q sp and S fr →Q fr interactive learning;

[0101] Step 3.2.1. S sp →Q sp For example, S sp and Q sp Copy each one and get and

[0102] Step 3.2.2. and Multiply the transpose of and pass through the softmax function to get the similarity matrix M;

[0103] Step 3.2.3. Similarity matrix M and as well as Multiply them separately and get and

[0104] Step 3.2.4. Calculate S sp and The Euclidean distance between them is recorded as scores;

[0105] Step 3.3. Calculate the loss based on the size of scores and perform backpropagation optimization.

[0106] Specifically, the loss function is calculated as follows:

[0107]

[0108] L=L s +L f

[0109] Among them, N is the number of categories, U is the number of query sets, τ a is the control parameter, and d is the Euclidean distance.

[0110] The disease classification model proposed in this embodiment is tested on the plant disease dataset to obtain the recognition result heat map. Figure 5 It can be seen that the method and disease classification model proposed in this embodiment can accurately identify plant disease areas and classification results, and can be applied to real plant disease identification scenarios, greatly reducing costs.

[0111] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A small sample fine-grained image classification method based on spatial-frequency domain hierarchical learning, characterized by: include: Acquire an image of a plant disease to be identified; Inputting the plant disease image to be identified into a preset disease classification model to obtain a plant disease identification result, wherein the disease classification model hierarchically fuses spatial domain features and frequency domain features in the plant disease image to identify the plant disease; The disease classification model includes a dual-frequency domain interactive enhancement module, a spatial-frequency domain hierarchical learning module, and a support set-query set interactive learning module. The dual-frequency domain interactive enhancement module is used to extract features from the input image and obtain the spatial domain features and frequency domain features of the input image; the spatial-frequency domain hierarchical learning module is used to enhance the spatial domain features and frequency domain features based on hierarchical learning to obtain enhanced spatial domain features and frequency domain features; the support set-query set interactive learning module is used to divide the enhanced spatial domain features and frequency domain features into support sets and query sets, and interactively learn the divided support sets and query sets, optimize the disease classification model based on the interactive learning results, and output the final classification results.

2. The small sample fine-grained image classification method based on spatial-frequency domain hierarchical learning according to claim 1 is characterized in that The dual-frequency domain interaction enhancement module extracts features from the input image to obtain spatial domain features and frequency domain features of the input image, including: Inputting the RGB image into the neural network model to obtain the spatial domain features; Performing color space conversion on the RGB image to obtain a YCbCr image, and performing block and DCT transformation on the YCbCr image to obtain a DCT block; Reorganizing the DCT blocks to construct DCT frequency bands, and splitting the DCT frequency bands into high-frequency features and low-frequency features; A self-attention operation is used to enhance the high-frequency features and the low-frequency features respectively, and then a splicing operation is performed to obtain a total feature. The self-attention operation is then used to enhance the total feature and then a Reshape operation is performed to obtain a frequency domain feature equal in size to the spatial domain feature.

3. The small sample fine-grained image classification method based on spatial-frequency domain hierarchical learning according to claim 2 is characterized in that: Reorganizing the DCT blocks to construct DCT frequency bands includes: The DCT blocks of the same frequency domain components are spliced, and the DCT blocks of different frequency domain components are sorted and reorganized from low frequency to high frequency to obtain the DCT frequency band.

4. The small sample fine-grained image classification method based on spatial-frequency domain hierarchical learning according to claim 1 is characterized in that The spatial-frequency hierarchical learning module enhances the spatial and frequency features based on hierarchical learning, and obtaining the enhanced spatial and frequency features includes: The spatial domain features and the frequency domain features are respectively scaled to obtain spatial domain features and frequency domain features at several levels; Adaptive learning between spatial domain features and frequency domain features is performed at each level, and the adaptively learned spatial domain features and frequency domain features are adaptively fused according to the level from large to small to obtain the spatial domain features and frequency domain features after hierarchical learning; Self-attention operations are performed on the spatial domain features and frequency domain features after hierarchical learning to obtain enhanced spatial domain features and frequency domain features.

5. The small sample fine-grained image classification method based on spatial-frequency domain hierarchical learning according to claim 4 is characterized in that: Adaptive learning between spatial domain features and frequency domain features at each level includes: The spatial domain features and frequency domain features of the same level are spliced to obtain spliced features, and the spliced features are subjected to average pooling and softmax function to obtain descriptors of each level; Multiplying the descriptor with the spatial features at the corresponding level and the first learnable matrix to obtain the adaptively learned spatial features; The descriptor is multiplied by the frequency domain features on the corresponding level and the second learnable matrix to obtain the frequency domain features after adaptive learning.

6. The small sample fine-grained image classification method based on spatial-frequency domain hierarchical learning according to claim 1 is characterized in that: The support set-query set interactive learning module divides the enhanced spatial domain features and frequency domain features into support sets and query sets, and interactively learns the divided support sets and query sets, including: The enhanced spatial domain features and frequency domain features are divided into support set and query set respectively to obtain spatial domain support set, spatial domain query set, frequency domain support set, and frequency domain query set; The spatial domain support set and the spatial domain query set are interactively learned with the frequency domain support set and the frequency domain query set respectively to obtain an interactive learning score, wherein the interactive learning score is obtained by calculating the Euclidean distance between the two parties performing the interactive learning.

7. The small sample fine-grained image classification method based on spatial-frequency domain hierarchical learning according to claim 6 is characterized in that: Optimizing the disease classification model based on the interactive learning result includes: performing back propagation optimization on the disease classification model based on the loss calculated based on the obtained interactive learning score.

Images