A fine-grained insect image classification method

Through the improved weak supervision classification model and measurement learning architecture, combined with channel attention and bilinear pooling technology, the insect image classification problem in the existing technology is solved, and the accurate classification of fine-grained insect images and the superior category of wrong results is realized. Restrictions improve the accuracy and reliability of insect recognition.

CN114187183BActive Publication Date: 2024-11-19CHENGDU XINGYINIAN INTELLIGENT TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202111395529.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-23
Publication Date
2024-11-19
Estimated Expiration
2041-11-23

AI Technical Summary

Technical Problem

The existing fine-grained insect image classification algorithm is insufficiently used in the field of insect recognition, making it difficult to accurately classify insects at the fine-grained level, and the existing supervision information is difficult to effectively improve classification performance.

Method used

Adopting an improved weakly supervised fine-grained classification model and a hierarchical multi-label constraint optimization architecture for metric learning, through three-layer category labels and channel attention mechanisms, bilinear pooling layers, combined with the hue enhancement and distance measurement loss function of label control, Improve the accuracy of fine-grained classification of insect images.

Benefits of technology

It realizes more accurate insect image classification at the fine-grained level, can effectively distinguish fine-grained categories and limit error results to the correct superior category, improving the accuracy and reliability of insect image recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114187183B_ABST
    Figure CN114187183B_ABST
Patent Text Reader

Abstract

The present invention discloses a fine-grained insect image classification method, which collects fine-grained pictures of different forms of different types of insects, deletes repeated, blurred, overexposed and low-quality pictures through manual screening, so that the remaining pictures meet the characteristics of large intra-class differences and small inter-class differences, and then establishes picture labels and enhances processing; builds a neural network model for insect classification and trains it, and finally classifies and detects the fine-grained insect pictures to be detected through the trained neural network model, thereby directly outputting the category to which the insects in the pictures belong.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification in the computer field, and more specifically, relates to a fine-grained insect image classification method. Background Art

[0002] Fine-grained image classification is a very important problem in the computer field and can be applied to many valuable professional application scenarios. Although there are a large number of deep learning applications in the field of insect recognition, there is no good research on the application of fine-grained image classification algorithms.

[0003] Fine-grained image classification is a computer vision task that is more difficult than traditional image classification and has more application value in professional scenarios. Insect image classification based on deep learning is of great significance in the prevention and control of insect pests in agriculture and forestry. Applying fine-grained image classification technology to insect image recognition can distinguish insect species that are difficult to classify, further improve the accuracy of insect classification, and thus improve the practicality and reliability of insect image recognition in actual production processes. However, the current research on fine-grained image classification algorithms in the field of insect image recognition is still not sufficient and in-depth.

[0004] The main ideas of fine-grained image classification algorithms are: fine-tuning methods based on general image classification networks, methods combining positioning and recognition based on strong or weak supervised attention mechanisms, bilinear pooling methods based on high-order feature fusion, methods based on metric learning, and transformer-based methods.

[0005] Among them, whether it is the Part-RCNN method based on strong supervision target detection and positioning, or the RA-CNN method based on weak supervision attention mechanism positioning, they are mainly committed to solving the problem of small differences between classes and large differences within classes by locating discriminative regions. Even the method based on metric learning technology, which seems to have nothing to do with discriminative region positioning, is essentially to suppress irrelevant features and find key discriminative features by converging intra-class distances and expanding inter-class distances.

[0006] In biological taxonomy, the classification of species is a systematic hierarchy based on levels, using a fixed number of levels, namely kingdom, phylum, class, order, family, genus, and species. As the classification level goes down, the biological images will become more and more similar visually, and the difficulty of classification will gradually reach the so-called "fine-grained classification" requirements in the field of computer vision. Insects that are recognized in human common sense are generally organisms under the class Insecta. For insects with huge differences in traits, they are generally divided according to "orders". As the classification level goes down, different insect categories will become more and more similar. In the process of annotating the collected insect images, insect experts may be able to identify the specific label of a certain insect sample at the "genus" or "species" level based on their rich experience and knowledge and comparison with insect atlases. However, since insects are the most colorful animal category among the organisms that can be encountered in daily life, with a wide variety of species and strange shapes, even experienced entomologists are unable to identify a specific insect sample at the "genus" and "species" level, but can only identify it at the "order", "suborder" and "family" level.

[0007] Secondly, in practical applications, insect classification algorithms are mainly used to identify agricultural pests. In some cases, the category information required to determine whether an insect is a pest and the corresponding prevention and control strategy does not need to be refined to the "species" level, because insects under the same "family" or "genus" often have similar morphological appearances and traits. In such application scenarios, a good algorithm should meet the following requirements: even if the correct fine-grained classification results cannot be obtained, the algorithm should limit the wrong results to the correct parent category. Finally, the supervision information used by these existing methods for fine-grained image classification problems is the category label at the finest level. However, in the field of fine-grained insect classification, the annotation of high-level labels does not require much additional cost, and its difficulty in obtaining must be much weaker than the bounding box information in the strong supervision method. Therefore, adding high-level category information to supervised learning should effectively improve the performance of fine-grained classification tasks. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a fine-grained insect image classification method based on an improved weakly supervised fine-grained classification model and a hierarchical multi-label constrained optimized fine-grained classification architecture of metric learning to achieve accurate classification of insect images.

[0009] To achieve the above-mentioned object of the invention, the present invention provides a fine-grained insect image classification method, characterized by comprising the following steps:

[0010] (1) Image acquisition and preprocessing;

[0011] Collect fine-grained images of different forms of different insect species, and manually filter out duplicate, blurred, overexposed, and low-quality images to ensure that the remaining images have large intra-class differences and small inter-class differences.

[0012] (2) Create image tags;

[0013] Each image is labeled with three layers of category labels, denoted as (y1, y2, y3). There is a tree-like mapping relationship between the three layers of category labels. The first layer of category labels y1 represents the "order" of insects, the second layer of category labels y2 represents the insect categories with similar visual features under the same order, and the third layer of category labels y3 represents the fine-grained category of insects.

[0014] (3) Image enhancement processing;

[0015] (3.1) Image scaling: The length and width of each image are sampled to a fixed pixel size using bilinear interpolation;

[0016] (3.2) Random rotation of the image: set the rotation factor factor; randomly sample a value in [-factor, factor] as the rotation angle of the image and control the rotation of the image. When the rotation angle is not 90° or 180°, fill the pixel-free area left by the rotation operation with black;

[0017] (3.3) Randomly flip the image horizontally or vertically: Set the probability factor p; randomly generate a probability value in [0, 1]. If the probability value is less than p, the image is randomly flipped horizontally or vertically, otherwise, the image is not flipped;

[0018] (3.4) Random cropping of images: First, enlarge the image to 1.25 times the original image, and then randomly crop an area of ​​the network input size from the enlarged image;

[0019] (3.5) Color dithering of images;

[0020] (3.5.1) Set the jitter factor;

[0021] (3.5.2) Enhance the brightness of the image: Randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the brightness scaling factor, and then multiply the brightness value of the original image by the scaling factor s to obtain the enhanced brightness value.

[0022]

[0023] (3.5.3) Enhance the contrast of the image: randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the contrast scaling factor, and then multiply the contrast of the original image by the scaling factor s to obtain the enhanced contrast value.

[0024]

[0025] (3.5.4) Enhance the saturation of the image: randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the saturation scaling factor, and then multiply the saturation of the original image by the scaling factor s to get the enhanced saturation value

[0026]

[0027] (3.5.5) Enhance the hue of the image through label control: initialize a category set s. When a category of images contains images collected by photographing specimens, add the category to the set. Before performing hue enhancement on the image, determine whether the category to which the image belongs belongs to set s. If it does, perform hue enhancement on the image. If it does not, do not perform enhancement. The hue enhancement operation is: randomly generate a value s between max(0,1-Factor) and (1+Factor) as the hue offset factor. Before performing hue enhancement on the image, the hue value hue of the original image is offset on the hue circle according to the offset factor s to obtain the enhanced hue value.

[0028]

[0029] (4) Setting a training data set: setting the enhanced images and corresponding labels as the training data set;

[0030] (5) Build a neural network model for insect classification;

[0031] Based on the existing resnet50 network model, the entire feature extraction part of Resnet50 is divided into 5 stages. A channel attention mechanism module is added after stage3, stage4, and stage5. The working principle of the channel attention mechanism module is as follows: the output feature I of each stage is a three-dimensional matrix of c×w×h, and each element in the matrix is ​​recorded as I i,j,k , where i∈[1,c],j∈[1,w],k∈[1,h], c is the number of channels, w and h are the width and height;

[0032] The channel attention mechanism module adds an attention weight W of length c to the output feature I to obtain the output feature of the channel attention mechanism The value of each element is recorded as Among them, w i is the weight coefficient;

[0033] After stage 5 and the channel attention mechanism module, the improved bilinear pooling layer is added. The specific working principle is as follows: the output feature Expanded into a two-dimensional vector of length c×(w×h), denoted as Then Perform bilinear fusion to obtain the final output features Finally, the output features Expanded into a one-dimensional vector of length c×c, denoted as

[0034] (6) Training the neural network model;

[0035] The enhanced images and corresponding labels are input into the neural network model, and the neural network model predicts the category of the corresponding insect in the input image.

[0036] (6.1) In the training data set, randomly select b pictures and their corresponding labels as the input for this round of training, and input them into the training neural network model;

[0037] (6.2) Extract the output features of b images through the neural network model i=1,2,…,b, where the output feature of each image is a one-dimensional vector of length c×c;

[0038] (6.3) Output features of each image Through the fully connected layer, the dimensionality reduction processing is performed to obtain the classification and discrimination results p of each image. i , where the classification result p i is a one-dimensional matrix, the length of which is the number of insect species in the training data set M. Each item in the classification result is the probability of the network judging it as the corresponding insect species. Finally, the one with the largest probability is taken as the prediction result, recorded as And according to Get the corresponding

[0039] (6.4) Calculate the loss function value L after this round of training is completed;

[0040]

[0041] Among them, β1 is a hyperparameter, and its value is 0.5 through several sets of comparative experiments. is the loss function value corresponding to the third-level label in the output feature, is the loss function value corresponding to the second-level label in the output feature, L CE is the loss function value corresponding to the classification result;

[0042]

[0043]

[0044]

[0045] in, Indicates that among the b input images of this training, when any two images i and j have the same third-level label is 1, otherwise is 0, Δ is the preset threshold; p is the output feature The dimension of , p∈[1,c×c];

[0046]

[0047] Among them, λ i Indicates the proportion of the number of images in the category to which the i-th input image belongs in the b input images in this round; y iτ is the discriminant coefficient, when the true label value of the i-th input image is equal to τ iτ is 1, otherwise it is 0; p iτ The value of is the classification result p i The value of the τth element in ;

[0048] (6.5) After this round of training is completed, the loss function value of this round is back-propagated through the gradient descent method to update the network parameters, and then return to step (6.1) for the next round of training until the network converges, thus obtaining a trained neural network model;

[0049] (7) Real-time classification of insect images;

[0050] The fine-grained insect image to be detected is scaled to the same size as the training data through an image scaling operation and input into the trained neural network, thereby directly outputting the category to which the insect in the image belongs.

[0051] The object of the invention of the present invention is achieved in this way:

[0052] The present invention discloses a fine-grained insect image classification method, which collects fine-grained images of different forms of different types of insects, deletes repeated, blurred, overexposed and low-quality images through manual screening, so that the remaining images meet the characteristics of large intra-class differences and small inter-class differences, and then establishes image labels and performs enhanced processing; builds a neural network model for insect classification and trains it, and finally classifies and detects the fine-grained insect images to be detected through the trained neural network model, thereby directly outputting the category to which the insects in the images belong.

[0053] At the same time, the fine-grained insect image classification method of the present invention also has the following beneficial effects:

[0054] (1) By using the label-controlled hue enhancement method, the network can learn color features and make some categories not use color features for discrimination;

[0055] (2) By embedding channel attention modules at different stages of the network, the network can extract visual features of different levels. At the same time, through the bilinear pooling technology, high-order fusion between different features is achieved to generate a large number of fine-grained features that are more effective for fine-grained image classification.

[0056] (3) The three-level label combined with the distance metric loss function is used to constrain the classification training process, so that the network can better distinguish the differences between fine-grained categories and expand the distance between dissimilar large categories, so that the network can limit the classification errors of samples to fine-grained categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of a fine-grained insect image classification method of the present invention;

[0058] Figure 2 It is a schematic diagram of the different forms of different insects;

[0059] Figure 3 It is a mapping relationship diagram between the three layers of insect labels;

[0060] Figure 4 This is a schematic diagram of random rotation of pictures;

[0061] Figure 5 This is a schematic diagram of random flipping of pictures;

[0062] Figure 6 This is the basic architecture diagram of the neural network model used for insect classification;

[0063] Figure 7 It is a diagram of the improved neural network model architecture of the present invention;

[0064] Figure 8 It is the result evaluation diagram of three network classifications;

[0065] Fig. 9 This is the experimental result diagram of the effectiveness and location selection of the SE module;

[0066] Fig.10 This is the effectiveness experimental result diagram of bilinear pooling;

[0067] Fig.11 This is a performance comparison chart of different network structures;

[0068] Fig.12 This is a comparison chart of classification performance with the introduction of multi-label supervision constraints;

[0069] Fig.13 This is a performance comparison chart with the introduction of metric loss;

[0070] Fig.14 It is a performance comparison chart of distance metric comparison;

[0071] Fig.15 It is a performance comparison chart of threshold selection comparison;

[0072] Fig.16 This is a performance comparison chart of several other distance metric losses. DETAILED DESCRIPTION

[0073] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0074] Example

[0075] Figure 1 This is a flow chart of a fine-grained insect image classification method of the present invention.

[0076] In this embodiment, if Figure 1 As shown, the present invention provides a fine-grained insect image classification method, comprising the following steps:

[0077] S1, image acquisition and preprocessing;

[0078] In this embodiment, fine-grained images of different forms of different types of insects are collected on common Internet search platforms and special insect database websites through on-site location shooting, specimen shooting, and crawler technology; wherein, the camera position and angle can be adjusted to obtain images containing visual content of different parts of the insect, such as Figure 2In addition, because the morphological structure of insects themselves will change, such as the different curled forms of soft-bodied insect larvae and the open and closed forms of adult insect wings, during the field shooting process, by deliberately extending the time of insect observation, we collected pictures of different forms of the same insect.

[0079] After manual screening, we deleted duplicate images collected from different platforms and images with different hash values ​​but very similar content due to continuous shooting of the same insect during field shooting. In addition, we deleted low-quality images that were blurred, overexposed, and unclear due to shooting problems. As a result, the remaining 3,719 fine-grained insect images of 100 categories all met the characteristics of large intra-class differences and small inter-class differences.

[0080] S2, create image tags;

[0081] Each image is labeled with three layers of category labels, denoted as (y1, y2, y3), where the first layer of category labels y1 represents the "order" of insects; the second layer of category labels y2 represents the category of insects with similar visual features under the same order, and the range is equivalent to a certain "suborder", "family", or "subfamily" under its first layer of label "order"; the third layer of category labels y3 are fine-grained categories at the "genus" and "species" level; there are 15 categories in the first layer of labels, 34 categories in the second layer of labels, and 100 categories in the third layer of labels. After the three layers of category labels are established, each image sample corresponds to a triplet label (y1, y2, y3), such as Figure 3 As shown, there is a tree-like mapping relationship between these three layers of labels.

[0082] S3, image enhancement processing;

[0083] S3.1, Image scaling: The length and width of each image are sampled to a fixed pixel size using a bilinear interpolation method;

[0084] S3.2, random rotation of the image: set the rotation factor factor; randomly sample a value in [-factor, factor] as the rotation angle of the image and control the rotation of the image. The random rotation example is as follows Figure 4 As shown, when the rotation angle is not 90° or 180°, the pixel-free area left during the rotation operation is filled with black;

[0085] S3.3, random horizontal or vertical flipping of the image: set the probability factor p; randomly generate a probability value in [0, 1]. If the probability value is less than p, the image is randomly flipped horizontally or vertically, otherwise, the image is not flipped; the random flipping example is as follows Figure 5 As shown,

[0086] S3.4, random cropping of images: first enlarge the image to 1.25 times the original image, and then randomly crop an area of ​​the network input size from the enlarged image;

[0087] S3.5, color dithering processing of the image;

[0088] The color dithering method is to change the basic properties of the original image, namely brightness, contrast, saturation and hue. Below we describe the specific process of color dithering, as follows:

[0089] S3.5.1. Set the jitter factor Factor;

[0090] S3.5.2. Enhance the brightness of the image: randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the brightness scaling factor, and then multiply the brightness value bright of the original image by the scaling factor s to obtain the enhanced brightness value.

[0091]

[0092] S3.5.3. Enhance the contrast of the image: randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the contrast scaling factor, and then multiply the contrast of the original image by the scaling factor s to obtain the enhanced contrast value.

[0093]

[0094] S3.5.4. Enhance the saturation of the image: randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the saturation scaling factor, and then multiply the saturation of the original image by the scaling factor s to obtain the enhanced saturation value

[0095]

[0096] S3.5.5. Enhance the hue of the image through label control: Initialize a category set s. When a category of images contains images collected by photographing specimens, add the category to the set. Before performing hue enhancement on the image, determine whether the category to which the image belongs belongs to set s. If it does, perform hue enhancement on the image. If it does not, do not perform enhancement. The hue enhancement operation is: randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the hue offset factor. Before performing hue enhancement on the image, the hue value hue of the original image is offset on the hue circle according to the offset factor s to obtain the enhanced hue value.

[0097]

[0098] S4, setting a training data set, setting the enhanced images and corresponding labels as the training data set;

[0099] S5. Build a neural network model for insect classification;

[0100] In this embodiment, the specific architecture of the neural network model for insect classification is as follows: Figure 6 As shown in the figure, f(·) is the neural network classification model. Figure 6 The last fully connected layer and loss function are deleted. The L1 loss function is a classification function, namely the cross entropy loss function. In this paper, it is modified to the cross entropy loss function of online weight statistics. It is connected to the category probability distribution output by the network at the end, and the category distribution feature dimension is consistent with the number of categories. The L2 loss function is a distance metric loss function. It is connected to the features output by the network feature extraction part, and its dimension is consistent with the network output feature. The distance metric loss function can calculate the distance between two features. The parameters of the network feature extraction are optimized through this loss function, thereby obtaining Figure 7 The network structure shown is as follows: Based on the existing resnet50 network model, the entire feature extraction part of Resnet50 is divided into 5 stages. A channel attention mechanism module is added after stage3, stage4, and stage5 respectively. The working principle of the channel attention mechanism module is as follows: The output feature I of each stage is a three-dimensional matrix of c×w×h, and each element in the matrix is ​​recorded as I i,j,k , where i∈[1,c], j∈[1,w], k∈[1,h], c is the number of channels, w and h are the width and height;

[0101] The channel attention mechanism module adds an attention weight W of length c to the output feature I to obtain the output feature of the channel attention mechanism The value of each element is recorded as Among them, wi is the weight coefficient;

[0102] After stage 5 and the channel attention mechanism module, the improved bilinear pooling layer is added. The specific working principle is as follows: the output feature Expanded into a two-dimensional vector of length c×(w×h), denoted as Then Perform bilinear fusion to obtain the final output features Finally, the output features Expanded into a one-dimensional vector of length c×c, denoted as

[0103] In this embodiment, since the final feature dimension of ResNet50 is 2048 dimensions, after bilinear pooling, 4194304-dimensional features are generated. If the fully connected layer outputs 100 categories, a fully connected layer network with 400M parameters will be generated. Therefore, this paper uses a convolutional layer of size 2048 times 1 times 1 times 512 to reduce the final feature dimension to 512 dimensions, reducing the final fully connected layer parameter amount by 16 times.

[0104] S6, training the neural network model;

[0105] The enhanced images and corresponding labels are input into the neural network model, and the neural network model predicts the category of the corresponding insect in the input image.

[0106] S6.1. Randomly select b=20 images and their corresponding labels from the training data set as the input for this round of training and input them into the training neural network model;

[0107] S6.2. Extract the output features of b images through the neural network model i = 1, 2, ..., b, where the output feature of each image is a one-dimensional vector of length c × c;

[0108] S6.3. Output features of each image Through the fully connected layer, the dimensionality reduction processing is performed to obtain the classification and discrimination results p of each image. i , where the classification result p i is a one-dimensional matrix, the length of which is the number of insect species in the training data set M. Each item in the classification result is the probability of the network judging it as the corresponding insect species. Finally, the one with the largest probability is taken as the prediction result, recorded as And according to Get the corresponding

[0109] S6.4. Calculate the loss function value L after the current round of training is completed;

[0110]

[0111] Among them, β1 is a hyperparameter, and its value is 0.5 through several sets of comparative experiments. is the loss function value corresponding to the third-level label in the output feature, is the loss function value corresponding to the second-level label in the output feature, L CE is the loss function value corresponding to the classification result;

[0112]

[0113]

[0114]

[0115] in, Indicates that among the b input images of this training, when any two images i and j have the same third-level label is 1, otherwise is 0, Δ is the preset threshold; p is the output feature The dimension of , p∈[1,c×c];

[0116]

[0117] Among them, λ i Indicates the proportion of the number of images in the category to which the i-th input image belongs in the b input images in this round; y iτ is the discriminant coefficient, when the true label value of the i-th input image is equal to τ iτ is 1, otherwise it is 0; p iτ The value of is the classification result p i The value of the τth element in ;

[0118] S6.5, after the current round of training is completed, the loss function value of this round is back-propagated by the gradient descent method to update the network parameters, and then return to step S6.1 for the next round of training until the network converges, thereby obtaining a trained neural network model;

[0119] S7, real-time classification of insect images;

[0120] The fine-grained insect image to be detected is scaled to the same size as the training data through an image scaling operation and input into the trained neural network, thereby directly outputting the category to which the insect in the image belongs.

[0121] experiment

[0122] This paper conducts a large number of comparative experiments and ablation experiments to ensure the effectiveness of every part of the network.

[0123] Network selection: By Figure 8 It can be seen that ResNet50 and InceptionV3 can achieve good performance in this task. We use acc and F1 as evaluation indicators. Although the result of ResNet50 is slightly worse than that of InceptionV3, the difference is not big. The difference of one thousandth of a point can be easily smoothed out by the randomness of the deep learning process. In addition, since the basic structure of the ResNet50 network is simpler, the composition logic of the entire network is also simpler, and it is easier to make changes to its basic structure.

[0124] The effectiveness of SE module and position selection experiment: Fig. 9 It can be seen that adding the attention mechanism to stage3, stage4, and stage5 has the best effect under the experimental conditions of this embodiment. The experimental results show that the channel attention mechanism of mid- and high-level features can indeed have a more profound impact on the feature learning process when the amount of data is small, and achieve better classification results.

[0125] Effectiveness experiment of bilinear pooling: Fig.10 As shown in the figure, the results are significantly improved after the introduction of the bilinear pooling method, with the accuracy increased by 1.6 percentage points and the F1 score increased by 1 percentage point. According to the original paper, this method can combine position features with content features, so it is helpful for fine-grained classification. According to the understanding and analysis of this article, the performance of the bilinear pooling operation will only be reflected when there are many classification categories. When there are fewer classification categories, there is not much need for effective distinguishing features between categories. When the number of classification categories increases, the number of effective features required to completely distinguish all categories will increase at a geometric rate. Since the last feature involved in the classification in the original ResNet50 network is only 2048 dimensions, after the outer product operation, the features are fused in the second order, and the feature space is increased, which better meets the task of insect classification in this article, which requires a large number of features.

[0126] Performance comparison of different network structures Fig.11 As shown, the evaluation index of the present invention shown in the last column of the figure is obviously better than the performance of other network structures.

[0127] Use acc2-3 for basic model performance evaluation: Fig.12 As shown, it is obvious that without introducing the multi-label supervision constraint framework, the accuracy of the model for insect super-level classification is poor.

[0128] Introduce experiments with metric loss. Fig.13As shown in the figure, the introduction of L+hinge3 loss improves the three-level classification accuracy performance by 6 and 9 thousand points in the two structures respectively, but the two-level classification accuracy in the error sample set decreases slightly. The introduction of L-hinge2 loss improves the three-level classification accuracy performance slightly in the two structures respectively, but significantly improves the two-level classification accuracy in the error sample set, among which ResNet50 improves by about 30 percentage points and ResNet50+SE structure 2 improves by about 20 percentage points. When the two loss functions are combined, the model performance is significantly improved in both evaluation indicators.

[0129] Distance metric comparison experiment: Fig.14 As shown in the figure, for the L-hinge2 loss function, using L1 distance can better improve the performance of acc2-3, and for the L+hinge3 loss function, using L1 distance can better improve the performance of acc3. In addition, since both loss functions optimize the features at the same position in the network, using the same form of distance metric function can limit the feature expression to the same feature space, which is better for the training process.

[0130] Threshold selection experiment: Fig.15 As shown, the best effect is achieved when the threshold Δ of the L1 distance between L-hinge2 and L+hinge3 is 0.7 and 0.2 respectively.

[0131] In addition, the present invention also verifies the ineffectiveness and inhibitory nature of several other distance metric losses. Fig.16 As shown in the figure, the four kinds of losses can suppress the fine-grained classification results of the three-level labels to varying degrees. Among them, the L-hinge3 loss has no obvious effect on the results, L+hinge2 suppresses the three-level fine-grained classification by 2 percentage points, and L+hinge1 and L-hinge1 have a greater inhibitory effect on the experimental results. According to the analysis, L+hinge2 requires the clustering of the features within the two-level class, and the two-level classification is similar to the conventional image classification, which will make the network learn more conventional discriminant features, while the fine-grained classification of the three-level labels requires the network to learn fine-grained discriminant features as much as possible, which will suppress the fine-grained classification performance. In addition, the huge suppression of the 1-level label constraint on the network performance is mainly reflected in the fact that the feature space required for the 1-level classification is inconsistent with the 3-level classification. Combining the two for training is similar to jointly learning two mutually exclusive tasks. In order to learn more 1-level classification features, the network affects the effect of the 3-level classification.

[0132] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

Claims

1. A fine-grained insect image classification method, characterized in that: The following steps are involved: (1) Image acquisition and preprocessing; Collect fine-grained images of different forms of different insect species, and manually filter out duplicate, blurred, overexposed, and low-quality images to ensure that the remaining images have large intra-class differences and small inter-class differences. (2) Create image tags; Each image is labeled with three layers of category labels, denoted as (y1, y2, y3). There is a tree-like mapping relationship between the three layers of category labels. The first layer of category labels y1 represents the "order" of insects, the second layer of category labels y2 represents the insect categories with similar visual features under the same order, and the third layer of category labels y3 represents the fine-grained category of insects. (3) Image enhancement processing; (3.1) Image scaling: The length and width of each image are sampled to a fixed pixel size using bilinear interpolation; (3.2) Random rotation of the image: set the rotation factor factor; randomly sample a value in [-factor, factor] as the rotation angle of the image and control the rotation of the image. When the rotation angle is not 90° or 180°, fill the pixel-free area left by the rotation operation with black; (3.3) Randomly flip the image horizontally or vertically: Set the probability factor p; randomly generate a probability value in [0, 1]. If the probability value is less than p, the image is randomly flipped horizontally or vertically, otherwise, the image is not flipped; (3.4) Random cropping of images: First, enlarge the image to 1.25 times the original image, and then randomly crop an area of ​​the network input size from the enlarged image; (3.5) Color dithering of images; (3.5.1) Set the jitter factor; (3.5.2) Enhance the brightness of the image: Randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the brightness scaling factor, and then multiply the brightness value of the original image by the scaling factor s to obtain the enhanced brightness value. (3.5.3) Enhance the contrast of the image: randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the contrast scaling factor, and then multiply the contrast of the original image by the scaling factor s to obtain the enhanced contrast value. (3.5.4) Enhance the saturation of the image: randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the saturation scaling factor, and then multiply the saturation of the original image by the scaling factor s to get the enhanced saturation value (3.5.5) Enhance the hue of the image through label control: initialize a category set s. When a category of images contains images collected by photographing specimens, add the category to the set; before enhancing the hue of the image, determine whether the category to which the image belongs belongs to the set s. If it does, perform hue enhancement on the image; if it does not, do not perform enhancement; the hue enhancement operation is: randomly generate a value s between max(0, 1-Factor) and (1+Factor) as the hue offset factor; before enhancing the hue of the image, perform an offset operation on the hue circle on the hue value of the original image according to the offset factor s to obtain the enhanced hue value (4) Setting a training data set: setting the enhanced images and corresponding labels as the training data set; (5) Build a neural network model for insect classification; Based on the existing resnet50 network model, the entire feature extraction part of Resnet50 is divided into 5 stages. A channel attention mechanism module is added after stage3, stage4, and stage5. The working principle of the channel attention mechanism module is as follows: the output feature I of each stage is a three-dimensional matrix of c×w×h, and each element in the matrix is ​​recorded as I i,j,k , where i∈[1,c], j∈[1,w], k∈[1,h], c is the number of channels, w and h are the width and height; The channel attention mechanism module adds an attention weight W of length c to the output feature I to obtain the output feature of the channel attention mechanism The value of each element is recorded as Among them, w i is the weight coefficient; After stage 5 and the channel attention mechanism module, the improved bilinear pooling layer is added. The specific working principle is as follows: the output feature Expanded into a two-dimensional vector of length c×(w×h), denoted as Then Perform bilinear fusion to obtain the final output features Finally, the output features Expanded into a one-dimensional vector of length c×c, denoted as (6) Training the neural network model; The enhanced images and corresponding labels are input into the neural network model, and the neural network model predicts the category of the corresponding insect in the input image. (6.1) In the training data set, randomly select b pictures and their corresponding labels as the input for this round of training, and input them into the training neural network model; (6.2) Extract the output features of b images through the neural network model Among them, the output feature of each picture is a one-dimensional vector of length c×c; (6.3) Output features of each image Through the fully connected layer, the dimensionality reduction processing is performed to obtain the classification and discrimination results p of each image. i , where the classification result p i is a one-dimensional matrix, the length of which is the number of insect species in the training data set M. Each item in the classification result is the probability of the network judging it as the corresponding insect species. Finally, the one with the largest probability is taken as the prediction result, recorded as And according to Get the corresponding (6.4) Calculate the loss function value L after this round of training is completed; Among them, β1 is a hyperparameter, and its value is 0.5 through several sets of comparative experiments. is the loss function value corresponding to the third-level label in the output feature, is the loss function value corresponding to the second-level label in the output feature, L CE is the loss function value corresponding to the classification result; in, Indicates that among the b input images of this training, when the third-level labels of any two images i and j are the same is 1, otherwise is 0, Δ is the preset threshold; p is the output feature The dimension of , p∈[1,c×c]; Among them, λ i Indicates the proportion of the number of images in the category to which the i-th input image belongs in the b input images in this round; y iτ is the discriminant coefficient, when the true label value of the i-th input image is equal to τ iτ is 1, otherwise it is 0; p iτ The value of is the classification result p i The value of the τth element in ; (6.5) After this round of training is completed, the loss function value of this round is back-propagated through the gradient descent method to update the network parameters, and then return to step (6.1) for the next round of training until the network converges, thus obtaining a trained neural network model; (6) Real-time classification of insect images; The fine-grained insect image to be detected is scaled to the same size as the training data through an image scaling operation and input into the trained neural network, thereby directly outputting the category to which the insect in the image belongs.

Citation Information

Patent Citations

  • Fine-grained image classification method based on detection and recognition of network architecture

    CN109086792A

  • Pest image classification method based on fine-grained classification technology

    CN113239947A

  • Channel interaction networks for image categorization

    US20210248421A1