Efficient training data set labeling method based on multi-class insect images

Abstract

The invention discloses an efficient labeling method based on a multi-class insect image training data set, and relates to the technical field of image processing, and the method is characterized in that the method comprises the following steps: S0, preprocessing the multi-class insect image data set; the method comprises the following steps: S1, constructing a universal insect detector GInsectD; s2, cutting to generate an insect target small graph with a unique index; s3, learning a good feature representation by using a self-supervised learning method DEIP; s4, classifying the target small graphs by using a hierarchical clustering algorithm; and S5, the classified small target images are traced back to the original image to form a final target detection data set. According to the invention, the open vocabulary target detector is modified and finely adjusted, so that the open vocabulary target detector can well detect various insect images, shows strong generalization performance, can be used as a universal insect detector to position all insects on the insect images, and can be used as a self-supervised representation for learning by means of the universal insect detector and self-supervised representation. The demand for artificial resources is greatly eliminated, and the marking efficiency of the data set is also improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a method for labeling an efficient training data set based on multiple types of insect images. Background Art

[0002] With the rapid development of deep learning technology, more and more insect counting methods have been transformed from inefficient manual statistics to efficient automatic identification and counting. However, an issue that is easily overlooked is that a robust and excellent deep learning-based algorithm often requires a large amount of labeled data sets to support it. Although deep learning models have shown excellent performance in tasks such as image classification and target detection in public data sets, their success depends largely on large-scale, high-quality, and diverse labeled data.

[0003] In the field of agricultural pest monitoring, with the addition of automatic insect-catching equipment, it has become feasible to collect a large number of original insect image samples, but the problem that comes with it is the difficulty in labeling data sets: compared with the pest images captured in the natural environment, the pests captured by the equipment become darker in color after death, and some pests may be missing limbs, resulting in large morphological differences; among them, the equipment that uses insect phototaxis to attract insects can capture a large number of insects, large and small insect targets are mixed together, high-density insects may block each other, the complexity of the background increases, and the mutual interference between pests with similar appearances increases the difficulty of labeling. High-quality data labeling requires not only professional biological knowledge, but also in-depth understanding of specific types of pests, otherwise it is easy to mislabel or miss labels. At the same time, because the distribution and living habits of agricultural pests vary with location and season, their appearance characteristics in different growth cycles will also vary. In order to make the model adapt to pest identification tasks in various scenarios, it is necessary to ensure that the training data set contains samples over a wide range of time spans and regions, all of which require inestimable and expensive human resources to complete.

[0004] Therefore, new solutions need to be proposed to solve this problem. Summary of the invention

[0005] In view of the shortcomings of the prior art, the object of the present invention is to provide an efficient annotation method based on a multi-class insect image training data set.

[0006] The above technical objectives of the present invention are achieved through the following technical solutions: A method for labeling a multi-class insect image-based efficient training data set, comprising the following steps: S0: Preprocessing of multiple insect image datasets, including data filtering, pseudo-label generation and automatic background removal; S1: Build a general insect detector GInsectD to locate all insect targets; S2: cropping to generate a small insect target image with a unique index; S3: Use the self-supervised learning method DEIP to learn good feature representation; S4: Classify the target image using the hierarchical clustering algorithm; S5: The classified target images are traced back to the original images to form the final target detection dataset.

[0007] The present invention is further configured as follows: in the step S0, preprocessing the multi-category insect image dataset comprises the following sub-steps: S01: Data filtering low quality images.

[0008] S02: Pseudo-label generation and automatic background removal include using a multimodal open set target detector to mark all insects and removing image files that do not contain target boxes, i.e. background image files, by appropriately lowering the confidence.

[0009] The present invention is further configured as follows: in the step S1, constructing a universal insect detector GInsectD comprises the following sub-steps: S11: Multimodal open-set object detection model architecture using dual encoder-single decoder; S12: Focal Loss is used as the classification loss function and L1Loss is used as the bounding box regression loss function; S13: Swin-T is used as the image backbone and BERT is used as the text backbone; S14: The detection category name uses 'insect'; S15: The learning rate scheduling strategy during model training adopts CosineAnnealingLR.

[0010] The present invention is further configured as follows: in the step S2, cropping and generating a small image of the insect target with a unique index includes using a python tool to crop the small image according to the detection frame coordinate information, and saving it as a jpg format file.

[0011] The present invention is further configured as follows: in step S3, using the self-supervised learning method DEIP to learn a good feature representation includes the following sub-steps: S31: The input image is subjected to different data augmentation to obtain a global cropped view and a local cropped view; S32: Calculate the cross entropy loss by comparing the features of different views; S33: In back propagation, only the parameters of the apprentice network are updated to minimize the loss function.

[0012] The present invention is further configured as follows: in the step S4, using a hierarchical clustering algorithm to classify the target image includes using a hierarchical clustering method similar to a hierarchical classification structure specific to biology for classification, and the number of clusters is set according to the maximum number of insect species in the image.

[0013] The present invention is further configured as follows: in step S5, tracing back the classified target small images to the original images to form the final target detection data set includes the following sub-steps: S51: Establish a category database, using the file names of multiple insect images and target frame coordinate information as thumbnail file names and database unique indexes; S52: associating the pseudo classification results obtained by the clustering or classification model and uploading them to a structured database; S53: import all cropped images into the dataset annotation software and load pre-annotated information from the database; S54: After the labeling expert corrects and confirms the data set, the labeling information is updated to the database. Finally, the script automatically traces the category back to the insect trap target detection data set based on the information in the index.

[0014] The present invention is further configured as follows: in the step S1, constructing the universal insect detector GInsectD also includes using a small sample iterative training method, that is, using the model weights of the previous iteration in subsequent iterations, and increasing the number of images to further improve the detection effect.

[0015] The present invention is further configured as follows: in the step S3, learning a good feature representation using the self-supervised learning method DEIP also includes using a multi crop technology, and calculating a cross entropy loss by comparing features of different views, wherein the calculation of the cross entropy loss involves a similarity comparison between classification tokens and patch tokens.

[0016] The present invention is further configured as follows: in the step S0, the data filtering and merging further comprises using an image quality assessment algorithm to automatically identify and exclude low-quality images.

[0017] In summary, the present invention has the following beneficial effects: by modifying some training strategies and fine-tuning small sample iterations of the open vocabulary target detector, the present invention enables it to detect various types of insect images well, exhibits strong generalization performance, and can be used as a general insect detector to locate all insects on multiple types of insect images. Through self-supervised representation learning of a large number of insect target small images, the proposed DEIP self-supervised learning method can learn the general feature representation of insects, so that the accuracy and robustness of feature extraction tasks or downstream classification tasks can be greatly improved. At the same time, by using a general insect detector and self-supervised representation learning, an efficient training data set annotation method based on multiple types of insect images is constructed, which greatly gets rid of the demand for artificial resources and greatly improves the labeling efficiency of the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 Schematic diagram of the Focal Loss formula in the present invention; Figure 3 Schematic diagram of the L1 Loss formula in the present invention; Figure 4 Schematic diagram of the cross entropy loss calculation formula in the present invention; Figure 5 It is a framework diagram of GInsectD in the present invention; Figure 6 A framework diagram of DEIP in the present invention; Figure 7 It is the detection result diagram of GInsectD in the present invention; Figure 8 This is a result diagram of the method of the present invention, with different types marked with different colors; DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] In order to better describe and illustrate the embodiments of the present application, reference may be made to one or more drawings, but the additional details or examples used to describe the drawings should not be considered as limiting the scope of the invention of the present application, any of the currently described embodiments or preferred methods.

[0021] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "top", "bottom", "inside" and "outside" are based on the positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention. They do not indicate that the device referred to must have a specific direction or operate in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0023] The present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0024] A method for labeling efficient training datasets based on multi-class insect images, such as Figure 1 As shown, the following steps are included: S0: Preprocessing of multi-category insect image datasets, including data filtering and merging, as well as pseudo-label generation and automatic background removal; S1: Build a general insect detector GInsectD to locate all insect targets; S2: cropping to generate a small insect target image with a unique index; S3: Use the self-supervised learning method DEIP to learn good feature representation; S4: Classify the target image using the hierarchical clustering algorithm; S5: The classified target images are traced back to the original images to form the final target detection dataset.

[0025] Specifically, in step S0, preprocessing a multi-class insect image dataset includes the following sub-steps: S01: Data filtering low quality images.

[0026] S02: Pseudo-label generation and automatic background removal include using a multimodal open set target detector to mark all insects, and removing image files that do not contain target boxes, i.e., background image files, by appropriately lowering the confidence level. The above settings remove low-quality images caused by equipment problems, ensuring the quality of images in the dataset for subsequent processing, and providing high-quality training data for building an accurate model. Removing unqualified images can reduce the processing of these images in subsequent steps, thereby saving computing resources and time and improving overall processing efficiency.

[0027] In step S1, constructing a general insect detector GInsectD includes the following sub-steps: S11: Multimodal open-set object detection model architecture using dual encoder-single decoder. By using a multimodal open-set object detection model architecture using dual encoder-single decoder, it can process image and text information simultaneously, improve the ability to understand complex scenes, enhance zero-shot detection capabilities, and improve the adaptability of the model in diverse and dynamically changing environments; S12: Focal Loss is used as the classification loss function and L1 Loss is used as the bounding box regression loss function. Specifically, the Focal Loss formula is as follows: Figure 2 As shown, is the model's predicted probability of the category to which the sample belongs; It is a regulating factor used to balance the ratio of positive and negative samples; is an adjustment parameter, called the focus parameter, and the L1 Loss formula is as follows Figure 3 As shown, is the number of samples. It is The true value of the samples. It is The predicted value of samples; S13: Swin-T is used as the image backbone and BERT is used as the text backbone; S14: The detection category name uses 'insect'; S15: The learning rate scheduling strategy during model training adopts CosineAnnealingLR.

[0028] Furthermore, in step S1, constructing the general insect detector GInsectD also includes a method of iterative training using small samples, that is, using the model weights of the previous iteration in subsequent iterations, and increasing the number of images to further improve the detection effect. The above settings are helpful for rapid iteration and performance improvement of new tasks. By freezing the BERT backbone network and fine-tuning it multiple times using the self-training label iterative generation method, the sample size of the data set increases successively. The purpose is to use the powerful generalization ability of the pre-trained model to continuously modify the pseudo-labels to improve the model performance.

[0029] In step S2, cropping and generating an insect target thumbnail with a unique index includes using a python tool to crop the thumbnail according to the detection frame coordinate information and saving it as a jpg format file. The insect target thumbnail file naming specification, that is, the unique index is obtained by concatenating the file names of the multiple types of insect images in which it is located and the target frame position where the target is located. By assigning a unique index based on its position in the original image to each cropped insect target thumbnail, each thumbnail can be easily tracked and associated with its source image, thereby simplifying the organization and management of data, and the unique index makes it easier to associate the cropped image with its metadata, which is very useful for data analysis and subsequent machine learning model training.

[0030] In step S3, learning a good feature representation using the self-supervised learning method DEIP includes the following sub-steps: S31: The input image is subjected to different data augmentation to obtain a global cropped view and a local cropped view; S32: Calculate the cross entropy loss by comparing the features of different views; S33: In back propagation, only the parameters of the apprentice network are updated to minimize the loss function.

[0031] In step S3, using the self-supervised learning method DEIP to learn a good feature representation also includes using the multicrop technique and calculating the cross entropy loss by comparing the features of different views. The calculation of the cross entropy loss involves comparing the similarity of the classification token and the patch tokens. The calculation formula of the cross entropy loss is as follows: Figure 4 As shown in the figure, the classification token obtained by the apprentice network is converted into a k-dimensional vector after projection, and the probability value Pa is obtained by applying softmax. Similarly, the master network feature is projected and softmax is applied, and then the moving average is used for centering to obtain Pm. By comparing the high-level semantic information of the image obtained by different perspectives and enhancement methods, the DEIP method can learn richer and more robust feature representations. These feature representations can capture the essential characteristics of the image without being affected by the specific perspective or enhancement method. At the same time, since the DEIP method learns the consistency of global features through self-supervised learning, this enables the model to maintain consistent feature representations under different data enhancements and transformations, thereby improving the generalization ability of the model. The DEIP method forces the model to learn small-scale dependencies and high-frequency detail information, which helps the model capture more refined image features and improve the ability to recognize image details.

[0032] In step S4, the target image is classified using a hierarchical clustering algorithm, including using a hierarchical clustering method similar to the hierarchical classification structure unique to biology for classification, and the number of clusters is set according to the maximum number of insect species in the image. Hierarchical clustering can show the hierarchy of data, which helps to understand the inherent hierarchical relationship of the data. Through the dendrogram, the inherent structure of the data and the clustering of each level can be deeply understood, which increases the interpretability of the clustering results. After the clustering is completed, a cross-section can be made at any level to obtain a specified number of clusters. This flexibility allows the user to select the most appropriate clustering level according to the needs.

[0033] In step S5, tracing back the classified target small images to the original images to form the final target detection data set includes the following sub-steps: S51: Establish a category database, using the file names of multiple insect images and target frame coordinate information as thumbnail file names and database unique indexes; S52: associating the pseudo classification results obtained by the clustering or classification model and uploading them to a structured database; S53: import all cropped images into the dataset annotation software and load pre-annotated information from the database; S54: After the labeling expert corrects and confirms the data set, the labeling information is updated to the database. Finally, the script automatically traces the category back to the insect trap target detection data set based on the information in the index.

[0034] The present invention modifies some training strategies and iterates small samples to fine-tune the open vocabulary target detector, so that it can detect various types of insect images well, show strong generalization performance, and can be used as a general insect detector to locate all insects on multiple types of insect images. Through self-supervised representation learning of a large number of insect target small images, the proposed DEIP self-supervised learning method can learn the general feature representation of insects, so that the accuracy and robustness of feature extraction tasks or downstream classification tasks can be greatly improved. At the same time, with the help of general insect detectors and self-supervised representation learning, an efficient training data set annotation method based on multiple types of insect images is constructed. It greatly gets rid of the demand for artificial resources and greatly improves the labeling efficiency of data sets.

[0035] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An efficient annotation method based on a multi-class insect image training dataset, characterized by: The following steps are involved: S0: Preprocessing of multi-category insect image datasets, including data filtering and merging, as well as pseudo-label generation and automatic background removal; S1: Build a general insect detector GInsectD to locate all insect targets; S2: cropping to generate a small insect target image with a unique index; S3: Use the self-supervised learning method DEIP to learn good feature representation; S4: Classify the target image using the hierarchical clustering algorithm; S5: The classified target images are traced back to the original images to form the final target detection dataset.

2. The efficient annotation method based on a multi-class insect image training data set according to claim 1, characterized in that: In step S0, preprocessing a multi-class insect image dataset includes the following sub-steps: S01: filter low-quality images; S02: Pseudo-label generation and automatic background removal include using a multimodal open set target detector to mark all insects and removing image files that do not contain target boxes, i.e. background image files, by appropriately lowering the confidence.

3. The efficient annotation method based on a multi-class insect image training data set according to claim 1, characterized in that: In step S1, constructing a general insect detector GInsectD includes the following sub-steps: S11: Multimodal open-set object detection model architecture using dual encoder-single decoder; S12: Focal Loss is used as the classification loss function and L1Loss is used as the bounding box regression loss function; S13: Swin-T is used as the image backbone and BERT is used as the text backbone; S14: The detection category name uses 'insect'; S15: The learning rate scheduling strategy during model training adopts CosineAnnealingLR.

4. The efficient annotation method based on a multi-class insect image training data set according to claim 1, characterized in that: In the step S2, cropping and generating a small image of the insect target with a unique index includes using a python tool to crop the small image according to the detection frame coordinate information and saving it as an image format file.

5. The efficient annotation method based on a multi-class insect image training data set according to claim 1, characterized in that: In step S3, using the self-supervised learning method DEIP to learn a good feature representation includes the following sub-steps: S31: The input image is subjected to different data augmentation to obtain a global cropped view and a local cropped view; S32: Calculate the cross entropy loss by comparing the features of different views; S33: In back propagation, only the parameters of the apprentice network are updated to minimize the loss function.

6. The efficient annotation method based on a multi-class insect image training data set according to claim 1, characterized in that: In step S4, using a hierarchical clustering algorithm to classify the target thumbnails includes using a hierarchical clustering method similar to a hierarchical classification structure specific to biology for classification, and the number of clusters is set according to the maximum number of insect species in the image.

7. The efficient annotation method based on a multi-class insect image training data set according to claim 1, characterized in that: In step S5, tracing back the classified target small images to the original images to form the final target detection data set includes the following sub-steps: S51: Establish an insect category database, using the image file names of multiple insect categories and target frame coordinate information as thumbnail file names and database unique indexes; S52: associating the pseudo classification results obtained by the clustering or classification model and uploading them to a structured database; S53: import all cropped images into the dataset annotation software and load pre-annotated information from the database; S54: After the labeling expert corrects and confirms the data set, the labeling information is updated to the database. Finally, the script automatically traces the category back to the target detection data set based on the information in the index.

8. The efficient annotation method based on a multi-class insect image training data set according to claim 3 is characterized by: In the step S1, constructing the universal insect detector GInsectD also includes using a small sample iterative training method, that is, using the model weights of the previous iteration in subsequent iterations and increasing the number of images to further improve the detection effect.

9. The efficient annotation method based on a multi-class insect image training data set according to claim 5, characterized in that: In the step S3, learning a good feature representation using the self-supervised learning method DEIP also includes using a multicrop technique, and calculating a cross entropy loss by comparing features of different views, wherein the calculation of the cross entropy loss involves a similarity comparison between classification tokens and patch tokens.

10. The efficient annotation method based on a multi-class insect image training data set according to claim 2, characterized in that: In step S0, data filtering and merging also includes using an image quality assessment algorithm to automatically identify and exclude low-quality images.

Images