Insect classification method and system based on multi-dimensional learning of convolutional network

Abstract

The present invention discloses an insect classification method based on multi-dimensional learning of a convolutional network, comprising the following steps: obtaining insect pictures containing various insect species, pre-processing the insect pictures and retaining the outer contours of the insects to form category insect pictures; dividing the category insect pictures into a training data set and a verification data set, and performing data enhancement processing on the training data set to obtain an enhanced training sample set; constructing a multi-dimensional insect recognition network pre-training model; training and optimizing the multi-dimensional insect recognition network pre-training model based on the enhanced training sample set to obtain a multi-dimensional insect recognition network model; inputting the pictures to be recognized into the multi-dimensional insect recognition network model to obtain recognition results, wherein the picture sets to be recognized have the same size. The present invention introduces multiple dimensional information of insects as auxiliary features to assist model recognition, so that the model can simultaneously consider information other than image features during the recognition process to improve the recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular relates to an insect classification method and system based on convolutional network multi-dimensional learning. Background Art

[0002] There are many kinds of insects at present. As of now, from the records of pest species, we know that there are about 300 kinds of rice pests, more than 300 kinds of cotton pests, more than 160 kinds of apple pests, and about 200 kinds of mulberry pests. Due to the wide variety of pests and the great harm to crops, the management and prevention of crop diseases and pests have a vital impact on the yield and quality of crops. Therefore, in order to help farmers better identify insect species, on the one hand, it can help farmers improve their understanding of unknown insects; on the other hand, it can effectively prevent and control known insects, that is, better identify insects, and then effectively guide pests and beneficial insects, so that they can serve agricultural products more effectively and expand economic benefits.

[0003] Traditional insect identification methods are generally performed by experts. Therefore, if experts are required to be present every time an insect identification scene is encountered, it will not only consume a lot of time and energy of insect experts, but also may encounter species that experts are not familiar with, making effective insect identification difficult. Therefore, insect identification methods based on deep learning technology have emerged, which can not only improve the efficiency of people who need to identify insects, but also accurately locate insects.

[0004] With the deepening of artificial intelligence technology in the agricultural field, it has been possible to learn the texture information and deep semantic information in insect pictures through deep learning technology to perform insect recognition, and effectively improve its accuracy and efficiency. However, the recognition method based on deep learning models still has certain bottlenecks that make the recognition results inaccurate, such as: it is difficult to effectively distinguish similar pictures, it is difficult to effectively solve the long-tail distribution phenomenon of training sample data, and traditional deep learning training methods need to uniformly scale pictures to the same size for training, thus losing the feature of obtaining insect scale to assist effective recognition. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides an insect classification method and system based on convolutional network multi-dimensional learning.

[0006] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0007] An insect classification method based on convolutional network multi-dimensional learning includes the following steps:

[0008] Obtaining insect pictures containing various insect species, preprocessing the insect pictures and retaining the outer contours of the insects to form category insect pictures;

[0009] Dividing the insect images of the category into a training data set and a verification data set, and performing data enhancement processing on the training data set to obtain an enhanced training sample set;

[0010] Constructing a multi-dimensional insect recognition network pre-training model, wherein the multi-dimensional insect recognition network pre-training model includes an image feature extraction module, a collection image feature extraction module, an image size fully connected layer, an ArcFace Loss loss function, a Focal Loss loss function and a total loss function;

[0011] Based on the enhanced training sample set, the multi-dimensional insect recognition network pre-training model is trained and optimized to obtain a multi-dimensional insect recognition network model;

[0012] The images to be identified are input into a multi-dimensional insect identification network model to obtain identification results, wherein the image sets to be identified have the same size.

[0013] As an implementable method, preprocessing the insect image and retaining the outer contour of the insect to form a classified insect image includes the following steps:

[0014] Performing insect outer contour calibration processing on the insect image to obtain an outer contour calibration image, wherein the calibration conditions of the calibration processing are: the main body area of ​​the insect accounts for at least 70% of the outer contour area of ​​the insect, and the identification features of the corresponding insect category are retained;

[0015] The outer contour of the insect is intercepted from the outer contour calibration image to obtain an insect outer contour image, which is used as a category insect image.

[0016] As an implementable method, the data enhancement processing includes: one or more of resizing, mixup, brightness enhancement, saturation enhancement, contrast enhancement, random erasing, random scale cropping and uniform scaling to the same size, and image flipping.

[0017] As an implementable method, the construction of a multi-dimensional insect recognition network pre-training model includes the following steps:

[0018] Construct an image feature extraction module, which includes an initial extraction Stem unit, an inverted residual unit and a head unit, wherein the initial extraction Stem unit includes a 3x3 convolutional layer and a first BN subunit; the inverted residual unit includes 16 inverted residual structures implemented based on an attention mechanism; the head unit includes a 1x1 convolutional layer, a second BN subunit, a pooling layer and a fully connected layer;

[0019] The length and width of each insect image are normalized, and the output dimension is 2D to obtain a 2D image.

[0020] The temporal information features of each insect image of each category are obtained and one-hot encoded, converted into feature vectors, and the feature vectors are processed by network embedding to obtain network embedding results as input features, wherein the grid of network embedding processing includes a first convolutional layer and a second convolutional layer, the first convolutional layer has a dimension of 1 x m1x 1 x 128, and the second convolutional layer has a dimension of 1 x 1 x 128 xn, m1 represents the dimension of the temporal information feature, and the feature output dimension is n-dimensional;

[0021] Construct a collection image feature extraction module and an image-sized fully connected layer with an input dimension of k and an output dimension k equal to the number of insect categories.

[0022] As an implementable method, the following steps are also included:

[0023] Construct ArcFace Loss loss function, Focal Loss loss function and total loss function respectively;

[0024] The dimensions of insect images are converted into numerical features based on the ArcFace Loss function; the time period features of insect photography are divided into time periods, and the division results are one-hot encoded and input into the multi-dimensional insect recognition network pre-training model to extract time features;

[0025] Based on the Focal Loss function, the enhanced training sample set is optimized to eliminate the long-tail distribution phenomenon that exists when the training is unbalanced.

[0026] Determine whether the training results converge based on the total loss function.

[0027] As an implementable method, the training of the multi-dimensional insect recognition network pre-training model includes the following steps:

[0028] Randomly selecting a number of image samples from the enhanced training sample set, wherein the image samples also include corresponding image scale information;

[0029] The image samples and the corresponding image scale information are input into the multi-dimensional insect recognition network pre-training model, and the network parameters of the multi-dimensional insect recognition network pre-training model are optimized and updated based on the SGD optimization algorithm;

[0030] The dimensions of insect images are converted into numerical features based on the ArcFace Loss function; the time period features of insect photography are divided into time periods, and the division results are one-hot encoded and input into the multi-dimensional insect recognition network pre-training model to extract time features;

[0031] Based on the Focal Loss function, the enhanced training sample set is optimized to eliminate the long-tail distribution phenomenon that exists when the training is unbalanced.

[0032] Whether the training results have converged is determined based on the total loss function, and the accuracy of the training results is verified based on the verification data set to obtain a multi-dimensional insect recognition network model.

[0033] As an implementable method, the method of using a multi-dimensional insect recognition network model to recognize a picture to be recognized includes the following steps:

[0034] Scale each image to be identified, and extract an image of the same size as the input image as the input image of the multi-dimensional insect identification network model;

[0035] Obtain the length and width information of the image and perform normalization processing as the numerical features of the scale layer;

[0036] Obtain the time information of insect activities in the image, convert it into corresponding time periods, and perform one-hot encoding and network embedding processing to extract time features;

[0037] The output results of the multi-dimensional insect recognition network model are matched with the index of the insect category, and the category with the highest confidence is selected as the final recognition result.

[0038] As an implementable method, the total loss function is:

[0039]

[0040] in, Represents the total loss function of the multi-dimensional insect recognition network model, Respectively represent ArcFaceLoss loss function and Focal Loss loss function;

[0041] The ArcFace Loss loss function is expressed as follows:

[0042]

[0043] in, represents the predicted output result, m represents the angle interval, and N represents the total sample size;

[0044] The Focal Loss loss function is expressed as follows:

[0045]

[0046] in, represents the output probability of the relevant category, Represents adjustable attention parameters.

[0047] An insect classification system based on convolutional network multi-dimensional learning, including a data set acquisition module, a partitioning processing module, a model building module, a model training module and a recognition module;

[0048] The data set acquisition module is used to acquire insect pictures containing various insect species, pre-process the insect pictures and retain the outer contours of the insects to form category insect pictures;

[0049] The division processing module is used to divide the insect images of the category into a training data set and a verification data set, and perform data enhancement processing on the training data set to obtain an enhanced training sample set;

[0050] The model construction module is used to construct a multi-dimensional insect recognition network pre-training model, wherein the multi-dimensional insect recognition network pre-training model includes an image feature extraction module, a collection image feature extraction module, an image size fully connected layer, an ArcFace Loss loss function, a Focal Loss loss function and a total loss function;

[0051] The model training module is configured to: train and optimize the multi-dimensional insect recognition network pre-training model based on the enhanced training sample set to obtain a multi-dimensional insect recognition network model;

[0052] The recognition module is used to input the pictures to be recognized into the multi-dimensional insect recognition network model to obtain the recognition results, wherein the picture sets to be recognized have the same size.

[0053] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following method is implemented:

[0054] Obtaining insect pictures containing various insect species, preprocessing the insect pictures and retaining the outer contours of the insects to form category insect pictures;

[0055] Dividing the insect images of the category into a training data set and a verification data set, and performing data enhancement processing on the training data set to obtain an enhanced training sample set;

[0056] Constructing a multi-dimensional insect recognition network pre-training model, wherein the multi-dimensional insect recognition network pre-training model includes an image feature extraction module, a collection image feature extraction module, an image size fully connected layer, an ArcFace Loss loss function, a Focal Loss loss function and a total loss function;

[0057] Based on the enhanced training sample set, the multi-dimensional insect recognition network pre-training model is trained and optimized to obtain a multi-dimensional insect recognition network model;

[0058] The images to be identified are input into a multi-dimensional insect identification network model to obtain identification results, wherein the image sets to be identified have the same size.

[0059] An insect classification device based on convolutional network multi-dimensional learning includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the following method is implemented:

[0060] Obtaining insect pictures containing various insect species, preprocessing the insect pictures and retaining the outer contours of the insects to form category insect pictures;

[0061] Dividing the insect images of the category into a training data set and a verification data set, and performing data enhancement processing on the training data set to obtain an enhanced training sample set;

[0062] Constructing a multi-dimensional insect recognition network pre-training model, wherein the multi-dimensional insect recognition network pre-training model includes an image feature extraction module, a collection image feature extraction module, an image size fully connected layer, an ArcFace Loss loss function, a Focal Loss loss function and a total loss function;

[0063] Based on the enhanced training sample set, the multi-dimensional insect recognition network pre-training model is trained and optimized to obtain a multi-dimensional insect recognition network model;

[0064] The images to be identified are input into a multi-dimensional insect identification network model to obtain identification results, wherein the image sets to be identified have the same size.

[0065] The present invention has significant technical effects due to the adoption of the above technical solution:

[0066] The present invention introduces multiple dimensions of information of insects as auxiliary features to assist model recognition, so that the model can consider information other than image features during the recognition process to improve the recognition accuracy. For recognition devices that are fixed after installation, there will be no situation where the same object obtains pictures of different pixel sizes when the mobile device takes pictures. Therefore, the scale feature is introduced into the insect recognition model, which can assist in recognition according to the size of the insect. After the introduction of the time information feature, insects with different living habits have better discrimination. In addition, a framework for multi-dimensional feature input is provided, and the crops, longitude and latitude regional information, and other related information where the insects live can be combined according to the model architecture of the present invention. Using Focal Loss as the loss function for label recognition can alleviate the data imbalance problem of the long-tail distribution phenomenon of training data to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0068] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0069] Figure 2 It is the structural diagram of the convolutional network model;

[0070] Figure 3 This is the structural diagram of the attention mechanism module of the convolutional network;

[0071] Figure 4 ArcFace Loss is more effective than traditional Softmax Loss in distinguishing multiple categories.

[0072] Figure 5 It is a demonstration of several data enhancement effects of the training set of the present invention;

[0073] Figure 6 The accuracy results of multi-dimensional insect features are compared with pure image features;

[0074] Figure 7 It is the training model structure;

[0075] Figure 8 It is a schematic diagram of the process of insect identification combined with target detection;

[0076] Fig. 9 This is a display of insect recognition results combined with target detection;

[0077] Fig.10 It is a schematic diagram of the overall structure of the system of the present invention. DETAILED DESCRIPTION

[0078] The present invention is further described in detail below in conjunction with embodiments. The following embodiments are for explanation of the present invention but the present invention is not limited to the following embodiments.

[0079] Embodiment 1:

[0080] An insect classification method based on multi-dimensional learning of convolutional networks, such as Figure 1 As shown, the following steps are included:

[0081] S100, obtaining insect pictures containing various insect species, preprocessing the insect pictures and retaining the outer contours of the insects to form category insect pictures;

[0082] S200, dividing the insect images into a training data set and a verification data set, and performing data enhancement processing on the training data set to obtain an enhanced training sample set;

[0083] S300, constructing a multi-dimensional insect recognition network pre-training model, wherein the multi-dimensional insect recognition network pre-training model includes an image feature extraction module, a collection image feature extraction module, an image size fully connected layer, an ArcFace Loss loss function, a Focal Loss loss function and a total loss function;

[0084] S400, training and optimizing the multi-dimensional insect recognition network pre-training model based on the enhanced training sample set to obtain a multi-dimensional insect recognition network model;

[0085] S500: Input the pictures to be identified into a multi-dimensional insect identification network model to obtain identification results, wherein the pictures to be identified have the same size.

[0086] An insect classification method based on multi-dimensional learning of convolutional networks. The detailed steps are as follows:

[0087] Step 1: Obtain insect pictures containing various insect species to form an original data set. In the original data set, all insect pictures need to meet the requirement that the main body of the insect occupies more than 70% of the total area of ​​the picture to reduce the interference of the background in picture recognition. If the condition that the main body of the insect occupies 70% of the total area of ​​the picture is not met, the insect pictures must be cropped or supplemented again until this condition is met. The sources of insect pictures will at least include insect pictures obtained through web crawlers, insect pictures taken by agricultural equipment or farmers and plant protection team members, and insect pictures in public data sets. The collected insect pictures are calibrated with insect outer contours, and then the insect outer contours are captured as category insect pictures. The calibration conditions for the calibration process are: the area of ​​the main body of the insect occupies at least 70% of the area of ​​the insect outer contour, and the identification features of the corresponding insect category are retained.

[0088] Step 2: Perform sample enhancement processing on the original data set and divide it into a training data set, a validation data set, and a test data set. After the division, the number ratio of the training data set, the validation data set, and the test data set can be 8:1:1.

[0089] Step 3: Perform data enhancement processing on the training sample set to obtain an enhanced training sample set. The data enhancement processing here may include: padding to a square size, Mixup, brightness enhancement, saturation enhancement, contrast enhancement, random erasing, random scale cropping and uniform scaling to the same size, image flipping, and random angle tilting of images.

[0090] The training sample set is padded with black to a square size, and randomly cropped and scaled to a fixed size, such as uniformly scaling to 224x224 pixel values. Filling the image with black to a square size can effectively maintain the integrity of the main part of the image. If the image is simply scaled to a fixed size according to traditional image processing methods, the outline or shape of the insects in the training sample set will be deformed. For example, if a long insect is scaled to a square, it will lose its original morphological features, thereby increasing the probability of misidentification.

[0091] In addition, the area ratio and aspect ratio of the randomly cropped training sample set can be set to (0.7, 1) and (0.7, 1.3), respectively, to obtain more comprehensive local information about insects, so that the subsequent recognition effect can be better when facing incomplete images and insects of the same category with similar features.

[0092] While obtaining the insect image information, the size information of the insect image can also be obtained, which is converted using min-max normalization processing. The formula is as follows:

[0093]

[0094] Among them, the maximum and minimum scale values ​​are obtained by traversing the width and height of all training data sets.

[0095] Since in ordinary recognition tasks, different shooting distances will lead to different pixel sizes of different images to be identified, even for the same target, the obtained insect images will show different scales, so in this case, the size information of the insect image cannot be used as a feature for image recognition. However, in some recognition equipment in the agricultural field, the camera is fixed, and in order to facilitate shooting, the base plate used to shoot insects is also fixed, which leads to the same size of the images to be identified when the same camera or the same insect to be photographed is obtained. So in this case, using the size of the insect image as an important feature for recognition can improve the recognition accuracy of the model. In addition, in traditional deep learning training, pictures of different sizes are uniformly scaled to the same size so as to serve as the unified input size of the model. In natural scenes, there are insects of different sizes, but many of them have extremely similar appearance features and are all black in color. After being uniformly scaled to the same size, it is difficult for ordinary deep learning models to learn the difference between them. Therefore, using the original size of the insect image as a dimensional feature for recognition will improve the recognition accuracy of this situation.

[0096] In traditional identification, the circadian rhythm of insect activities cannot generally be considered as a feature for insect identification. In insect activities, such as hatching, pupation, emergence, mating, and egg laying, a rhythm is formed that is consistent with the law of day and night changes in nature. The activities of insects often show a certain rhythm of change with the alternation of day and night. This phenomenon is also called circadian rhythm. Sun-rising insects such as butterflies, dragonflies, ground beetles, and tiger beetles are all active during the day; nocturnal insects such as cutworms and most moths, underground pests, borers, blood-sucking mosquitoes, etc. are all active at night; diurnal insects such as ants are active both during the day and at night. Therefore, using time period information as an identification feature can effectively improve the recognition rate of insects in a specific time period.

[0097] The fourth step is to build a multi-dimensional insect recognition network pre-training model. First, an image feature extraction module is constructed. When selecting the structure, the characteristics of many deep learning model structures are referred to. The convolution structure has better generalization than the cutting-edge Transformer series, and the attention mechanism has better feature extraction effect on the texture information and semantic information of detail features. Therefore, this embodiment adopts a series of attention modules as the backbone structure for image feature extraction. An image feature extraction module is constructed, and the image feature extraction module includes an initial extraction Stem unit, an inverted residual unit and a head unit, wherein the initial extraction Stem unit includes a 3x3 convolutional layer and a first BN subunit; the inverted residual unit includes 16 inverted residual structures implemented based on the attention mechanism; the head unit includes a 1x1 convolutional layer, a second BN subunit, a pooling layer and a fully connected layer; the length and width of each category insect picture are normalized, and the output dimension is 2 dimensions to obtain a two-dimensional image; the time information feature of each category insect picture is obtained and one-hot encoded, and converted into a feature vector, and the feature vector is processed by network embedding to obtain a network embedding result, and used as an input feature, wherein the grid of the network embedding processing includes the first convolutional layer and the second convolutional layer, the dimension of the first convolutional layer is 1 x m1x 1 x 128, and the dimension of the second convolutional layer is 1 x 1 x128 xn, m1 represents the dimension of the time information feature, and the feature output dimension is n; a collection image feature extraction module and an image size fully connected layer are constructed, with an input dimension of k and an output dimension k that is the number of insect categories.

[0098] like Figure 2 As shown in the figure, the first Conv3x3 includes a 3x3 convolution layer and the first BN subunit, the padding parameter value is 1, and the number of channels is increased. The number of input channels and the number of output channels remain consistent with the previous Conv3x3. The second Conv3x3 is also composed of a 3x3 convolution layer and the first BN subunit, the padding parameter value is 1, and the number of channels is reduced. In the image feature post-processing stage after the 16 inverted residual Linear BottleNecks structures, after the 1x1 kernel size convolution layer, the second BN subunit and the pooling layer, it is converted to the input of the fully connected layer, and then the label is classified by the fully connected layer. The input dimension of the fully connected layer is 1280, and the output dimension is 100 dimensions for image feature representation; the length and width of each picture are obtained, and the length and width are normalized respectively, and the output dimension is 2; the shooting time information of the insect picture is obtained, divided into three time blocks of early, middle and late, and one-hot encoded before input into the network embedding. Through Figure 2 To demonstrate the structure of this model, Figure 3The structure of the attention mechanism module is presented. The output of the image feature is 100-dimensional, the image width and height are 2-dimensional, and the time information after network embedding is 10-dimensional. The image feature dimension, width and height dimension, and time information dimension are merged through the concat layer of deep learning. Because all features have become numerical features after processing, they can be directly merged and used as input features of the fully connected layer. The merged 112 dimensions are used as the final input dimension of the fully connected layer, and the output dimension is the number of categories of the trained insects. In addition, it is necessary to construct the ArcFace Loss loss function, Focal Loss loss function, and total loss function respectively;

[0099] The dimensions of insect images are converted into numerical features based on the ArcFace Loss function; the time period features of insect shooting are divided into time periods, and the division results are one-hot encoded and input into the multi-dimensional insect recognition network pre-training model to extract time features; based on the Focal Loss function, the enhanced training sample set is optimized to eliminate the long-tail distribution phenomenon that exists when imbalanced; based on the total loss function, whether the training results converge is determined;

[0100] The total loss function is:

[0101]

[0102] in, Represents the total loss function of the multi-dimensional insect recognition network model, Respectively represent ArcFaceLoss loss function and Focal Loss loss function;

[0103] The ArcFace Loss loss function is expressed as follows:

[0104]

[0105] in, represents the predicted output result, m represents the angle interval, and N represents the total sample size;

[0106] The Focal Loss loss function is expressed as follows:

[0107]

[0108] in, represents the output probability of the relevant category, Represents adjustable attention parameters.

[0109] Step 5: Randomly select 64 image samples and their corresponding image scales from the training set, input them into the multi-dimensional metric learning network, and use the SGD optimization algorithm to optimize and update the network parameters of the metric learning network; ArcFace is used as the loss function of the image embedding feature. In addition to expanding the distance between classes and reducing the distance within classes, it can also convert the image features into a specified dimensional representation, so as to facilitate cascading and merging with features of other dimensions to obtain a multi-dimensional insect feature representation. FocalLoss is used as the loss function for label classification to optimize the long-tail distribution phenomenon that exists when the training data is unbalanced;

[0110] Considering that the traditional Softmax Loss has a slow convergence speed and average final accuracy during the training process, we consider introducing the metric learning loss function ArcFace Loss to strengthen the distinction between different categories of insects. The recognition prediction result depends only on the angle between the feature and the weight, and is mapped to the feature space of the hypersphere. The additive angular margin parameter m is introduced to simultaneously enhance the intra-class compactness and inter-class differences, so as to improve the recognition accuracy of the model. Figure 4 It shows that ArcFace Loss is more effective in distinguishing different categories than the traditional Softmax Loss in the image recognition process.

[0111] In the multi-category insect dataset collected during the training process, it is inevitable that some insects are difficult to collect, resulting in a small number of training datasets for this category; some insects are easy to collect, resulting in a large number of training datasets for this category. In this embodiment, the number of insects in more categories is tens of thousands, and the number of insects in fewer categories is dozens. The dataset presents a more obvious long-tail distribution and data imbalance. Therefore, Focal Loss is introduced to give higher losses to insect categories with fewer numbers to improve their recognition rate.

[0112] Here, the ArcFace Loss loss function, insect size dimension, insect activity time dimension, and Focal Loss loss function respectively solve the problems of increasing the discrimination of similar samples, distinguishing insects of different sizes but with similar apparent characteristics, distinguishing insects with different living habits, and unbalanced long-tail distribution of training data samples.

[0113] Step 6: Determine whether the total loss of metric learning has converged, and save the best model based on the accuracy of the validation set.

[0114] Compared with the traditional convolutional network, the main improvements are: the training method of image features is different. The original method adds Softmax Loss after the final fully connected layer to train the label category, but the training speed of this method is slow and the accuracy is average. Therefore, the metric learning method is introduced to extract image embedding features, expand the distance between classes and reduce the distance within classes, and can also represent insects as specified dimensional features. In addition, the scale features of insects are added to the training as continuous numerical features. The model trained based on this method can have better discrimination between insects with different scales but similar appearance features; the activity time information of insects is converted from unstructured data to structural vector features that can assist in identification. The time dimension features obtained after network embedding conversion can have better discrimination for insects that come out and move in different time periods. In addition, this embodiment provides an architecture method for cascading multiple different dimensional insect information, and other information of insects can also be extracted through network embedding in a similar way to perform feature extraction for final classification. Finally, Focal Loss is added to the final label training, so that the imbalanced data of long-tail distribution has a better performance effect.

[0115] In this invention, the data set currently contains photos of 138 kinds of insects. These photos include various types of insects under different lighting conditions, different sizes, and different shooting angles. First, the training set is enhanced by flipping, rotating, Mixup image fusion, brightness enhancement, contrast enhancement and other data enhancement methods to increase the number of samples and increase the robustness of the model. Figure 5 An example of sample enhancement that may be used in the present invention is shown. All data are collected and sorted to form an original data set, and the original data set is evenly and randomly divided into a training data set, a validation data set, and a test data set in a ratio of 8:1:1.

[0116] The present invention is divided into two parts, training part and testing part. Among them, the development operating system of the training part can be Linux-ubuntu-20.04, the cpu model is Inter Core i7-8700, the memory size is 64GB, two NVIDIAGeForce RTX 3090s are used as gpus to support model training, the graphics card driver version is 430.40, the cuda version is 10.0, the cudnn version is cudnn-v7.3.1, the main development languages ​​are c language and python, and the opencv version 2.4.9 is used when configuring the convolutional network. Of course, other operating systems developed in this way can also be realized.

[0117] Based on the above improvements, the specific training process of the network model based on the attention mechanism is as follows: Figure 2The convolutional network is constructed with the following hyperparameters during training: the momentum parameter that affects the gradient descent to the optimal speed is set to 0.9; the weight decay regularization coefficient is set to 0.0005, which can effectively prevent overfitting; the learning rate is gradually reduced by warming up. The ArcFace Loss loss function and the Focal Loss loss function are constructed respectively, and the sum of the two is used as the total loss function for training. The best result of the validation set is used as the final model. The accuracy performance of the test set of various local insects is as follows: Figure 6 As shown, the structure of the network is shown as Figure 7 shown.

[0118] In other embodiments, the following steps are also included: performing insect target detection on the insect image to be classified, and then calling the trained insect recognition model to obtain the recognition result, see the attached Figure 8 As shown, the recognition result is finally fed back to the recognition software or APP on the mobile terminal or computer. Generally, the display result of the recognition software or APP on the mobile terminal or computer includes: insect target frame, insect recognition category, and the effect diagram is as follows Fig. 9 As shown, Fig. 9 The e13 category is the training label corresponding to the mole cricket in the training set.

[0119] In one embodiment, the insect sizes in the original data set can also be roughly classified based on deep learning according to the morphological characteristics of the insects, such as Lepidoptera, beetles and other major categories. Usually, the coarse classification can achieve extremely high accuracy because there are fewer classification categories and the similarity within the class is high. Then, the results of the coarse classification are further refined based on deep learning to classify the insects, such as the Chilo suppressalis, Sedge borer, and Corn borer under Lepidoptera. In this way, the hierarchical classification effect of insects can be achieved.

[0120] Embodiment 2:

[0121] An insect classification system based on multi-dimensional learning of convolutional networks, such as Fig.10 As shown, it includes a data set acquisition module 100, a division processing module 200, a model construction module 300, a model training module 400 and a recognition module 500;

[0122] The data set acquisition module 100 is used to acquire insect pictures containing various insect species, pre-process the insect pictures and retain the outer contours of the insects to form category insect pictures;

[0123] A division processing module 200 is used to divide the insect images into a training data set and a verification data set, and perform data enhancement processing on the training data set to obtain an enhanced training sample set;

[0124] A model building module 300 is used to build a multi-dimensional insect recognition network pre-training model, wherein the multi-dimensional insect recognition network pre-training model includes an image feature extraction module, a collection image feature extraction module, an image size fully connected layer, an ArcFace Loss loss function, a Focal Loss loss function and a total loss function;

[0125] The model training module 400 is configured to: train and optimize the multi-dimensional insect recognition network pre-training model based on the enhanced training sample set to obtain a multi-dimensional insect recognition network model;

[0126] The recognition module 500 is used to input the pictures to be recognized into the multi-dimensional insect recognition network model to obtain the recognition results, wherein the pictures to be recognized have the same size.

[0127] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0128] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0129] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] In addition, it should be noted that the shapes and names of the parts and components of the specific embodiments described in this specification may be different. Any equivalent or simple changes made based on the structure, features and principles described in the patent concept of the present invention are included in the protection scope of the patent of the present invention. The technicians in the technical field of the present invention can make various modifications or supplements to the specific embodiments described or replace them in a similar manner, as long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. An insect classification method based on multi-dimensional learning of convolutional networks, characterized in that: The following steps are involved: Obtaining insect pictures containing various insect species, preprocessing the insect pictures and retaining the outer contours of the insects to form category insect pictures; Dividing the insect images of the category into a training data set and a verification data set, and performing data enhancement processing on the training data set to obtain an enhanced training sample set; Constructing a multi-dimensional insect recognition network pre-training model, wherein the multi-dimensional insect recognition network pre-training model includes an image feature extraction module, a collection image feature extraction module, an image size fully connected layer, an ArcFace Loss loss function, a Focal Loss loss function and a total loss function; Based on the enhanced training sample set, the multi-dimensional insect recognition network pre-training model is trained and optimized to obtain a multi-dimensional insect recognition network model; Input the pictures to be identified into the multi-dimensional insect identification network model to obtain the identification results, wherein the picture sets to be identified have the same size; The construction of a multi-dimensional insect recognition network pre-training model comprises the following steps: Construct an image feature extraction module, which includes an initial extraction Stem unit, an inverted residual unit and a head unit, wherein the initial extraction Stem unit includes a 3x3 convolutional layer and a first BN subunit; the inverted residual unit includes 16 inverted residual structures implemented based on an attention mechanism; the head unit includes a 1x1 convolutional layer, a second BN subunit, a pooling layer and a fully connected layer; Normalize the length and width of each insect image, and output the dimension to 2D to obtain a 2D image. The temporal information features of each insect image of each category are obtained and one-hot encoded, converted into feature vectors, and the feature vectors are processed by network embedding to obtain network embedding results as input features, wherein the grid of the network embedding processing includes a first convolutional layer and a second convolutional layer, the first convolutional layer has a dimension of 1 x m1 x 1 x 128, and the second convolutional layer has a dimension of 1 x 1 x 128 xn, m1 represents the dimension of the temporal information feature, and the feature output dimension is n-dimensional; Construct a collection image feature extraction module and an image size fully connected layer, with an input dimension of k and an output dimension k equal to the number of insect categories; The training of the multi-dimensional insect recognition network pre-training model comprises the following steps: Randomly selecting a number of image samples from the enhanced training sample set, wherein the image samples also include corresponding image scale information; The image samples and the corresponding image scale information are input into the multi-dimensional insect recognition network pre-training model, and the network parameters of the multi-dimensional insect recognition network pre-training model are optimized and updated based on the SGD optimization algorithm; The dimensions of insect images are converted into numerical features based on the ArcFace Loss function; the time period features of insect photography are divided into time periods, and the division results are one-hot encoded and input into the multi-dimensional insect recognition network pre-training model to extract time features; Based on the Focal Loss function, the enhanced training sample set is optimized to eliminate the long-tail distribution phenomenon that exists when the training is unbalanced. Whether the training results have converged is determined based on the total loss function, and the accuracy of the training results is verified based on the verification data set to obtain a multi-dimensional insect recognition network model.

2. The insect classification method based on convolutional network multi-dimensional learning according to claim 1, characterized in that: The step of preprocessing the insect image and retaining the outer contour of the insect to form a classified insect image includes the following steps: Performing insect outer contour calibration processing on the insect image to obtain an outer contour calibration image, wherein the calibration conditions of the calibration processing are: the main body area of ​​the insect accounts for at least 70% of the outer contour area of ​​the insect, and the identification features of the corresponding insect category are retained; The outer contour of the insect is intercepted from the outer contour calibration image to obtain an insect outer contour image, which is used as a category insect image.

3. The insect classification method based on convolutional network multi-dimensional learning according to claim 1, characterized in that: The data enhancement processing includes: one or more of: resizing, mixup, brightness enhancement, saturation enhancement, contrast enhancement, random erasing, random scale cropping and uniform scaling to the same size, and image flipping.

4. The insect classification method based on convolutional network multi-dimensional learning according to claim 1, characterized in that: The following steps are also included: Construct ArcFace Loss loss function, Focal Loss loss function and total loss function respectively; The dimensions of insect images are converted into numerical features based on the ArcFace Loss function; the time period features of insect photography are divided into time periods, and the division results are one-hot encoded and input into the multi-dimensional insect recognition network pre-training model to extract time features; Based on the Focal Loss function, the enhanced training sample set is optimized to eliminate the long-tail distribution phenomenon that exists when the training is unbalanced. Determine whether the training results converge based on the total loss function.

5. The insect classification method based on convolutional network multi-dimensional learning according to claim 1, characterized in that: Using the multi-dimensional insect recognition network model to identify the image to be identified includes the following steps: Scale each image to be identified, and extract an image of the same size as the input image as the input image of the multi-dimensional insect identification network model; Obtain the length and width information of the image and perform normalization processing as the numerical features of the scale layer; Obtain the time information of insect activities in the image, convert it into corresponding time periods, and perform one-hot encoding and network embedding processing to extract time features; The output results of the multi-dimensional insect recognition network model are matched with the index of the insect category, and the category with the highest confidence is selected as the final recognition result.

6. The insect classification method based on convolutional network multi-dimensional learning according to claim 4, characterized in that: The total loss function is: in, Represents the total loss function of the multi-dimensional insect recognition network model, They represent ArcFace Loss loss function and Focal Loss loss function respectively; The ArcFace Loss loss function is expressed as follows: in, represents the predicted output result, m represents the angle interval, and N represents the total sample size; The Focal Loss loss function is expressed as follows: in, represents the output probability of the relevant category, Represents an adjustable attention parameter.

7. An insect classification system based on multi-dimensional learning of convolutional networks, characterized in that: It includes a data set acquisition module, a partitioning processing module, a model building module, a model training module and a recognition module; The data set acquisition module is used to acquire insect pictures containing various insect species, pre-process the insect pictures and retain the outer contours of the insects to form category insect pictures; The division processing module is used to divide the insect images of the category into a training data set and a verification data set, and perform data enhancement processing on the training data set to obtain an enhanced training sample set; The model construction module is used to construct a multi-dimensional insect recognition network pre-training model, wherein the multi-dimensional insect recognition network pre-training model includes an image feature extraction module, a collection image feature extraction module, an image size fully connected layer, an ArcFace Loss loss function, a Focal Loss loss function and a total loss function; The model training module is configured to: train and optimize the multi-dimensional insect recognition network pre-training model based on the enhanced training sample set to obtain a multi-dimensional insect recognition network model; The recognition module is used to input the pictures to be recognized into the multi-dimensional insect recognition network model to obtain the recognition results, wherein the pictures to be recognized have the same size; The construction of a multi-dimensional insect recognition network pre-training model comprises the following steps: Construct an image feature extraction module, which includes an initial extraction Stem unit, an inverted residual unit and a head unit, wherein the initial extraction Stem unit includes a 3x3 convolutional layer and a first BN subunit; the inverted residual unit includes 16 inverted residual structures implemented based on an attention mechanism; the head unit includes a 1x1 convolutional layer, a second BN subunit, a pooling layer and a fully connected layer; Normalize the length and width of each insect image, and output the dimension to 2D to obtain a 2D image. The temporal information features of each insect image of each category are obtained and one-hot encoded, converted into feature vectors, and the feature vectors are processed by network embedding to obtain network embedding results as input features, wherein the grid of the network embedding processing includes a first convolutional layer and a second convolutional layer, the first convolutional layer has a dimension of 1 x m1 x 1 x 128, and the second convolutional layer has a dimension of 1 x 1 x 128 xn, m1 represents the dimension of the temporal information feature, and the feature output dimension is n-dimensional; Construct a collection image feature extraction module and an image size fully connected layer, with an input dimension of k and an output dimension k equal to the number of insect categories; The training of the multi-dimensional insect recognition network pre-training model comprises the following steps: Randomly selecting a number of image samples from the enhanced training sample set, wherein the image samples also include corresponding image scale information; The image samples and the corresponding image scale information are input into the multi-dimensional insect recognition network pre-training model, and the network parameters of the multi-dimensional insect recognition network pre-training model are optimized and updated based on the SGD optimization algorithm; The dimensions of insect images are converted into numerical features based on the ArcFace Loss function; the time period features of insect photography are divided into time periods, and the division results are one-hot encoded and input into the multi-dimensional insect recognition network pre-training model to extract time features; Based on the Focal Loss function, the enhanced training sample set is optimized to eliminate the long-tail distribution phenomenon that exists when the training is unbalanced. Whether the training results have converged is determined based on the total loss function, and the accuracy of the training results is verified based on the verification data set to obtain a multi-dimensional insect recognition network model.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An insect classification device based on convolutional network multi-dimensional learning, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images