Mosquito detection method based on image fragment information difference factor

Abstract

The invention provides a mosquito detection method based on an image fragment information difference factor, and the method comprises the steps: obtaining a mosquito image, constructing a mosquito image data set, and carrying out the preprocessing of the mosquito image data set; performing data enhancement on the preprocessed mosquito image data set by applying an unbalanced learning algorithm, and dividing into a training set, a verification set and a test set; constructing an image recognition network model based on an improved CeiT algorithm; initializing an image recognition network model, setting training parameters, and training an image recognition network by using the divided training set; inputting the images of the test set into the trained image recognition network model, and predicting image recognition and classification results; and transplanting the trained image recognition network model to a Jetson Xavier NX embedded development platform, and carrying out recognition verification by using a verification set. According to the method provided by the invention, the recognition precision and speed of fine-grained mosquitoes which have complex backgrounds and are difficult to distinguish are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular, to a mosquito detection method based on an image segment information difference factor. Background Art

[0002] Problems such as small differences among mosquitoes and complex environments make it difficult to identify them. Therefore, developing a mosquito detection device with fine-grained recognition ability is of great significance for protecting ecological security.

[0003] Currently, the fine-grained mosquito detection technology mainly faces the following difficulties:

[0004] (1) Sample imbalance problem: The number of some mosquito samples is small, and the network model trained with them focuses on the majority class, and the recognition of minority class samples is more blurred;

[0005] (2) High similarity between classes problem: The differences between mosquito classes are small, and the key features for promoting recognition are difficult to capture. Therefore, some classes with relatively high similarity are prone to confusion;

[0006] (3) Complex background problem: The living environment of mosquitoes is relatively complex, and background information has a great interference on the capture of sample features. However, traditional mosquito recognition methods have many defects. First, through the inspection method of manual observation, most mosquitoes are hidden in the environment and are not easy to find. In addition, for multiple classes of mosquitoes with very high similarity but different control methods, only through manual inspection is time-consuming and laborious and prone to errors.

[0007] With the development of artificial intelligence technology, mosquito recognition methods based on deep learning have been widely developed. Currently, the most commonly used fine-grained recognition method based on deep learning is the two-stage recognition method of localization-recognition. The localization-recognition method divides the fine-grained image recognition into two parts: discriminative region localization and fine-grained feature learning in the region. When performing discriminative region localization, the convolutional feature response of a deep neural network is usually utilized in a strongly supervised or weakly supervised manner; while in fine-grained feature learning, features are separately extracted from each located region, and the features are combined together and finally classified. The strongly supervised method means that: during model training, in order to obtain better classification accuracy, in addition to the class label of the image, additional artificial annotation information such as object bounding boxes and part annotations is also used; the weakly supervised method uses means such as attention mechanisms and clustering to automatically discover discriminative regions, does not require component annotations, and can complete training only with classification labels. Because the strongly supervised method has a high labor cost and is time-consuming and laborious, the weakly supervised learning method has become the first choice for detection devices. Summary of the Invention

[0008] According to the technical problems proposed above, a mosquito detection method based on the information difference factor of image segments is provided. The present invention is based on the improved CeiT for mosquito image recognition. By converting the RGB image into a vector dimension and adopting a high-quality preprocessing technology and a noise suppression method to generate a robust sample with the least noise; by performing deep fine-grained feature extraction on the input image and selecting different extraction methods for different categories of image segments; by identifying through the difference in the amount of information of image segments and eliminating the background segments in the image segments. The present invention can achieve high-precision detection of fine-grained mosquitoes in complex backgrounds.

[0009] The technical means adopted by the present invention are as follows:

[0010] A mosquito detection method based on the information difference factor of image segments, comprising:

[0011] S1. Obtain mosquito images, construct a mosquito image data set, and preprocess the mosquito image data set;

[0012] S2. Apply an imbalanced learning algorithm to perform data augmentation on the preprocessed mosquito image data set, and divide it into a training set, a validation set, and a test set;

[0013] S3. Based on the improved CeiT algorithm, construct an image recognition network model;

[0014] S4. Initialize the image recognition network model, set training parameters, and use the divided training set to train the image recognition network;

[0015] S5. Input the images in the test set into the trained image recognition network model to predict the image recognition and classification results;

[0016] S6. Transplant the trained image recognition network model to the Jetson Xavier NX embedded development platform and use the validation set for recognition verification.

[0017] Further, step S1 specifically includes:

[0018] S11. Collect mosquito images from Google and Kaggle platforms;

[0019] S12. Crop the collected mosquito images, and the pixel size after cropping is 224×224 to complete the construction of the data set.

[0020] Further, step S2 specifically includes:

[0021] S21. Perform feature fusion on the data set, obtain a sample feature map through the convolutional layer, and perform denoising to eliminate the noise generated by the convolutional layer;

[0022] S22. Highlight the feature information of the samples through the attention layer and convert the processed image samples into vector dimensions;

[0023] S23. Use the K-means algorithm to cluster the features into K clusters, and select some clusters for the data augmentation algorithm;

[0024] S24. Perform SMOTE processing on the selected oversampling clusters and further eliminate noise to obtain robust samples with the least noise;

[0025] S25. Divide the complete dataset into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0026] Further, step S3 specifically includes:

[0027] S31. Use the CeiT network as the feature extraction network;

[0028] S32. Embed a fine-grained feature extraction module that fuses the CNN and Transformer structures in the feature extraction network;

[0029] S33. Embed a background removal module based on the information difference factor determination algorithm before the classifier.

[0030] Further, step S32 specifically includes:

[0031] S321. Input the image fragments (p1, p2, p3... p n ) after shallow feature extraction processing into the fine-grained feature extraction module;

[0032] S322. Initially calculate the information difference between the information before and after the input of the image fragment in the block layer structure and the output of the block layer structure. The calculation formula is as follows:

[0033]

[0034] Among them, IDF represents the information difference; dist(·) represents the Euclidean distance calculation symbol; X, Y represent any two samples in the sample set; x i , y i represent two one-dimensional vectors obtained by flattening the previous step samples to a length of n;

[0035] S323. Represent the information difference IDF as I i , i = 1, 2, 3,... n, and divide it into two groups according to the size of the information difference IDF, as follows:

[0036]

[0037] Among them, a + b = n;

[0038] S324. Introduce a hyperparameter M, where 0 < M < 1, and select the M with the smallest information difference for the n blocks to perform the Multi-Head Attention operation, and perform the convolutional neural network operation on the remaining M n blocks;

[0039] S325. Segment each image patch in the divided group to extract fine-grained features.

[0040] Further, step S33 specifically includes:

[0041] S331. Calculate the information difference IDF of each image patch passing through the block layer structure, and accumulate the information difference IDF layer by layer to obtain the information difference factor IF of the entire network i , and the calculation formula is as follows:

[0042]

[0043] where depth represents the network depth; p i,j represents the information volume of the jth image patch in the ith layer of the network; p i-1,j represents the information volume of the jth image patch in the (i - 1)th layer of the network;

[0044] S332. Structurally organize and classify the information difference factor IF according to the hyperparameter M i and select the largest M n blocks for removal to eliminate low-information background blocks.

[0045] Further, step S4 specifically includes:

[0046] S41. Initialize the backbone CeiT network using the pre-trained weights on ImageNet;

[0047] S42. During the network training process, use Adam with Weight Decay as the optimizer, set the momentum factor to 0.9, and the weight decay to 0.05;

[0048] S43. Set the initial learning rate to 0.0005 and the batch size to 64.

[0049] Further, step S5 specifically includes:

[0050] S51. Define the number of network model parameters as follows:

[0051] (K_h * K_w * C_in) * C_out

[0052] Among them, \(K_h\) and \(K_w\) represent the input size of the convolutional kernel, \(C_{in}\) represents the number of layers of the input feature map, and \(C_{out}\) represents the number of layers of the output feature map;

[0053] S52. Define the floating-point operation number parameter of the network model as follows:

[0054] (\(K_h * K_w * C_{in} * C_{out}\)) * (\(H_{out} * W_{out}\))

[0055] Among them, \(H_{out}\) and \(W_{out}\) respectively represent the size of the output feature map;

[0056] S53. Calculate the precision rate, and the calculation formula is as follows:

[0057]

[0058] Among them, \(TP\) represents the number of true positive samples; \(FP\) represents the number of false positive samples; \(FN\) represents the number of false negative samples.

[0059] Furthermore, the Jetson Xavier NX embedded development platform in step S6 uses the Ubuntu18.04LTS operating system and deploys the PyTorch deep learning framework and the OpenCV library.

[0060] Furthermore, the Jetson Xavier NX embedded development platform in step S6 comes with a CSI camera interface and selects the IMX219 camera to capture specimen images.

[0061] Compared with the prior art, the present invention has the following advantages:

[0062] 1. A mosquito detection method based on the image segment information difference factor provided by the present invention designs an image processing method based on vector dimension for the problem of imbalanced learning. The generated samples retain the main recognition features of the original category and reasonably change some recognition features. It not only solves the problem of imbalanced mosquito data but also improves the robustness of the network.

[0063] 2. A mosquito detection method based on the image segment information difference factor provided by the present invention designs a new fine-grained feature extraction module for the problem of difficult recognition of fine-grained features. This module is divided into two methods according to the amount of information for extracting fine-grained features. This enables the distinction of highly similar species among mosquitoes.

[0064] 3. A mosquito detection method based on the information difference factor of image segments provided by the present invention designs a new background removal method for the problem of the complex and changeable living environment of mosquitoes. This method uses the information change of each patch after passing through the network as the standard for eliminating background interference.

[0065] 4. A mosquito detection method based on the information difference factor of image segments provided by the present invention uses a trained network model on the Jetson Xavier NX embedded development platform and uses an IMX219 camera to collect sample images to achieve real-time monitoring of mosquitoes.

[0066] For the above reasons, the present invention can be widely promoted in the fields of image recognition and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0068] Figure 1 It is a flow chart of the method of the present invention.

[0069] Figure 2 It is a sample diagram of each category in the mosquito image dataset provided by the embodiment of the present invention.

[0070] Figure 3 It is a schematic diagram of the improved CeiT object detection network provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0071] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0072] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0073] As Figure 1 shown, the present invention provides a mosquito detection method based on the image segment information difference factor, including:

[0074] S1. Obtain mosquito images, construct a mosquito image data set, and preprocess the mosquito image data set;

[0075] S2. Apply an imbalance learning algorithm to perform data augmentation on the preprocessed mosquito image data set, and divide it into a training set, a validation set and a test set;

[0076] S3. Based on the improved CeiT algorithm, construct an image recognition network model;

[0077] S4. Initialize the image recognition network model, set training parameters, and use the divided training set to train the image recognition network;

[0078] S5. Input the images in the test set into the trained image recognition network model to predict the image recognition and classification results;

[0079] S6. Transplant the trained image recognition network model to the Jetson Xavier NX embedded development platform and use the validation set for recognition verification.

[0080] Specifically, as a preferred embodiment of the present invention, step S1 specifically includes:

[0081] S11. Collect mosquito images from Google and Kaggle platforms;

[0082] S12. Crop the collected mosquito images, and the pixel size after cropping is 224×224 to complete the construction of the data set, as Figure 2 shown.

[0083] Specifically, as a preferred embodiment of the present invention, step S2 specifically includes:

[0084] S21. Perform feature fusion on the data set, obtain the sample feature map through the convolutional layer, and perform denoising to eliminate the noise generated by the convolutional layer;

[0085] S22. Highlight the feature information of the sample through the attention layer, and convert the processed image sample into a vector dimension;

[0086] S23. Use the K-means algorithm to cluster the features into K clusters, and select some clusters for the data augmentation algorithm;

[0087] S24. Perform SMOTE processing on the selected oversampling clusters and further eliminate noise to obtain robust samples with the least noise;

[0088] S25. Divide the complete data set into a training set, a validation set, and a test set according to the ratio of 8:1:1.

[0089] When specifically implemented, as a preferred implementation manner of the present invention, step S3 specifically includes:

[0090] S31. As shown in Figure 3 , use the CeiT network as the feature extraction network;

[0091] S32. Embed a fine-grained feature extraction module that fuses the CNN and Transformer structures in the feature extraction network;

[0092] S33. Embed a background removal module based on the information difference factor determination algorithm before the classifier.

[0093] When specifically implemented, as a preferred implementation manner of the present invention, step S32 specifically includes:

[0094] S321. Input the image fragments (p1, p2, p3... p n ) after shallow feature extraction processing into the fine-grained feature extraction module;

[0095] S322. Initially calculate the information difference value between the information before and after the input of the image fragment (patch) to the block layer structure and the output of the block layer structure. The calculation formula is as follows:

[0096]

[0097] Among them, IDF represents the information difference value; dist(·) represents the Euclidean distance calculation symbol; X, Y represent any two samples in the sample set; x i , y i represent two one-dimensional vectors obtained by flattening the previous step samples to a length of n;

[0098] S323. Represent the information difference value IDF as Ii , where \(i = 1, 2, 3, \cdots, n\), and they are divided into two groups according to the magnitude of the information difference IDF as follows:

[0099]

[0100] where \(a + b=n\);

[0101] S324. Introduce a hyperparameter \(M\), \(0 < M < 1\), and perform Multi - Head Attention (MSA) operation on the \(M\) n blocks with the smallest information difference, and perform Convolutional Neural Network (CNN) operation on the remaining \(M\) n blocks;

[0102] S325. Segment each image patch in the divided groups to extract fine - grained features. For example, \(p1\) is divided into \((p\) 11 , \(p\) 12 , \(p\) 13 … \(p\) 1k ), where \(k = 16\); then perform fine - grained extraction processing on the mini - patches in the two groups respectively, and finally splice them onto the patch.

[0103] Specifically in implementation, as a preferred implementation manner of the present invention, step S33 specifically includes:

[0104] S331. Calculate the information difference IDF of each image patch through the block layer structure, and accumulate the information difference IDF layer by layer to obtain the information difference factor IF of the entire network i , and the calculation formula is as follows:

[0105]

[0106] where \(depth\) represents the network depth; \(p\) i,j represents the information volume of the \(j\) - th image patch in the \(i\) - th layer of the network; \(p\) i-1,j represents the information volume of the \(j\) - th image patch in the \((i - 1)\) - th layer of the network;

[0107] S332. Structure and classify the information difference factor IF according to the hyperparameter \(M\) i , and select the largest \(M\) n blocks for removal to eliminate low - information background blocks.

[0108] Specifically in implementation, as a preferred implementation manner of the present invention, step S4 specifically includes:

[0109] S41. Initialize the backbone CeiT network using the pre - trained weights on ImageNet;

[0110] S42. During the network training process, Adam with Weight Decay (adamw) is used as the optimizer, the momentum factor is set to 0.9, and the weight decay is set to 0.05;

[0111] S43. The initial learning rate is set to 0.0005, and the batch size is set to 64.

[0112] Specifically, as a preferred embodiment of the present invention, step S5 specifically includes:

[0113] S51. Define the number of network model parameters as follows:

[0114] (K_h*K_w*C_in)*C_out

[0115] Among them, K_h and K_w represent the input size of the convolution kernel (kernel), C_in represents the number of layers of the input feature map, and C_out represents the number of layers of the output feature map;

[0116] S52. Define the floating-point operation count (FLOPs) as follows:

[0117] (K_h*K_w*C_in*C_out)*(H_out*W_out)

[0118] Among them, H_out and W_out respectively represent the size of the output feature map;

[0119] S53. Calculate the precision, and the calculation formula is as follows:

[0120]

[0121] Among them, TP represents the number of true positive samples; FP represents the number of false positive samples; FN represents the number of false negative samples.

[0122] Specifically, as a preferred embodiment of the present invention, the Jetson Xavier NX embedded development platform in step S6 uses the Ubuntu 18.04 LTS operating system, and deploys the PyTorch deep learning framework and the OpenCV library. In this embodiment, the model is trained on the host side, and then the generated model weight file is transferred to the embedded development platform.

[0123] In specific implementation, as a preferred implementation manner of the present invention, the Jetson Xavier NX embedded development platform in step S6 comes with a CSI camera interface, and an IMX219 camera (with a resolution of 3280*2464 pixels and a 77-degree wide angle) is selected to capture specimen images. In this embodiment, in order to realistically simulate the actual environment, the camera is used to capture images of crop pests hidden between branches and leaves, and the captured images are preprocessed, including operations such as scaling and normalization; finally, the preprocessed images are input into the image recognition network model to predict the results.

[0124] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mosquito detection method based on image segment information difference factor, characterized in that: include: S1. Acquire mosquito images, construct a mosquito image dataset, and preprocess the mosquito image dataset; S2. Apply the unbalanced learning algorithm to perform data enhancement on the preprocessed mosquito image dataset and divide it into training set, validation set and test set; S3. Build an image recognition network model based on the improved CeiT algorithm; S4, initializing the image recognition network model, setting training parameters, and using the divided training set to train the image recognition network; S5, input the images of the test set into the trained image recognition network model to predict the image recognition and classification results; S6. Port the trained image recognition network model to the Jetson Xavier NX embedded development platform and use the validation set for recognition verification.

2. The mosquito detection method based on image segment information difference factor according to claim 1, characterized in that: Step S1 specifically includes: S11, collect mosquito images from Google and Kaggle platforms; S12, cropping the collected mosquito images, the pixel size after cropping is 224×224, and the construction of the data set is completed.

3. The mosquito detection method based on image segment information difference factor according to claim 1, characterized in that: Step S2 specifically includes: S21, perform feature fusion on the data set, obtain the sample feature map through the convolution layer, and perform denoising to eliminate the noise generated by the convolution layer; S22, highlight the characteristic information of the sample through the attention layer, and convert the processed image sample into a vector dimension; S23, using K-means algorithm to cluster the features into K clusters, and select some clusters for data enhancement algorithm; S24, performing SMOTE processing on the selected oversampled clusters and further eliminating noise to obtain robust samples with minimum noise; S25. Divide the complete dataset into training set, validation set and test set in a ratio of 8:1:

1.

4. The mosquito detection method based on image segment information difference factor according to claim 1, characterized in that: Step S3 specifically includes: S31, using CeiT network as feature extraction network; S32, embedding a fine-grained feature extraction module that integrates CNN and Transformer structures into the feature extraction network; S33, embedding a background removal module based on an information difference factor determination algorithm before the classifier.

5. The mosquito detection method based on image segment information difference factor according to claim 4, characterized in that: Step S32 specifically includes: S321, the image segments (p1, p2, p3...p n ) is input into the fine-grained feature extraction module; S322, performing an initial calculation on the information difference of the image segment before the block layer structure is input and after the block layer structure is output, and the calculation formula is as follows: Where IDF represents information difference; dist(·) represents the Euclidean distance calculation symbol; X and Y represent any two samples in the sample set; x i ,y i Indicates that the previous step samples are flattened into two one-dimensional vectors of length n; S323, the information difference IDF is represented as I i , i = 1, 2, 3, ···n, and divide them into two groups according to the size of the information difference value IDF, as follows: Where a+b=n; S324. Introduce a hyperparameter M, where 0 < M < 1, and perform the Multi-Head Attention operation on the M blocks with the smallest information difference, and perform the convolutional neural network operation on the remaining (1 - M) blocks; n Perform the Multi-Head Attention operation on the M blocks with the smallest information difference, and perform the convolutional neural network operation on the remaining (1 - M) blocks; n Perform the convolutional neural network operation on the remaining (1 - M) blocks; S325 , segment each image segment in the divided group and extract fine-grained features.

6. The mosquito detection method based on image segment information difference factor according to claim 4, characterized in that: Step S33 specifically includes: S331, calculate the information difference value IDF of each image segment in the block layer structure, accumulate the information difference value IDF layer by layer, and obtain the information difference factor IF of the entire network i , the calculation formula is as follows: Among them, depth represents the network depth; p i,j represents the information content of the jth image fragment of the i-th layer network; p i-1,j represents the amount of information of the jth image fragment in the i-1th layer network; S332, according to the hyperparameter M information difference factor IF i Row structuring and classification, select the maximum M n Blocks are removed to eliminate low-information background blocks.

7. The mosquito detection method based on image segment information difference factor according to claim 1, characterized in that: Step S4 specifically includes: S41. Initialize the backbone CeiT network using the pre-trained weights on ImageNet. S42. Adam with Weight Decay was used as the optimizer during network training, with the momentum factor set to 0.9 and the weight decay set to 0.05; S43, the initial learning rate is set to 0.0005 and the batch size is set to 64.

8. The mosquito detection method based on image segment information difference factor according to claim 1, characterized in that: Step S5 specifically includes: S51, define the network model parameters as follows: (K_h*K_w*C_in)*C_out Among them, K_h and K_w represent the input size of the convolution kernel; C_in represents the number of layers of the input feature map; C_out represents the number of layers of the output feature map; S52. Define the floating-point operation number parameters of the network model as follows: (K_h*K_w*C_in*C_out)*(H_out*W_out) Among them, H_out and W_out represent the size of the output feature map respectively; S53, calculate the accuracy, the calculation formula is as follows: Among them, TP represents the number of true positive samples; FP represents the number of false positive samples; FN represents the number of false negative samples.

9. The mosquito detection method based on image segment information difference factor according to claim 1, characterized in that: The Jetson Xavier NX embedded development platform in step S6 uses the Ubuntu 18.04 LTS operating system and deploys the PyTorch deep learning framework and OpenCV library.

10. The mosquito detection method based on image segment information difference factor according to claim 1, characterized in that: In step S6, the Jetson Xavier NX embedded development platform has a built-in CSI camera interface, and the IMX219 camera is selected to capture the specimen image.

Images