Tea garden green plant bug automatic detection method based on deep neural network

Abstract

The invention provides a tea garden green plant bug automatic detection method based on a deep neural network, and relates to the field of pest detection, comprising data acquisition, data processing, model training, autonomous evaluation and automatic fine tuning. The data acquisition adopts an unmanned aerial vehicle to scan tea trees in a tea garden to obtain scanning data; data processing: processing an image in the scanning data to obtain a normalized image; the model training comprises training data annotation, model construction and model training; the autonomous evaluation is performed through a contour coefficient; the automatic fine tuning comprises new data screening, automatic labeling and model fine tuning. According to the method, green plant bug pests can be automatically and efficiently monitored, manual intervention is not needed, manpower and material resources are effectively saved, interference of external environmental factors is avoided, and an accurate guiding effect is provided for the subsequent pest control process.

Description

Technical Field

[0001] The present invention relates to the technical field of insect pest detection, and in particular to an automatic detection method for green stink bugs in tea gardens based on a deep neural network. Background Art

[0002] The green stink bug belongs to the order Hemiptera and the family Miridae. Both its nymphs and adults can harm host plants, sucking the tender parts and reproductive organs of host plants through their mouth needles. For cotton, it harms its young buds, tender leaves, flower buds and young fruits, causing problems such as irregular holes on the leaves, broken and torn leaf edges, cessation of flower bud development, corky fruit surfaces and even rigidity and falling off. At the same time, in recent years, the damage caused by the green stink bug in my country's tea garden planting areas has shown an upward trend. Its (i.e. green stink bug) adults and nymphs like to eat young tea shoots, which causes brown necrotic spots to appear on the buds and leaves. As the damaged buds and leaves grow, countless holes form on the leaves and the leaf edges crack. In severe cases, the leaves may even twist and shrink, appear rough and old or deformed, and eventually present "broken leaves madness", affecting the yield and quality of the entire tea tree and even the entire planting area.

[0003] In order to ensure the yield and quality of tea, pest control is generally required during the planting process; however, the green stink bug is small in size (the adult body length is about 5-5.5mm and the width is about 2.2mm), the whole body is yellow-green, and it likes to live in hidden places. It usually moves and feeds on the buds in the early morning, dusk, cloudy days and other weather conditions, and moves to hide in the tea bushes when the sunlight is slightly stronger. Therefore, in the process of preventing and controlling the green stink bug pest, it is often an important problem to detect and identify it (i.e. the green stink bug), which directly hinders the prevention and control of the green stink bug pest. It is very easy for the external environment (such as light intensity, etc.) to interfere with the detection of pests, and it is impossible to accurately monitor or identify pests. The best pest control period is misplaced, and the tea trees are inevitably damaged, resulting in poor corresponding prevention and control quality, which greatly consumes the manpower and material resources for prevention and control and increases the cost of tea tree planting. Summary of the invention

[0004] In view of the problems existing in the above prior art, the object of the present invention is to provide an automatic detection method for green stink bugs in tea gardens based on deep neural network. The method can automatically and efficiently monitor green stink bug pests without human intervention, thereby effectively saving manpower and material resources, avoiding interference from external environmental factors, and providing accurate guidance for subsequent pest control processes.

[0005] The purpose of the present invention is achieved through the following technical solutions: An automatic detection method for green stink bugs in tea gardens based on deep neural networks, including data collection, data processing, model training, autonomous evaluation and automated fine-tuning; Among them, data collection includes: using drones to scan tea trees in tea gardens and obtain scanning data; Data processing is as follows: according to the altitude of the drone, the image in the scanned data is processed to obtain a normalized image; Model training includes training data annotation, model building and model training, which are used to obtain a model for detecting and identifying pest images; Autonomous evaluation is to calculate the contour coefficient of the prediction results in model training and realize autonomous evaluation; Automated fine-tuning includes new data screening, automatic labeling, and model fine-tuning. It achieves model adaptability by fine-tuning the parameters in model training using new data.

[0006] Based on the further optimization of the above scheme, the UAV is equipped with a high-definition camera and a multi-spectral camera, so as to obtain high-definition images and spectral images of the tea trees, and the field of view matrix of the high-definition camera and the multi-spectral camera overlap (achieved by the installation position and method of the high-definition camera and the multi-spectral camera on the UAV); at the same time, the UAV is respectively equipped with a light intensity sensor (for obtaining light intensity information) and a rangefinder (for obtaining the height information of the UAV from the tea tree directly below it).

[0007] Based on further optimization of the above scheme, the scanning data consists of multiple groups of data packets at different positions, each group of data packets includes a high-definition image, a spectral image of the same field of view (as the high-definition image), the corresponding drone altitude information (for obtaining the high-definition image), light intensity information, etc.; wherein the high-definition image is a three-channel RGB image, and the spectral image is a multi-channel image, and the number of channels is the number of different bands of the multi-spectral image.

[0008] Based on the further optimization of the above scheme, the steps of data processing are specifically as follows: First, calculate the scaling factor :

[0009] Where: h b Represents a fixed height value (the fixed height value is a hyperparameter obtained through a large amount of experimental data). By unifying the fixed height value, images taken by drones at different heights are restricted to a unified scale; h 0 indicates the drone altitude information value, obtained by the rangefinder; Then, according to the scaling factor, the width and height of the high-definition image and the multispectral image are scaled and transformed respectively; Finally, taking the center point of the image as the origin, the scaled high-definition image and multispectral image are intercepted along the width and height directions, and the width interception length is w , height intercept length is h , obtain the intercepted high-definition image and multi-spectral image; among which, the dimension of the intercepted high-definition image is , the dimension of the intercepted multispectral image is , n Indicates the number of bands of the multispectral image.

[0010] Based on the further optimization of the above solution, the model training is specifically as follows: Training data labeling: Label the high-definition image samples after data processing, and label the high-definition images into two categories: pest-0 and pest-free. Build the model: Build a deep neural network model including input layer, feature layer and classification layer. The input layer includes the illumination weight network and fusion network. The input parameters of each sample include high-definition pictures, multi-spectral images and light intensity. The feature layer uses lightweight and efficient mobileNet v3 The network structure represents the output features of the input layer. The output of the feature layer is connected to a fully connected layer and a custom normalization layer, so that the output feature is represented as a one-dimensional feature vector with an L2 norm of 1. The input of the fully connected layer is mobileNet v3 The output features are output as a one-dimensional feature vector. The custom normalization layer performs L2 normalization on the input one-dimensional feature vector and outputs it. The classification layer uses a classification network to represent the features output by the above feature layer as a fully connected layer and a softmax layer. The output of the softmax layer is directly used as the confidence of the model for the two labels of pest-0 and pest-free-1. Training model: Use the loss function as the supervision signal and use the gradient descent method to train the network model parameters. The loss function includes the feature representation loss function and the classification loss function.

[0011] Based on the further optimization of the above scheme, the illumination weight network is as follows: the illumination intensity is first obtained through the illumination intensity data layer to obtain the illumination intensity weight, and then through two different convolutional layers to obtain the illumination correction weight of the high-definition image and the illumination correction weight of the spectral image with the same dimensions as the high-definition image and the spectral image respectively; The light intensity data layer is a custom layer. Its input is the light intensity value of a batch, and its output is The light intensity weight, and the i indivual Each value in the weight is equal to the i The light intensity values; the convolution and parameters in the two convolutional layers are: the size of the first convolution kernel is , the number of convolution kernels is 3, and the size of the second convolution kernel is , the number of convolution kernels is n Two convolutional layers are used to learn the correction of high-definition images and multi-spectral images with different light intensities, thereby reducing the impact of different light intensities.

[0012] Based on the further optimization of the above scheme, the fusion network is specifically: , Spectral image illumination correction weight Respectively with high-definition images , Multispectral Image Perform a dot multiplication operation and concatenate the operation results along the channel dimension, and finally use the concatenated result as the output of the input layer:

[0013] Where: Represents the concatenation function.

[0014] Based on the further optimization of the above scheme, the input of the feature representation loss function is the output feature representation of the feature layer and the sample label, and the feature representation loss function protects the two steps of data pair construction and loss calculation; Data pair construction: First, according to the sample labels, the samples are divided into a positive set P and a negative set Q, with the number of elements in their sets being p and q respectively; then, each sample in the positive set P and each sample in the negative set Q are constructed into a data pair, with a total of pq data pairs and similarity labels all being 0, forming different data pairs; each sample in the positive set P is combined with each other to construct data pairs, with a total of p(p-1) data pairs and similarity labels all being 1, forming positive similar data pairs; each sample in the negative set Q is combined with each other to construct data pairs, with a total of q(q-1) data pairs and similarity labels all being 1, forming negative similar data pairs; then, according to the principle of positive and negative example balance and positive and negative similar data pair balance, the total number of data pairs is obtained N :

[0015] At the same time, the number of different data pairs is obtained N 0: , the number of similar data pairs of positive examples N 1: , the number of negative similar data pairs N 2: ; Finally, random sampling is performed according to the number of data pairs required for each set until the required data pairs are met; The loss calculation is as follows: Based on the constructed data pairs, the loss is calculated using the contrast loss function:

[0016] Where: D W Represents the Euclidean distance between two sample features; Y Indicates whether two samples match. Y=1 When , it means that the two samples are similar or matched. Y=0 When , it means the two samples do not match; m Indicates the set threshold; M represents the number of samples;

[0017] Right now D W Representation sample X 1 and sample X The Euclidean distance between 2, R Represents the characteristic dimension of the sample.

[0018] Based on the further optimization of the above scheme, the input of the classification loss function is the classification confidence and sample label output by the classification layer, and the classification loss function adopts the cross entropy loss function:

[0019] Where: U Indicates the probability of pests - 0; Indicates the probability of no pests -1.

[0020] Based on the further optimization of the above scheme, the specific steps of the autonomous assessment are as follows: First, all samples were divided into two categories: "pest-infested" and "pest-free" according to the model prediction results; Then, the average distance between a sample and all samples of the same category is taken as the aggregation degree of the sample. a (i) :

[0021] Where: Indicates v The samples and u The Euclidean distance between samples; J Indicates the number of samples of the same category; After that, the minimum distance between the sample and all samples of different categories is taken as the aggregation degree of the sample b (i) :

[0022] Where: G Indicates the number of samples in different categories; Finally, the silhouette coefficient of each sample is obtained s(i) :

[0023] The silhouette coefficient is used to reflect the quality of the model's prediction and classification effect. The closer its value is to 1, the better the classification effect is, and the closer it is to -1, the worse the model classification effect is. The mean silhouette coefficient of all samples is taken as the total silhouette coefficient to evaluate the classification results of the model.

[0024] Based on the further optimization of the above solution, the specific steps of the automatic fine-tuning are as follows: New data screening: First, use the manually annotated training data and classify them into two categories: "pests" and "no pests" according to the annotated labels. Then calculate the normalized mean center point of the feature representation of samples in the two categories and use it as the focus:

[0025] Where: X Indicates category C i Feature representation of samples in Indicates category C i The number of samples in the represents the normalized formula; Then, based on the model's predicted classification results, the relationship between the new sample and the two focal points is compared to obtain the sample confidence:

[0026] Where: x(i) Indicates i The feature representation of a sample is y(i) Represents the predicted classification results of the classification model; conf (i) Indicates i The confidence level of the sample; e 0 、e 1 indicates the focus with and without pests, respectively; Finally, data samples whose confidence level is not less than a preset confidence level threshold are selected as training data; Automatic labeling: For newly added training sets, the predicted classification labels are used as training labels; Model fine-tuning: Use manually annotated training sets and newly added training sets, and use fine-tuning learning rates to fine-tune the parameters of the classification model obtained after model training, so as to obtain a classification model that adapts to new data.

[0027] Based on further optimization of the above solution, the confidence threshold is 2.

[0028] Based on the further optimization of the above scheme, the fine-tuning learning rate is 1x10 -5 .

[0029] The following are the technical effects of the technical solution of the present invention: This application combines high-definition images with multi-spectral images, and automatically learns the factors affecting light intensity, thereby effectively reducing the impact of environmental factors (i.e., external environmental light, etc.) on monitoring and identification accuracy, and improving the classification and identification accuracy of green busy pests; at the same time, this application uses drones to collect image data, and uses deep neural networks to detect and identify pests in real time and efficiently, thereby effectively saving labor costs, improving detection efficiency, and avoiding missing the best period for pest control. In addition, this application uses automated adaptive learning and adaptive fine-tuning of the model without excessive human intervention. First, it reduces the accumulation of errors generated during manual intervention, and second, it further reduces labor costs, thereby realizing self-updating and adjustment of the system, further improving the accuracy of detection and the adaptability of the method, and can be effectively applied to green stink bug pest detection in different environments and regions, thereby providing necessary guidance for pest control work and improving the yield and quality of crops. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention will be described clearly and completely below. In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are proposed to provide a thorough understanding of the embodiments of the present invention.

[0031] Embodiment 1: An automatic detection method for green stink bugs in tea gardens based on deep neural networks, including data collection, data processing, model training, autonomous evaluation and automated fine-tuning; Among them, data collection is: using a drone to scan the tea trees in the tea garden and obtain scanning data; wherein the drone is equipped with a high-definition camera and a multi-spectral camera (the conventional high-definition camera and multi-spectral camera in this field can be used, and no specific limitation is made in this embodiment), so as to obtain high-definition images and spectral images of the tea trees, and the field of view matrix of the high-definition camera and the multi-spectral camera overlap (realized by the installation position and method of the high-definition camera and the multi-spectral camera on the drone); at the same time, the drone is respectively equipped with a light intensity sensor (used to obtain light intensity information, using the common light intensity sensor model in this field, and no specific limitation is made in this embodiment) and a rangefinder (used to obtain the height information of the drone from the tea tree directly below it, using the rangefinder carried by the conventional drone in this field, such as an infrared rangefinder, etc.). The scanning data consists of multiple groups of data packets at different positions, each group of data packets includes a high-definition image, a spectral image of the same field of view (as the high-definition image), the corresponding drone height information (obtaining the high-definition image), light intensity information, etc.; wherein the high-definition image is a three-channel RGB image, and the spectral image is a multi-channel image, and the number of channels is the number of different bands of the multi-spectral image.

[0032] Data processing is as follows: according to the height of the drone, the image in the scanned data is processed to obtain a normalized image; the specific steps are as follows: First, calculate the scaling factor :

[0033] Where: h b Represents a fixed height value (the fixed height value is a hyperparameter obtained through a large amount of experimental data). By unifying the fixed height value, images taken by drones at different heights are restricted to a unified scale; h 0 indicates the drone altitude information value, obtained by the rangefinder; Then, according to the scaling factor, the width and height of the high-definition image and the multispectral image are scaled and transformed respectively; Finally, taking the center point of the image as the origin, the scaled high-definition image and multispectral image are intercepted along the width and height directions, and the width interception length is w , height intercept length is h , obtain the intercepted high-definition image and multi-spectral image; among which, the dimension of the intercepted high-definition image is , the dimension of the intercepted multispectral image is , n Indicates the number of bands of the multispectral image.

[0034] Model training includes training data annotation, model construction and model training, which are used to obtain a model for detecting and identifying pest images; specifically, it includes: Training data annotation: Annotate the high-definition image samples after data processing (using existing conventional annotation tools, such as labelimg The high-definition images are labeled as having pests (0) and not having pests (1); Construct the model: Construct a deep neural network model including input layer, feature layer and classification layer; the input layer includes illumination weight network and fusion network, and the input parameters of each sample include HD picture, multispectral image and illumination intensity; the illumination weight network is as follows: the illumination intensity is first obtained through the illumination intensity data layer to obtain the illumination intensity weight, and then through two different convolutional layers to obtain the illumination correction weight of HD picture and spectral image with the same dimensions as HD image and spectral image respectively; the illumination intensity data layer is a custom layer, and its input is a batch The light intensity value is output as The light intensity weight, and the i indivual Each value in the weight is equal to the i The light intensity values; the convolution and parameters in the two convolutional layers are: the size of the first convolution kernel is , the number of convolution kernels is 3, and the size of the second convolution kernel is , the number of convolution kernels is n Two convolutional layers are used to learn the correction of high-definition images and multi-spectral images with different light intensities, thereby reducing the impact of different light intensities.

[0035] The fusion network is specifically: the high-definition picture illumination correction weight , Spectral image illumination correction weight Respectively with high-definition images , multispectral image Perform a dot multiplication operation and concatenate the operation results along the channel dimension, and finally use the concatenated result as the output of the input layer:

[0036] Where: Represents the concatenation function.

[0037] The feature layer uses lightweight and efficient mobileNet v3 The network structure represents the output features of the input layer. The output of the feature layer is connected to a fully connected layer and a custom normalization layer, so that the output feature is represented as a one-dimensional feature vector with an L2 norm of 1. The input of the fully connected layer is mobileNet v3 The output features are output as a one-dimensional feature vector. The customized normalization layer performs L2 normalization on the input one-dimensional feature vector and outputs it.

[0038] The classification layer uses a classification network to represent the features output by the above feature layer as a fully connected layer and a softmax layer, where the output of the fully connected layer is batch*2*1 The output of the softmax layer is directly used as the model's confidence in the two labels of pest-0 and pest-free-1. Training model: Use the loss function as the supervision signal and use the gradient descent method to train the network model parameters. The loss function includes the feature representation loss function and the classification loss function.

[0039] The input of the feature representation loss function is the output feature representation of the feature layer and the sample label. The feature representation loss function protects the two steps of data pair construction and loss calculation; Data pair construction: First, according to the sample labels, the samples are divided into a positive set P and a negative set Q, with the number of elements in their sets being p and q respectively; then, each sample in the positive set P and each sample in the negative set Q are constructed into a data pair, with a total of pq data pairs and similarity labels all being 0, forming different data pairs; each sample in the positive set P is combined with each other to construct data pairs, with a total of p(p-1) data pairs and similarity labels all being 1, forming positive similar data pairs; each sample in the negative set Q is combined with each other to construct data pairs, with a total of q(q-1) data pairs and similarity labels all being 1, forming negative similar data pairs; then, according to the principle of positive and negative example balance and positive and negative similar data pair balance, the total number of data pairs is obtained N :

[0040] At the same time, the number of different data pairs is obtained N 0: , the number of similar data pairs of positive examples N 1: , the number of negative similar data pairs N 2: ; Finally, random sampling is performed according to the number of data pairs required for each set until the required data pairs are met; The loss calculation is as follows: Based on the constructed data pairs, the loss is calculated using the contrast loss function:

[0041] Where: D W Represents the Euclidean distance between two sample features; Y Indicates whether two samples match. Y =1 When , it means that the two samples are similar or matched. Y=0 When , it means the two samples do not match; m Indicates the set threshold; M represents the number of samples;

[0042] Right now D W Representation sample X 1 and sample X The Euclidean distance between 2, R Represents the characteristic dimension of the sample.

[0043] The input of the classification loss function is the classification confidence and sample label output by the classification layer. The classification loss function uses the cross entropy loss function:

[0044] Where: U Indicates the probability of pests - 0; Indicates the probability of no pests -1.

[0045] Self-evaluation is to calculate the contour coefficient of the prediction results in model training and realize self-evaluation. The specific steps are: First, all samples were divided into two categories: "pest-infested" and "pest-free" according to the model prediction results; Then, the average distance between a sample and all samples of the same category is taken as the aggregation degree of the sample. a (i) :

[0046] Where: Indicates v The samples and u The Euclidean distance between samples; J Indicates the number of samples of the same category; After that, the minimum distance between the sample and all samples of different categories is taken as the aggregation degree of the sample b (i) :

[0047] Where: G Indicates the number of samples in different categories; Finally, the silhouette coefficient of each sample is obtained s(i) :

[0048] The silhouette coefficient is used to reflect the quality of the model's prediction and classification effect. The closer its value is to 1, the better the classification effect is, and the closer it is to -1, the worse the model classification effect is. The mean silhouette coefficient of all samples is taken as the total silhouette coefficient to evaluate the classification results of the model.

[0049] When the model's classification effect on new data becomes worse and the error in monitoring the green stink bug pest in tea gardens becomes larger, automatic fine-tuning is needed, which includes new data screening, automatic labeling and model fine-tuning. By using new data to fine-tune the parameters in model training, the model's adaptive ability is achieved. The specific steps of automatic fine-tuning are: New data screening: First, use the manually annotated training data and classify them into two categories: "pests" and "no pests" according to the annotated labels. Then calculate the normalized mean center point of the feature representation of samples in the two categories and use it as the focus:

[0050] Where: X Indicates category C i Feature representation of samples in Indicates category C i The number of samples in the represents the normalized formula; Then, based on the model's predicted classification results, the relationship between the new sample and the two focal points is compared to obtain the sample confidence:

[0051] Where: x(i) Indicates i The feature representation of a sample is y(i) Represents the predicted classification results of the classification model; conf (i) Indicates i The confidence level of the sample; e 0 、e 1 indicates the focus with and without pests, respectively; Finally, data samples whose confidence level is not less than a preset confidence threshold (in this embodiment, the confidence threshold is 2) are selected as training data; Automatic labeling: For newly added training sets, the predicted classification labels are used as training labels; Model fine-tuning: Use the manually annotated training set and the newly added training set to fine-tune the parameters of the classification model obtained after model training using the fine-tuning learning rate (the fine-tuning learning rate is one tenth of the pre-training learning rate. In this embodiment, the fine-tuning learning rate is 1x10 -5 ), thereby obtaining a classification model that adapts to new data.

[0052] Embodiment 2: As another preferred embodiment of the present application, a method for automatic detection of green stink bugs in tea gardens based on a deep neural network, based on the scheme of Example 1, a method for fine-tuning the parameters of the green stink bug pest classification model using a fine-tuning learning rate is specifically: a parameter fine-tuning model is adopted, and the parameter initialization value of the parameter fine-tuning model is the parameter value of the classification model, and the learning rate is the fine-tuning learning rate (the size of the learning rate determines the speed of model parameter learning, that is, the speed of gradually approaching the optimal solution), including forward propagation of training data (the conventional forward propagation process in this field can be adopted, and is not specifically limited in this embodiment), loss calculation (error calculation, the conventional loss calculation method in this field can be adopted), and back propagation (according to the error and the set learning rate, the gradient descent method is adopted to update the model network parameters); The above process is repeated continuously with different data to complete the iterative training of the model.

[0053] Embodiment 3: As another preferred embodiment of the present application, a tea garden green stink bug detection system includes a drone equipped with a high-definition camera, a multi-spectral camera, a light intensity sensor, an infrared rangefinder, and a temperature and humidity sensor, wherein the field of view matrix of the high-definition camera and the multi-spectral camera overlap; at the same time, the drone and the remote control module perform wireless transmission, and the remote control module integrates the data acquisition method, data processing method, model training method, autonomous evaluation method, and automated fine-tuning method in Example 1; the drone adopts a multi-rotor drone commonly used in the field and the drone has an even number of rotors.

Claims

1. A method for automatic detection of green stink bugs in tea gardens based on deep neural network, characterized in that: Including data collection, data processing, model training, autonomous evaluation and automated fine-tuning; Among them, data collection includes: using drones to scan tea trees in tea gardens and obtain scanning data; Data processing is as follows: according to the altitude of the drone, the image in the scanned data is processed to obtain a normalized image; Model training includes training data annotation, model building and model training, which are used to obtain a model for detecting and identifying pest images; Autonomous evaluation is to calculate the contour coefficient of the prediction results in model training and realize autonomous evaluation; Automated fine-tuning includes new data screening, automatic labeling, and model fine-tuning. It achieves model adaptability by fine-tuning the parameters in model training using new data.

2. The method for automatic detection of green stink bugs in tea gardens based on a deep neural network according to claim 1, characterized in that: The drone is equipped with a high-definition camera and a multi-spectral camera, so as to obtain high-definition images and spectral images of tea trees, and the field of view matrix of the high-definition camera and the multi-spectral camera overlap; at the same time, the drone is respectively equipped with a light intensity sensor and a rangefinder.

3. A method for automatically detecting green stink bugs in tea gardens based on a deep neural network according to claim 1 or 2, characterized in that: The scanning data consists of multiple groups of data packets at different locations, each group of data packets includes a high-definition image, a spectral image of the same field of view, corresponding drone altitude information, light intensity information, etc.; wherein the high-definition image is a three-channel RGB image, and the spectral image is a multi-channel image, and the number of channels is the number of different bands of the multi-spectral image.

4. The method for automatic detection of green stink bugs in tea gardens based on deep neural network according to any one of claims 1 to 3, characterized in that: The steps of data processing are specifically as follows: First, calculate the scaling factor : Where: h b Indicates a fixed height value; h 0 indicates the drone altitude information value, obtained by the rangefinder; Then, according to the scaling factor, the width and height of the high-definition image and the multispectral image are scaled and transformed respectively; Finally, taking the center point of the image as the origin, the scaled high-definition image and multispectral image are intercepted along the width and height directions, and the width interception length is w , height cut length is h , obtain the intercepted high-definition image and multi-spectral image; among which, the dimension of the intercepted high-definition image is , the dimension of the intercepted multispectral image is , n Indicates the number of bands of the multispectral image.

5. The method for automatic detection of green stink bugs in tea gardens based on deep neural network according to any one of claims 1 to 4, characterized in that: The model training is specifically as follows: Training data labeling: Label the high-definition image samples after data processing, and label the high-definition images into two categories: pest-0 and pest-free. Build the model: Build a deep neural network model including input layer, feature layer and classification layer. The input layer includes the illumination weight network and fusion network. The input parameters of each sample include high-definition pictures, multi-spectral images and light intensity. The feature layer uses lightweight and efficient mobileNet v3 The network structure represents the output features of the input layer. The output of the feature layer is connected to a fully connected layer and a custom normalization layer, so that the output feature is represented as a one-dimensional feature vector with an L2 norm of 1. The input of the fully connected layer is mobileNet v3 The output features are output as a one-dimensional feature vector. The custom normalization layer performs L2 normalization on the input one-dimensional feature vector and outputs it. The classification layer uses a classification network to represent the features output by the above feature layer as a fully connected layer and a softmax layer. The output of the softmax layer is directly used as the confidence of the model for the two labels of pest-0 and pest-free-1. Training model: Use the loss function as the supervision signal and use the gradient descent method to train the network model parameters. The loss function includes the feature representation loss function and the classification loss function.

6. The method for automatic detection of green stink bugs in tea gardens based on deep neural network according to claim 5, characterized in that: The illumination weight network is as follows: the illumination intensity is first obtained through the illumination intensity data layer to obtain the illumination intensity weight, and then through two different convolutional layers to obtain the illumination correction weight of the high-definition image and the illumination correction weight of the spectral image with the same dimensions as the high-definition image and the spectral image respectively; The light intensity data layer is a custom layer. Its input is the light intensity value of a batch, and its output is The light intensity weight of i indivual Each value in the weight is equal to the i The light intensity values; the convolution and parameters in the two convolutional layers are: the size of the first convolution kernel is , the number of convolution kernels is 3, and the size of the second convolution kernel is , the number of convolution kernels is n Two convolutional layers are used to learn the correction of high-definition images and multi-spectral images with different light intensities, thereby reducing the impact of different light intensities.

7. The method for automatic detection of green stink bugs in tea gardens based on deep neural network according to claim 5, characterized in that: The fusion network specifically includes: , Spectral image illumination correction weight Respectively with high-definition images , multispectral image Perform a dot multiplication operation and concatenate the operation results along the channel dimension, and finally use the concatenated result as the output of the input layer: Where: Represents the concatenation function.

8. The method for automatic detection of green stink bugs in tea gardens based on deep neural network according to claim 5, characterized in that: The input of the feature representation loss function is the output feature representation of the feature layer and the sample label. The feature representation loss function protects the two steps of data pair construction and loss calculation. Data pair construction: First, according to the sample labels, the samples are divided into a positive set P and a negative set Q, with the number of elements in their sets being p and q respectively; then, each sample in the positive set P and each sample in the negative set Q are constructed into a data pair, with a total of pq data pairs and similarity labels all being 0, forming different data pairs; each sample in the positive set P is combined with each other to construct data pairs, with a total of p(p-1) data pairs and similarity labels all being 1, forming positive similar data pairs; each sample in the negative set Q is combined with each other to construct data pairs, with a total of q(q-1) data pairs and similarity labels all being 1, forming negative similar data pairs; then, according to the principle of positive and negative example balance and positive and negative similar data pair balance, the total number of data pairs is obtained N : At the same time, the number of different data pairs is obtained N 0: , the number of similar data pairs of positive examples N 1: , the number of negative similar data pairs N 2: ; Finally, random sampling is performed according to the number of data pairs required for each set until the required data pairs are met; The loss calculation is as follows: Based on the constructed data pairs, the loss is calculated using the contrast loss function: Where: D W Represents the Euclidean distance between two sample features; Y Indicates whether two samples match. Y=1 When , it means that the two samples are similar or matched. Y=0 When , it means the two samples do not match; m Indicates the set threshold; M represents the number of samples; Right now D W Representation sample X 1 and sample X The Euclidean distance between 2, R Represents the characteristic dimension of the sample.

9. The method for automatic detection of green stink bugs in tea gardens based on deep neural network according to claim 5, characterized in that: The input of the classification loss function is the classification confidence and sample label output by the classification layer. The classification loss function adopts the cross entropy loss function: Where: U Indicates the probability of pests-0; Indicates the probability of no pests -1.

Images