A nighttime detection method for cicada nymphs based on lightweight convolutional neural network

By reducing network depth and width in the MobileNet-SSD model, the lightweight cicada nymph detection model is designed, which solves the problem of excessive model size and calculation volume in resource-constrained systems, and realizes efficient cicada nymph detection. Yu Jinchan's artificial breeding and robotic capture.

CN114519805BActive Publication Date: 2024-11-05NANTONG VOCATIONAL COLLEGE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210155415.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-21
Publication Date
2024-11-05
Estimated Expiration
2042-02-21

AI Technical Summary

Technical Problem

The existing nymph detection model of the Cicada nymph detection model based on convolutional neural network is difficult to deploy in embedded systems with limited resources. It is mainly due to the large size and computational volume, which affects its artificial breeding and robotic capture of Cicada. application.

Method used

By reducing the network depth and width based on the MobileNet-SSD model, three initial model structures (models M0, M1, M2) are designed to reduce the scale convolutional layer structure, reduce the network width and increase the depth of the middle and high-level convolutional layer, forming a Lightweight target detection model, suitable for the detection of nymphs of cicadas on trees at night.

Benefits of technology

While maintaining high recognition performance, the model size and calculation amount are greatly reduced, making it suitable for resource-constrained embedded real-time systems, providing a good foundation for artificial breeding and robot capture of cicadas, and significantly improving detection Speed ​​and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure BDA0003512343850000081
    Figure BDA0003512343850000081
  • Figure BDA0003512343850000101
    Figure BDA0003512343850000101
  • Figure BDA0003512343850000111
    Figure BDA0003512343850000111
Patent Text Reader

Abstract

The present invention discloses a method for detecting cicada nymphs at night based on a lightweight convolutional neural network. The method takes cicada nymph images taken in a natural environment and collected from the Internet as research objects, and divides them into training sets and validation sets after sample amplification. The method designs a model based on the MobileNet‑SSD framework to meet the requirements of miniaturization of mobile terminal detection models and lightweight calculation process, and obtains a lightweight target detection model suitable for detecting cicada nymphs on trees at night. The training parameters are set, and the trained model is obtained after multiple iterations. The validation set is input into the trained model to perform target detection on the images of cicada nymphs on trees. The lightweight model designed by the present invention maintains the model performance at a high level while greatly reducing the model size and its calculation amount, which helps to deploy the model in resource-constrained embedded systems, and lays a good foundation for the artificial breeding of cicadas and the capture of cicadas by robots.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of agricultural breeding, and in particular to a nighttime detection method for cicada nymphs based on a lightweight convolutional neural network. Background Art

[0002] The golden cicada belongs to the family of insects of Cicadidae. Its growth and development goes through three stages: eggs, nymphs, and adults. Every summer, the eggs begin to hatch, and the nymphs fall to the ground and dive into the soil. After 3 to 5 years of growth, they drill out of the ground, climb up the tree and become adults. They die after mating and laying eggs, and enter the next cycle. The commonly known cicada turtle or cicada monkey refers to the mature nymphs of the golden cicada that are ready to emerge. On the one hand, as an agricultural and forestry pest, the larvae and adults of the golden cicada suck sap from trees for survival. The adults will also insert their ovipositors into branches to lay eggs, causing the branches to die and endangering the healthy growth of the trees. On the other hand, the golden cicada is rich in fatty acids and amino acids, providing a variety of trace elements needed by the human body, playing a role in lowering cholesterol, improving endocrine, and preventing cardiovascular diseases. It has high edible and medicinal value and is increasingly favored by people. As an emerging breeding project, golden cicada breeding is being widely carried out in various regions of my country.

[0003] In recent years, with the rise of deep learning technology, convolutional neural networks have been widely used in modern agriculture. Image target detection refers to locating the target of interest from the image, accurately determining its category, and predicting the bounding box of each target. At present, target detection technology based on convolutional neural networks is mainly divided into two categories: one is the two-stage algorithm represented by Faster-RCNN. This type of algorithm first generates a series of candidate boxes as samples, and then classifies the samples through convolutional neural networks, which improves the detection accuracy on the basis of consuming part of the detection time. The other is the single-stage algorithm represented by YOLO and SSD. Single-stage detection does not require the extraction of candidate boxes, but directly takes the entire original image as input, regresses the specific location and category of pests and diseases in the image, and significantly improves the detection speed on the basis of slightly sacrificing the detection accuracy.

[0004] SSD (Single Shot MultiBox Detector) is a widely used high-performance single-stage target detection model. It combines the regression idea of ​​YOLO and the anchor boxes mechanism of Faster-RCNN, and adds the feature pyramid detection method of the RPN network, which better balances the detection accuracy and detection speed. The original SSD model uses VGG16 as the backbone network to extract features from images, but VGG16 has a large number of parameters, the model size reaches 500MB, and the model calculation amount reaches 30.49GFLOPs, which is difficult to deploy in embedded systems. MobileNet is a lightweight neural network that combines detection accuracy and speed. It changes the network's calculation method by constructing a depthwise separable convolution to reduce the number of network parameters while improving the model's detection speed. It can replace VGG16 as the backbone feature extraction network of SSD to form a MobileNet-SSD model structure. Summary of the invention

[0005] The purpose of the present invention is to provide a nighttime detection method for cicada nymphs based on a lightweight convolutional neural network, which greatly reduces the model size and the amount of calculation while maintaining a high recognition performance, so as to be suitable for resource-constrained embedded real-time systems, and lay a good foundation for the artificial breeding of cicadas and the capture of cicadas by robots.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for detecting cicada nymphs at night based on a lightweight convolutional neural network, comprising the following steps:

[0007] 1) The robot is simulated to detect cicada nymph targets at close range. The images of cicada nymphs on trees at night taken in natural environments and collected from the Internet are used as research objects. The original dataset is expanded, and the targets in the dataset are labeled using the LabelImg tool. The dataset is randomly split into a training set and a validation set at a ratio of 80% and 20%, and stored in the format of the PASCAL VOC dataset.

[0008] 2) Based on the MobileNet-SSD model structure, the model structure was streamlined and improved by reducing the network depth and width, and three initial model structures were designed;

[0009] 3) Using the established cicada nymph data training set, the training parameters of the three designed initial models were set respectively, and the trained models were obtained through multiple iterations;

[0010] 4) Input the validation set images into the trained model for performance testing, and select the best solution to obtain a lightweight target detection model suitable for detecting cicada nymphs on trees at night;

[0011] 5) The robot takes a close-up photo of the tree trunk area at night, inputs the captured image into a computer, and uses the detection model in the above steps to detect. If there are cicada nymphs on the tree trunk, the position detection frames of each target are superimposed on the output image; if there are no cicada nymphs on the tree trunk, the output returns to the original image.

[0012] Preferably, in the step 2), the three initial models are model M0, model M1, and model M2, wherein model M0 is based on the original MobileNet-SSD model by deleting the three small-scale convolutional layer structures of 3×3, 2×2, and 1×1 of the backbone network, reducing the network width, and increasing the depth of some mid- and high-level convolutional layers; model M1 is based on model M0, and the network width is further reduced; model M2 is based on model M1, and the depth of some mid- and high-level convolutional layers is reduced.

[0013] Preferably, the design steps of the model M0 include:

[0014] ① Delete the three small-scale convolutional layer structures of 3×3, 2×2, and 1×1 in the backbone network of the original MobileNet-SSD model to reduce the model size and computational complexity;

[0015] ② Reduce the network width, reducing the number of output channels of the first convolutional layer of the original MobileNet-SSD model from 32 to 24, and reducing the output channels of the subsequent convolutional layers 2, 3, 5, 9, and 13 to 48, 48, 96, 192, and 384 respectively;

[0016] ③. Increase the depth of some mid- and high-level convolutional layers, and perform 4 convolution operations on the convolutional layers with feature map sizes of 38×38, 19×19, and 10×10 to obtain model M0.

[0017] Preferably, the structure of the model M0 is as follows: convolution layers 1 to 2 perform two convolutions on the output feature map of the original model of 150×150, and the number of output channels is reduced to 24 and 48 respectively; convolution layers 3 to 4 perform two convolutions on the output feature map of the original model of 75×75, and the number of output channels is reduced from 128 to 48; convolution layers 5 to 8 perform four convolutions on the output feature map of the original model of 38×38, the depth of the convolution layer is increased from 2 layers to 4 layers, and the width of the convolution layer is reduced from 256 to 96; convolution layers 9 to 12 perform four convolutions on the output feature map of the original model of 19×19, the depth of the convolution layer is reduced from 6 layers to 4 layers, and the width of the convolution layer is reduced from 512 to 192; convolution layers 13 to 17 perform four convolutions on the output feature map of the original model of 10×10, the depth of the convolution layer is increased from 2 layers to 4 layers, and the width of the convolution layer is reduced from 1024 to 384.

[0018] Preferably, model M1 continues to reduce the network width based on the structure of model M0, reducing the number of output channels of the first convolutional layer of model M0 from 24 to 16, and reducing the number of output channels of convolutional layers 2, 5, 9, and 13 to 32, 64, 128, and 256, respectively.

[0019] Preferably, model M2 is based on model M1, keeps the overall architecture of model M1 unchanged, and reduces the number of convolutional layers of model M1 with feature map sizes of 38×38 and 19×19 from 4 to 2. At this time, the backbone feature network is reduced from 17 layers to 13 layers.

[0020] Beneficial effects:

[0021] The present invention discloses a method for constructing a cicada nymph detection model based on a convolutional neural network. Starting from the close-range cicada nymph target detection of a simulated robot, in order to adapt to the miniaturization of the model size and the lightweight calculation process, the model structure is improved and designed in combination with a single target detection task on the basis of the MobileNet-SSD model framework structure. The model size and its calculation amount are greatly reduced while the model performance is maintained at a high level, which is helpful to deploy the model in resource-constrained embedded systems and lays a good foundation for the artificial breeding of cicadas and the capture of cicadas by robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is an example of the enhancement effect on the original image data;

[0023] Figure 2 This is the structure diagram of the original MobileNet-SSD;

[0024] Figure 3 It is the structural diagram of model M0;

[0025] Figure 4 It is the structural diagram of model M1;

[0026] Figure 5 It is the structural diagram of model M2;

[0027] Figure 6 It is the loss function curve of the four models on the training set;

[0028] Figure 7 It is the loss function curve of the four models on the validation set;

[0029] Figure 8 It is the PR curve of the four models on the validation set;

[0030] Fig. 9 This is an example of the detection effect of night pictures in the validation set;

[0031] Fig.10 This is an example of the detection effect of a daytime image collected separately. DETAILED DESCRIPTION

[0032] The technical contents of the present invention are described in detail below in conjunction with the contents shown in the accompanying drawings.

[0033] The present invention discloses a method for constructing a cicada nymph detection model based on a convolutional neural network, comprising the following steps:

[0034] 1) The robot is simulated to detect cicada nymph targets at close range. The images of cicada nymphs on trees at night taken in natural environments and collected from the Internet are used as research objects. The original dataset is expanded, and the targets in the dataset are labeled using the LabelImg tool. The dataset is randomly split into a training set and a validation set at a ratio of 80% and 20%, and stored in the format of the PASCAL VOC dataset.

[0035] 2) Since most of the images are taken at close range, the size of the cicada nymphs in the images is similar and the target category is single, so few learning features are required. In order to facilitate the deployment of the model on resource-constrained mobile terminals, the optimization design is directly based on the MobileNet-SSD model. The model structure is streamlined and improved by reducing the network depth and network width, and three initial model structures are designed;

[0036] The three initial models are model M0, model M1, and model M2. Model M0 is based on the original MobileNet-SSD model by deleting the three small-scale convolutional layer structures of 3×3, 2×2, and 1×1 of the backbone network, reducing the network width, and increasing the depth of some mid- and high-level convolutional layers; Model M1 is based on model M0 by further reducing the network width; Model M2 is based on model M1 by reducing the depth of some mid- and high-level convolutional layers.

[0037] 3) Using the established cicada nymph data training set, the training parameters of the three designed initial models were set respectively, and the trained models were obtained through multiple iterations;

[0038] 4) Input the validation set images into the trained model for performance testing, and select the best solution to obtain a lightweight target detection model suitable for detecting cicada nymphs on trees at night;

[0039] 5) The robot takes a close-up photo of the tree trunk area at night, inputs the captured image into a computer, and uses the detection model in the above steps to detect. If there are cicada nymphs on the tree trunk, the position detection frames of each target are superimposed on the output image; if there are no cicada nymphs on the tree trunk, the output returns to the original image.

[0040] The model M0 is designed based on the original MobileNet-SSD model. The original MobileNet-SSD model structure performs border regression and classification on the Conv12, Conv14, Conv15, Conv16, Conv17, and Conv18 layers, and the corresponding resolutions are 19×19, 10×10, 5×5, 3×3, 2×2, and 1×1, respectively. The small feature maps of the last three layers are mainly used for large target object detection. Specifically, in the objective reality of the moderate target size in the present invention, if these three layers are removed during the adjustment process, there is almost no effect on the detection effect.

[0041] Therefore, when designing model M0, we first delete the three small-scale convolutional layer structures of 3×3, 2×2, and 1×1 in the backbone network of the original MobileNet-SSD model to reduce the model size and computational complexity.

[0042] Because the size of the model parameters and the amount of computation are mainly determined by the width of the network, the features that need to be learned for single target detection are greatly reduced. Therefore, the network width is reduced, and the number of output channels of the first convolutional layer of the original MobileNet-SSD model is reduced from 32 to 2. At the same time, the output channels of the subsequent convolutional layers 2, 3, 5, 9, and 13 are also reduced to 48, 48, 96, 192, and 384 respectively.

[0043] In order to maintain the nonlinear expression ability of the model while reducing the network width, the depth of some mid- and high-level convolutional layers was increased. After repeated adjustments, 4 convolution operations were performed on the convolutional layers with feature map sizes of 38×38, 19×19, and 10×10 to obtain model M0.

[0044] The structure of the model M0 is as follows Figure 3 As shown in the figure: convolution layers 1-2 perform two convolutions on the output feature map of the original model (150×150), and the number of output channels is reduced to 24 and 48 respectively; convolution layers 3-4 perform two convolutions on the output feature map of the original model (75×75), and the number of output channels is reduced from 128 to 48; convolution layers 5-8 perform four convolutions on the output feature map of the original model (38×38), the depth of the convolution layer increases from 2 layers to 4 layers, and the width of the convolution layer decreases from 256 to 96; convolution layers 9-12 perform four convolutions on the output feature map of the original model (19×19), the depth of the convolution layer decreases from 6 layers to 4 layers, and the width of the convolution layer decreases from 512 to 192; convolution layers 13-17 perform four convolutions on the output feature map of the original model (10×10), the depth of the convolution layer increases from 2 layers to 4 layers, and the width of the convolution layer decreases from 1024 to 384.

[0045] The detection network part of model M0 only performs target detection on 3-scale feature maps of 19×19, 10×10 and 5×5. At each pixel on the feature map, K (4, 6, 6) default detection boxes with different aspect ratios are generated, and the bounding boxes predicted by the model are trained based on these default detection boxes. The total number of default detection boxes is 19×19×4+10×10×6+5×5×6=2194. According to calculations, the number of M0 parameters has been greatly reduced to 845,000, the model size has been reduced to 3.23MB (Million Bytes), and the model calculation amount is 255.41MFLOPs (Million Floating-point Operations, MFLOPs).

[0046] For model M1, based on the structure of model M0, the network width is further reduced, the number of output channels of the first convolutional layer of model M0 is reduced from 24 to 16, and the number of output channels of convolutional layers 2, 5, 9, and 13 are also reduced to 32, 64, 128, and 256 respectively. The model structure is as follows Figure 4 As shown, after calculation, the parameters of M1 continued to decrease to 395,000, the model size further decreased to 1.51MB, and the model calculation amount continued to decrease to 125.57MFLOPs.

[0047] On the basis of obtaining model M1, in order to verify the necessity of increasing the depth of the mid- and high-level convolutional layers while reducing the network width, model M2 was designed based on M1 for comparison, that is, the overall architecture of model M1 was kept unchanged, and only the number of convolutional layers with feature map sizes of 38×38 and 19×19 in M1 was reduced from 4 to 2 layers, and the backbone feature network was reduced from 17 to 13 layers, thus obtaining model M2. According to calculations, the number of parameters was further reduced to 383,000, the model size was 1.46MB, and the model calculation amount was reduced to 99.8MFlops. Specific embodiments

[0049] The present invention is described in detail below with reference to specific test examples.

[0050] The dataset uses 366 images of cicada nymphs in a natural environment. The image collection location is on the campus of Jiangsu University. Since cicada nymphs begin to drill out of the ground and climb up nearby tree trunks to molt at 18:00-22:00 in the summer, a smooth transparent tape is placed around the tree trunk 1 meter above the ground to prevent it from climbing up. A mobile phone is held at a distance of about 0.5 to 1 meter from the tree trunk to simulate a robot near the tree trunk for close-up photography. Cicada nymphs on different tree trunks were photographed for several consecutive days. The shooting time included evening and night on sunny days. The shooting angles included front and side views. The number of cicadas in the pictures varied. In order to enrich the diversity of data sources, a total of 180 pictures collected in the green belt of the residential area in the same way and downloaded from the Internet were used as supplements. Then these data were merged, and the original dataset contained a total of 546 pictures. In order to reduce the overfitting of the model and increase the number of picture samples, data enhancement technology is used here. Data enhancement is to add random noise to the sample, interfere with the data, or generate new synthetic samples through certain rules. The present invention uses the open source python library imgaug (https: / / github.com / aleju / imgaug) to perform data enhancement on the cicada nymph picture. The enhancement methods include horizontal mirror flipping, vertical mirror flipping, random interception, adding noise, changing brightness and contrast, etc., and normalizing the size to 300×300 pixels. For the original picture, one of the above data enhancement techniques is randomly selected to enhance the image. After data enhancement and screening, some unclear pictures are eliminated, and the final data set contains 1030 pictures of cicada nymphs. The above data set is labeled with the LabelImg tool, and the data set is randomly split into a training set and a validation set at a ratio of 80% and 20%. The training set has 824 pictures of cicada nymphs and 2155 labeled samples with borders. The validation set has 206 pictures of cicada nymphs and 573 labeled samples with borders, which are stored in the PASCAL VOC data set format. Such a data set is obviously too small, and subsequent migration learning is carried out with the help of the VOC2007 data set.

[0051] like Figure 1 The figure shows an example of the enhancement effect of the original image data, where label 1 is the original image; label 2 is the horizontal flip of 1; label 3 is the enhanced contrast of 1; label 4 is the random crop of 1.

[0052] The basic working process of the SSD network model is as follows: First, SSD extracts features from the input image through the backbone network; then, it selects feature maps of different sizes from different convolutional layers of the backbone network, and then predicts multiple bounding boxes from feature maps of different sizes; finally, the position box and category of the target to be detected in the image are determined by non-maximum suppression (NMS). In the SSD network, large shallow feature maps are used to predict small targets, and small deep feature maps are used to predict large targets. However, the original SSD model uses VGG16 as the backbone network to extract features from images. VGG16 has a large number of parameters, the model size reaches 500MB, and the model calculation amount reaches 30.49GFLOPs, which cannot be applied in embedded systems. MobileNet-SSD has a better balance between detection speed and accuracy, and is more suitable for deployment in embedded systems. The backbone network parameters of MobileNet-SSD are shown in Table 1.

[0053] Table 1: Backbone network parameters of MobileNet-SSD

[0054] Network Layers Step Length type Filter size Output size 1 2 Convolutional Layer 3×3 150×150×32 2 1 Separable Convolution 3×3 150×150×64 3 2 Separable Convolution 3×3 75×75×128 4 1 Separable Convolution 3×3 75×75×128 5 2 Separable Convolution 3×3 38×38×256 6 1 Separable Convolution 3×3 38×38×256 7 2 Separable Convolution 3×3 19×19×512 8 1 Separable Convolution 3×3 19×19×512 9 1 Separable Convolution 3×3 19×19×512 10 1 Separable Convolution 3×3 19×19×512 11 1 Separable Convolution 3×3 19×19×512 12 1 Separable Convolution 3×3 19×19×512 13 2 Separable Convolution 3×3 10×10×1024 14 1 Separable Convolution 3×3 10×10×1024 15 2 Separable Convolution 3×3 5×5×512 16 2 Separable Convolution 3×3 3×3×256 17 2 Separable Convolution 3×3 2×2×256 18 2 Separable Convolution 3×3 1×1×128

[0055] As shown in Table 1, except for the first convolution layer Conv1 in the backbone network of MobileNet-SSD, the remaining convolution layers all use depthwise separable convolution. The principle is as follows: For a convolution layer, assume that the input feature map is H×W×M, where M is the number of input channels, H is the height of the input feature map, and W is the width of the input feature map. Taking the 3×3 convolution kernel as an example, the traditional convolution calculation amount is N×H×W×M×9. For depthwise separable convolution, its calculation amount is divided into two parts: one is depthwise convolution, whose calculation amount is H×W×M×9; the other part is pointwise convolution, whose calculation amount is H×W×M×N, so the ratio of the calculation amount of depthwise separable convolution to the calculation amount of traditional convolution can be expressed as:

[0056]

[0057] Where N is the number of output feature maps, usually N>>9. Therefore, the amount of computation using depthwise separable convolution is greatly reduced, about 1 / 9 of the traditional convolution. Therefore, in the subsequent model design process, the present invention uses depthwise separable convolution instead of traditional 3×3 convolution in both the classification layer of target detection and the regression layer of prediction frame, effectively reducing the amount of model computation.

[0058] Figure 2The structure diagram of the original MobileNet-SSD model corresponding to Table 1. In order to cope with targets of different sizes, 6 feature maps of different scales are selected in the original MobileNet-SSD structure for target detection, with sizes of 19×19, 10×10, 5×5, 3×3, 2×2, and 1×1. For feature maps of different scales, the anchor mechanism of Faster-RCNN is used to refer to each point on the feature map. K (4, 6, 6, 6, 6) default detection boxes are selected for target prediction. The total number of detection boxes on each image is 19×19×4+10×10×6+5×5×6+3×3×6+2×2×6+1×1×6=2278.

[0059] Generally speaking, network depth is defined as the number of layers in the network, and network width is defined as the number of channels in each layer. Generally speaking, deeper networks have better nonlinear expression capabilities and can learn more complex features. The width of the network determines the number of features that can be learned, and also dominates the model size and its computational complexity. On the one hand, if the width is not enough, the features are not sufficient and the network performance is limited. On the other hand, if the width is too wide, too many repeated features will be extracted, which will not contribute substantially to the improvement of network performance, but will increase the model computational complexity. Therefore, under the premise of keeping the network performance basically unchanged, designing a suitable streamlined structure is also one of the current research directions for lightweight detection models.

[0060] The task of this invention is to simulate the robot's single target close-range detection. Since most of the images are taken at close range, the size of the golden cicada nymphs in the images is similar and the target category is single, and few learning features are required. In order to facilitate the deployment of the model on resource-constrained mobile terminals, this invention improves the MobileNet-SSD structure based on the idea of ​​appropriately reducing the network depth and network width, and proposes three initial model structures:

[0061] The structure of model M0 is as follows Figure 3 The internal parameters are shown in Table 2. The specific structure is as follows: Convolutional layers 1 to 2 perform two convolutions on the output feature map of size 150×150, and the number of output channels is reduced from 32 and 64 to 24 and 48 respectively; Convolutional layers 3 to 4 perform two convolutions on the output feature map of size 75×75, and the number of output channels is reduced from 128 to 48; Convolutional layers 5 to 8 perform four convolutions on the output feature map of size 38×38, which is smaller than that of convolutional layers 1 to 2. Figure 2 , the depth of the convolution layer increases from 2 layers to 4 layers, and the width of the convolution layer decreases from 256 to 96; the convolution layers 9 to 12 perform four convolutions on the output feature map of size 19×19, which is relatively Figure 2, the depth of the convolution layer is reduced from 6 layers to 4 layers, and the width of the convolution layer is reduced from 512 to 192; the convolution layers 13 to 16 perform four convolutions on the output feature map of size 10×10, which is relatively Figure 2 , the depth of the convolutional layer increased from 2 layers to 4 layers, and the width of the convolutional layer decreased from 1024 to 384.

[0062] Table 2: Backbone network parameters of model M0

[0063] Network Layers Step Length type Filter size Output size 1 2 Convolutional Layer 3×3 150×150×24 2 1 Separable Convolution 3×3 150×150×48 3 2 Separable Convolution 3×3 75×75×48 4 1 Separable Convolution 3×3 75×75×48 5 2 Separable Convolution 3×3 38×38×96 6 1 Separable Convolution 3×3 38×38×96 7 1 Separable Convolution 3×3 38×38×96 8 1 Separable Convolution 3×3 38×38×96 9 2 Separable Convolution 3×3 19×19×192 10 1 Separable Convolution 3×3 19×19×192 11 1 Separable Convolution 3×3 19×19×192 12 1 Separable Convolution 3×3 19×19×192 13 2 Separable Convolution 3×3 10×10×384 14 1 Separable Convolution 3×3 10×10×384 15 1 Separable Convolution 3×3 10×10×384 16 1 Separable Convolution 3×3 10×10×384 17 2 Separable Convolution 3×3 5×5×384

[0064] The detection network part of M0 only performs target detection on 3-scale feature maps such as 19×19, 10×10 and 5×5. At each pixel on the feature map, K (4, 6, 6) default detection boxes with different aspect ratios are generated, and the bounding boxes predicted by the model are trained based on these default detection boxes. The total number of default detection boxes is 19×19×4+10×10×6+5×5×6=2194. According to calculations, the number of M0 parameters has been greatly reduced to 845,000, the model size has been reduced to 3.23MB (Million Bytes), and the model calculation amount is 255.41MFLOPs (Million Floating-point Operations, MFLOPs).

[0065] The structure of model M1 is as follows Figure 4 As shown in Table 3, its internal parameters are shown in Table 3. On the basis of obtaining model M0, for further investigation and comparison, the idea of ​​reducing network width is continued, and the number of output channels of the first convolutional layer of M0 is reduced from 24 to 16. Correspondingly, the number of output channels of convolutional layers 2, 5, 9, and 13 is also reduced to 32, 64, 128, and 256 respectively, and this model is identified as M1. After calculation, the parameters of M1 continue to be reduced to 395,000, the model size is further reduced to 1.51MB, and the amount of calculation continues to be reduced to 125.57MFLOPs.

[0066] Table 3 Internal parameters of model M1

[0067] Network Layers Step Length type Filter size Output size 1 2 Convolutional Layer 3×3 150×150×16 2 1 Separable Convolution 3×3 150×150×32 3 2 Separable Convolution 3×3 75×75×32 4 1 Separable Convolution 3×3 75×75×32 5 2 Separable Convolution 3×3 38×38×64 6 1 Separable Convolution 3×3 38×38×64 7 1 Separable Convolution 3×3 38×38×64 8 1 Separable Convolution 3×3 38×38×64 9 2 Separable Convolution 3×3 19×19×128 10 1 Separable Convolution 3×3 19×19×128 11 1 Separable Convolution 3×3 19×19×128 12 1 Separable Convolution 3×3 19×19×128 13 2 Separable Convolution 3×3 10×10×256 14 1 Separable Convolution 3×3 10×10×256 15 1 Separable Convolution 3×3 10×10×256 16 1 Separable Convolution 3×3 10×10×256 17 2 Separable Convolution 3×3 5×5×256

[0068] The structure of model M2 is as follows Figure 5As shown in Table 4, its internal parameters are shown in Table 4. Based on the model M1, in order to verify the necessity of increasing the depth of the convolution layer while reducing the network width, the model M2 is designed on the basis of M1 for comparison, that is, the overall architecture of the model M1 is kept unchanged, and only the number of convolution layers with the feature map size of M1 of 38×38 and 19×19 is reduced from 4 to 2. At this time, the backbone feature network is reduced from 17 to 13 layers, and this model is identified as M2. After calculation, the number of parameters is further reduced to 383,000, the model size is 1.46MB, and the calculation amount of the model is reduced to 99.8MFlops.

[0069] Table 4 Internal parameters of model M2

[0070]

[0071]

[0072] Several models are tested below. The software environment of the test is Ubuntu 16.04LTS 64-bit system, using the currently popular PyTorch (https: / / pytorch.org / ) deep learning open source framework. The computer memory is 16GB, equipped with Intel Core i5-8300 CPU, and GPU uses NVIDIA's GTX1050Ti to accelerate the deep learning model.

[0073] The present invention directly uses a small sample data set for training, which may lead to slow model convergence and unstable model. Therefore, the present invention uses a pre-trained model under the VOC2007 data set to adapt to the Jinchan target detection task through transfer learning to save model training costs and accelerate model convergence. In the experiment, the batch size of training and test data is set to 32, and traversing all the pictures in the training set once is called an epoch. The stochastic gradient descent optimization algorithm (SGD) is used when training the model, the momentum is set to 0.9, the initial learning rate is set to 0.01, and the learning rate decay strategy is the cosine annealing strategy. The specific training adopts the anaconda environment, the framework is pytorch1.0, and the training iteration is a total of 200 rounds.

[0074] The main indicators for evaluating the performance of the target detection model are F1 score, average precision (AP), and average intersection over Union (IoU). First, define the variables in the confusion matrix: TP, the number of true values ​​that are positive and predicted to be positive; TN, the number of true values ​​that are negative and predicted to be negative; FP, the number of true values ​​that are negative but predicted to be positive; FN, the number of true values ​​that are positive but predicted to be negative.

[0075] Precision is defined as follows:

[0076]

[0077] The precision rate is the proportion of correct predictions among the positive samples predicted by the classifier. Its value range is [0,1]. The larger the value, the better the model's prediction ability.

[0078] The recall rate is defined as follows:

[0079]

[0080] The recall rate, also known as the recall rate, is the ratio of the correct positive samples predicted by the classifier to all positive samples. Its value range is [0, 1]. The larger the value, the better the prediction ability of the model.

[0081] The F1 score is defined as follows:

[0082]

[0083] The definition of average accuracy (AP) is as follows:

[0084]

[0085] The intersection over union (IoU) is used to evaluate the accuracy of the predicted box and is defined as follows:

[0086]

[0087] Where A is defined as the predicted box given by the network, B is defined as the real box of the object, and the IoU threshold is set to 0.5 in the present invention.

[0088] The original MobileNet-SSD model and the three designed models were trained using the established golden cicada nymph dataset. The loss function curves of the four detection models in the training set and the validation set are shown in the following table. Figure 6 , Figure 7As shown in the figure, the four models have good performance on both the training set and the validation set. As the number of cycles increases, the models gradually converge. When the number of cycles reaches about 180, the models converge on both the training set and the validation set. Among them, the original MobileNet-SSD model is slightly better than the improved models M0 and M1.

[0089] Figure 8 The PR curves of the four models are given. The PR curve depicts the relationship between the model's accuracy and recall. The accuracy P and recall R are often a pair of contradictory metrics, while the average accuracy AP is an indicator that measures both P and R at the same time. It is the area under the PR curve and is used to measure the detection effect of the cicada nymph category. To compare the performance of the four models, Figure 8 The Break-Even Point (BEP) curve is drawn in the figure. The break-even point is the point on the PR curve where the precision and recall are equal. It can be seen that the break-even point of model M2 is about 0.9, and the break-even points of models M0 and M1 coincide with each other (0.93), which shows that the performance of M0 and M1 is very close, and both are slightly better than M2. The break-even point of the original MobileNet-SSD model is about 0.96, which is slightly better than models M0 and M1.

[0090] Table 5 comprehensively compares the parameter size, total computational complexity, average accuracy (AP), average intersection over union (Average IoU) and F1 score of the M0, M1, M2 and original MobileNet-SSD models. See Table 5 for the comparison results.

[0091] Table 5 Model parameters and performance comparison

[0092]

[0093] As shown in Table 5, the original MobileNet-SSD model performs best in terms of average intersection-over-union, which is about 1.5 and 1.9 percentage points higher than models M0 and M1, respectively. This shows that the original model is slightly better than M0 and M1 in positioning accuracy. The main reason is that the original model selects 6 feature maps of different scales to predict the target, which is more than the 3 scales of M0 and M1, but the effect is minimal. In terms of average accuracy AP, the original MobileNet-SSD model is only 0.1, 0.3 and 0.75 percentage points higher than M0, M1 and M2, respectively; in terms of F1 score, the original model is only 0.24, 0.37 and 0.8 percentage points higher than M0, M1 and M2, respectively. Overall, the performance of models M0 and M1 is basically the same as that of the original model, and the gap is not obvious. However, in terms of model computational complexity, M0 and M1 are far superior to the original model. The computational complexity of M1 is only 125.57MFLOPs, which is about 1 / 9 of the original model's 1130MFLOPs. In terms of model size, M0 and M1 are also much smaller than the original model. M1 is only 1.51MB, which is significantly reduced to about 1 / 10 of the original model's 15.2MB.

[0094] As shown in Table 5, the size of model M2 is only 1.46MB, and the amount of computation is only 99.8MFLOPs, which are the smallest among all models. However, the average intersection-over-union ratio is only 81.71%, which is about 3.7 percentage points lower than the original model. The positioning accuracy has dropped significantly, and the other indicators are also the lowest. The reason is that the convolution depth of model M2 in the feature map size of 38×38 and 19×19 convolution layer is shallow, and the extraction of complex features is not sufficient, resulting in a slight decline in performance indicators.

[0095] Looking at Table 5, for close-range target detection, the models M0 and M1 designed in this paper have obvious advantages in model size and computational complexity under the premise that the accuracy indicators are roughly the same. Among them, model M1 has a more balanced comprehensive performance in detection accuracy and speed, and is more in line with the application requirements in actual scenarios. Therefore, it is used as the final model design result of the present invention.

[0096] Table 6 shows the comparison of the detection speed of model M1 and the original model on GPU and CPU. Generally, mobile robots are mainly equipped with ordinary CPU laptops. The GPU of the present invention adopts NVIDIA GTX1050Ti, the CPU model is Intel Core i5-8300, the batch size is set to 1, and the processing speed on the two platforms is counted.

[0097] Table 6 Comparison of detection speed under different platforms

[0098]

[0099] As shown in Table 6, the parallel acceleration calculation of GPU greatly improves the detection speed. The GPU detection speed of model M1 is 179.30 frames per second, which is 2.58 times that of the original MobileNet-SSD; while the CPU detection speed of model M1 is 6.49 frames per second, which is 4.16 times that of the original MobileNet-SSD. It can be seen that regardless of the GPU or CPU platform, model M1 significantly improves the detection speed, and the processing speed of the CPU platform basically meets practical needs.

[0100] Fig. 9 In order to intuitively display the detection effect of the model of the present invention, several night images with different numbers, postures, lighting, backgrounds, climbing objects, etc. were selected and tested using model M1. Fig. 9 Some representative effect diagrams are given. Fig. 9 (a) Two separate cicada nymphs are sparsely distributed; Fig. 9 (b) contains densely packed numbers; Fig. 9 (c) shows two adhered targets in a side-view environment; Fig. 9 (d) shows images taken at different viewing angles to simulate target detection in different postures; the above images all have ideal lighting and clear contrast, and the detection difficulty is not high. Fig. 9 In (e), there is a certain degree of overexposure at the location of the target, and the contrast is slightly weaker; Fig. 9 (f) Some dry soil is attached to the back of the cicada. Fig. 9 (g) The surface of the tree trunk and the back of the cicada are similarly mottled brown. Figure 8 (h) is a cicada on a tree trunk after rain. The cicada nymphs in these three pictures are similar to the background area, showing different degrees of camouflage effect, but the model M1 can also successfully detect them. In addition, on other climbing backgrounds, Fig. 9 (i) Low shrubs were selected. Due to strong lighting, the target image was slightly blurred. Fig. 9 (j) A golden cicada in the molting state on the back of a large green leaf was selected, and M1 was tested successfully. The above results show that the lightweight model designed by the present invention achieves good target detection effects in many scenarios.

[0101] Fig.10 To further investigate the test effect of model M1 in non-nighttime environments, some scene images in daytime environments are selected for testing. The results are shown in Fig.10 Compared with night images, the background interference in daytime images is more significant. Since the cicadas have already shed their shells and flown away during the day, the model proposed in this paper can be used to detect the empty shells of cicadas to simulate the detection of cicada nymphs in the summer evening. Fig. 9(a) and 9(b) show the empty shell samples on the tree trunk on a sunny day and after rain. Fig. 9 (c) shows the empty shell samples on the cement wall of the residential area during the day, which can be successfully detected by model M1. In general, the model designed by the present invention has good anti-interference ability and good generalization effect.

[0102] The technical contents and technical features of the present invention have been disclosed as above. However, those skilled in the art may still make various substitutions and modifications based on the teachings and disclosures of the present invention without departing from the spirit of the present invention. Therefore, the protection scope of the present invention should not be limited to the contents disclosed in the embodiments, but should include various substitutions and modifications that do not depart from the present invention and are covered by the claims of this patent application.

Claims

1. A nighttime detection method for cicada nymphs based on a lightweight convolutional neural network, characterized in that The steps include: 1) The robot is simulated to detect cicada nymph targets at close range. The images of cicada nymphs on trees at night taken in natural environments and collected on the Internet are used as research objects. The original dataset is expanded, and the dataset is labeled with the LabelImg tool. The dataset is randomly split into a training set and a validation set at a ratio of 80% and 20%, and stored in the format of the PASCAL VOC dataset. 2) Based on the MobileNet-SSD model structure, the model structure was streamlined and improved by reducing the network depth and network width, and three initial model structures were designed. The three initial models are model M0, model M1, and model M2. Model M0 is based on the original MobileNet-SSD model by deleting the three small-scale convolutional layer structures of 3×3, 2×2, and 1×1 of the backbone network, reducing the network width, and increasing the depth of some mid- and high-level convolutional layers; Model M1 is based on model M0 by further reducing the network width; Model M2 is obtained by reducing the depth of some mid- and high-level convolutional layers on the basis of model M1; The design steps of the model M0 include: ① Delete the three small-scale convolutional layer structures of 3×3, 2×2, and 1×1 in the backbone network of the original MobileNet-SSD model to reduce the model size and computational complexity; ② Reduce the network width, reduce the number of output channels of the first convolutional layer of the original MobileNet-SSD model from 32 to 24, and reduce the output channels of the subsequent convolutional layers 2, 3, 5, 9, and 13 to 48, 48, 96, 192, and 384 respectively; ③. The depth of some mid- and high-level convolutional layers was increased. Four convolution operations were performed on the convolutional layers with feature map sizes of 38×38, 19×19, and 10×10, respectively, to obtain model M0. The structure of the model M0 is as follows: convolutional layers 1-2 perform two convolutions on the output feature map of the original model of 150×150, and the number of output channels is reduced to 24 and 48 respectively; convolutional layers 3-4 perform two convolutions on the output feature map of the original model of 75×75, and the number of output channels is reduced from 128 to 48; convolutional layers 5-8 perform four convolutions on the output feature map of the original model of 38×38, the depth of the convolutional layer increases from 2 layers to 4 layers, and the width of the convolutional layer decreases from 256 to 96; convolutional layers 9-12 perform four convolutions on the output feature map of the original model of 19×19, the depth of the convolutional layer decreases from 6 layers to 4 layers, and the width of the convolutional layer decreases from 512 to 192; convolutional layers 13-17 perform four convolutions on the output feature map of the original model of 10×10, the depth of the convolutional layer increases from 2 layers to 4 layers, and the width of the convolutional layer decreases from 1024 to 384; Based on the structure of model M0, model M1 further reduces the network width, reduces the number of output channels of the first convolutional layer of model M0 from 24 to 16, and reduces the number of output channels of convolutional layers 2, 5, 9, and 13 to 32, 64, 128, and 256 respectively; The model M2 is based on the model M1, keeps the overall architecture of the model M1 unchanged, and reduces the number of convolutional layers of the model M1 with feature map sizes of 38×38 and 19×19 from 4 to 2. At this time, the backbone feature network is reduced from 17 layers to 13 layers; 3) Using the established cicada nymph data training set, the training parameters of the three designed initial models were set respectively, and the trained models were obtained through multiple iterations; 4) Input the validation set images into the trained model for performance testing, and select the best solution to obtain a lightweight target detection model suitable for detecting cicada nymphs on trees at night; 5) The robot takes a close-up photo of the tree trunk area at night, inputs the captured image into the computer, and uses the detection model in the above steps to detect it. If there are cicada nymphs on the tree trunk, the position detection frames of each target are superimposed on the output image; if there are no cicada nymphs on the tree trunk, the output returns to the original image.