A YOLOv4 recognition method fusing coordinate attention mechanism and improved combined convolution

Abstract

The application discloses a YOLOv4 recognition method fusing a coordinate attention mechanism and a combined convolution, and is based on a YOLOv4 model for recognition, selects a GhostNet feature extraction network to replace a CSPDarkNet53 as a YOLOv4 backbone network, adopts a structure for generating a large number of feature maps through a small amount of calculation, uses a lower cost calculation to obtain redundant information in a feature layer, and reduces model parameter quantity and calculation quantity; a coordinate attention mechanism is introduced to enhance the extraction of fine-grained features of a target; an improved combined convolution module is designed to save calculation capacity, prevent effective feature loss, and improve recognition precision. The improved YOLOv4 model can reach a fruit recognition accuracy of 99.23%, and can meet the requirements of fruit recognition precision and speed in a natural environment, and provides protection for rapid and accurate operation of a picking robot.

Description

TECHNICAL FIELD

[0001] The application relates to an image recognition method, in particular to a YOLOv4 recognition method fusing a coordinate attention mechanism and an improved combined convolution. BACKGROUND

[0002] In recent years, a convolutional neural network is widely applied to target recognition and detection, and provides a new idea for fruit and vegetable recognition. The original image is used as input, corresponding features can be effectively learned from a large number of samples, and a complex feature extraction process is avoided, and only a small amount of manual participation is required in the whole process. Therefore, target recognition based on the convolutional neural network has become a new development trend of fruit and vegetable recognition.

[0003] The YOLOv4 network uses a CSPDarknet53 backbone network, and although the parameter amount can be reduced and the recognition accuracy can be improved, the model calculation is complex, a large memory space is consumed, the recognition accuracy and speed need to be improved, and it is difficult to meet the requirements of a picking robot on fast and accurate operation of target fruits in a complex environment.

[0004] Domestic and foreign scholars have recognized different types of fruits and vegetables, but it is difficult to apply to fruits and vegetables with complex branch growth, for example, the growth of dragon fruit branches is complex, resulting in problems such as occlusion and adhesion of dragon fruit, which brings great difficulty to target recognition of dragon fruit. How to quickly and accurately recognize fruits has become a difficult problem to be solved for realizing automatic picking of agricultural robots and improving work efficiency. SUMMARY

[0005] To solve the above technical problems, the application provides a YOLOv4 recognition method fusing a coordinate attention mechanism and an improved combined convolution, which can quickly and accurately recognize fruit and vegetable fruits.

[0006] To achieve the above technical purpose, the technical scheme adopted is as follows: a YOLOv4 recognition method fusing a coordinate attention mechanism and an improved combined convolution, comprising the following steps:

[0007] Step 1, selecting fruit and vegetable images in different environments to establish a data set;

[0008] Step 2, improving the target detection algorithm YOLOv4 to obtain an improved target recognition model including an input end, a backbone feature extraction network, a feature fusion unit and an output end, inputting the training samples selected from the data set into the established improved target recognition network for training, and the specific improvement includes:

[0009] The backbone feature extraction network adopts a GhostNet feature extraction network;

[0010] The feature fusion unit fuses the coordinate attention mechanism and the improved combined convolution module, the improved combined convolution module introduces separable convolution and residual edges in two traditional combined convolutions;

[0011] Step 3, using the improved target recognition model obtained to detect the photographed image, and identifying the fruits in the fruit and vegetable image.

[0012] Further, the conventional convolution module in the output end in step 2 is replaced by a Ghost Module.

[0013] Further, the two traditional combined convolutions are respectively a traditional combined convolution after the first up-sampling and a third traditional combined convolution.

[0014] Further, the specific method for the improved combined convolution module to introduce separable convolution in the two traditional combined convolutions is that the second convolution operation and the fourth convolution operation of the traditional combined convolution are replaced by separable convolution.

[0015] Further, the specific method for the improved combined convolution module to increase residual structure in the two traditional combined convolutions is that residual edges are added beside the first and second convolution operations of the traditional combined convolution.

[0016] Further, the specific method for the feature fusion unit to fuse the coordinate attention mechanism is that the coordinate attention mechanism is introduced between the second convolution operation and the third convolution operation of the traditional combined convolution.

[0017] The present application has the following advantages: the CSPDarknet53 backbone feature extraction network in YOLOV4 is replaced by a lightweight GhostNet feature extraction network, which uses lower cost calculation to obtain redundant information in the feature layer, reduces the model parameter quantity and calculation quantity, introduces the coordinate attention mechanism to enhance the extraction of fine-grained features of the target, designs the improved combined convolution module to save calculation capacity, prevent effective feature loss, and improve recognition accuracy, references the Ghost Module in the output port Yolo Head to improve calculation speed and reduce latency, reduces the parameter quantity and calculation quantity of the model, improves the running speed and detection efficiency of the model, and can be suitable for mobile terminal deployment, the improved target recognition network has an average precision improvement of nearly 5 percentage points compared with the traditional YOLOv4 network, the AP value can reach 98.94%, the accuracy can reach 99.23%, the model recognition speed can reach 37.2 frames per second, and the weight size is 59.4MB, which can meet the demand for fruit recognition accuracy and speed in a natural environment, and provides protection for the rapid and accurate operation of the picking robot. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a photographed image of pitaya under different conditions in a greenhouse environment;

[0019] Figure 2 is a traditional YOLOv4 network structure diagram;

[0020] Figure 3 is a comparison diagram of conventional convolution and Ghost Module;

[0021] Figure 4 is an improved combination convolution-CA module diagram;

[0022] Figure 5 is a coordinate attention mechanism structure diagram;

[0023] Figure 6 is an improved combination convolution-CA module diagram fused;

[0024] Figure 7 is a pitaya target detection network flowchart;

[0025] Figure 8 is a recognition result diagram of different algorithms for pitaya fruits in three scenes. DETAILED DESCRIPTION

[0026] The technical solutions of the present application are described in detail below in combination with the drawings and specific embodiments. The specific content listed in the following examples is not limited to the technical features necessary for the technical problems to be solved by the technical solutions recited in the claims. At the same time, the listed examples are only a part of the present application, not all examples.

[0027] Taking the recognition of pitaya fruits as an example, the YOLOv4 recognition method of the present application fusing the coordinate attention mechanism and the improved combination convolution is improved on the basis of the traditional YOLOv4 recognition algorithm, including the following steps:

[0028] Step 1, selecting pitaya images in different environments to establish a data set;

[0029] Step 2, improving the target detection algorithm YOLOv4 to obtain an improved target recognition model including an input end, a backbone feature extraction network Beakbone, a feature fusion unit Neck and an output end YOLO Head, inputting the training samples selected from the data set into the established improved target recognition network for training, and the specific improvements of the improved target recognition model based on the YOLOv4 recognition algorithm include:

[0030] The backbone feature extraction network Beakbone adopts the GhostNet feature extraction network to reduce the model calculation amount, generate more feature mappings and realize fast target recognition;

[0031] The feature fusion unit Neck fuses the coordinate attention mechanism and the improved combined convolution module, the improved combined convolution module introduces separable convolution and residual edge in two traditional combined convolutions, and the coordinate attention mechanism is introduced to save computing power, learn more features and process more data in a shorter time;

[0032] The conventional convolution module in the output end is replaced by a Ghost Module to compress the model, improve the calculation speed and reduce the delay;

[0033] Step 3, using the improved target recognition model obtained to detect the photographed image, and identifying the pitaya fruits in the pitaya image.

[0034] YOLO is a target recognition and positioning algorithm based on deep neural network, which realizes end-to-end prediction. YOLOv4 combines a large number of previous research technologies and makes innovative combinations, and is a high-efficiency and powerful target detection model. The network structure of YOLOv4 is shown in Figure 2 The loss function Loss of YOLOv4 is composed of positive sample coordinate loss, positive sample confidence loss, negative sample confidence loss and positive sample classification loss. In this study, complete intersection over union (CIOU) is used as the loss function. CIOU makes the prediction box more consistent with the real box, makes the target box accurate positioning, and avoids the problem that the intersection over union (IOU) of the loss function is 0 due to the non-intersection of the prediction box and the real box. The expression of CIOU is:

[0035]

[0036] Where, ρ 2 (b,b gt ) represents the Euclidean distance between the center points of the prediction box and the real box; c represents the diagonal distance of the rectangle box tangent to the prediction box and the real box; β is a parameter for measuring the consistency of the length-width ratio; v is a weighting parameter. The expression of YOLOv4 loss function is:

[0037]

[0038] Where, λ coord is the weight coefficient of the positive sample; represents traversing all prediction boxes; and represent whether it is a positive sample, 1 if yes, otherwise 0; w i is the width of the center point of the prediction box; h i is the height of the center point of the prediction box; is the sample value; C i is the predicted value; λ boobjis the weight coefficient of negative samples, and classes is 20 in VOC dataset; P i (c) is the confidence of the predicted target; is the confidence of the actual target.

[0039] The application proposes a fast, accurate and lightweight improved target recognition model, which is based on the traditional YOLOv4 network, in order to reduce the model calculation amount, generate more feature maps and realize fast target recognition, the GhostNet network is used as the feature extraction backbone network; in order to save the calculation capacity, learn more features and process more data in a shorter time, the improved combined convolution module is used instead of the traditional combined convolution at the feature fusion position, and the coordinate attention mechanism (CA) is introduced; in order to compress the model, improve the calculation speed and reduce the delay, the Ghost Module is cited in Yolo Head.

[0040] The GhostNet proposes a structure for generating a large number of feature maps through a small amount of calculation-Ghost Module, which generates feature maps through a series of linear operations, and the feature maps generated through linear operations are called Ghost feature maps, and the feature maps operated are called Intrinsic feature maps. The conventional convolution module such as Figure 3 (a) is shown, and the GhostModule is shown in Figure 3 (b). If the size of the input feature map is h1xw1xc, the size of the output feature map is h2xw2xn, the convolution kernel size is kxk, and the step is s, then the FLOPs (model complexity) of the conventional convolution and the Ghost Module are respectively:

[0041] FLOPs (a) = n x h2 x w2 x c x k x k (3)

[0042]

[0043] The ratio of the two is:

[0044]

[0045] As can be seen, the Ghost Module FLOPs can be reduced to 1 / s of the conventional convolution, reducing the model complexity.

[0046] As Figure 2As shown, the traditional combined convolution is adopted at the *5 mark, which includes 5 convolution operations performed sequentially. The 5 convolution operations at the fusion are improved to generate an improved combined convolution module, i.e., a separable convolution and a residual edge structure are introduced. Among them, the separable convolution reduces the network calculation complexity and runs faster; using the residual edge can improve the model learning effect, prevent the loss of effective features, and effectively solve the gradient disappearance problem. Based on this, the two regular convolution operations in the traditional combined convolution are replaced by separable convolution, i.e., the second convolution operation and the fourth convolution operation are replaced by separable convolution; a residual edge is added beside the first two convolutions, i.e., a residual edge is added beside the first and second convolution operations, to obtain an improved combined convolution module. To help the model more accurately locate and identify the object of interest, an improved combined convolution-CA module is used, i.e., a coordinate attention mechanism is introduced between the second convolution operation and the third convolution operation. As shown in Figure 4 The coordinate attention mechanism embeds position information into channel attention, which can reduce attention to secondary information and enhance the extraction of target feature granularity to improve model accuracy and generalization performance. The coordinate attention mechanism includes coordinate information embedding and coordinate attention generation, and the structure is as shown in Figure 5 As shown. Among them, the coordinate information embedding operation corresponds to X Avg Pool and Y Avg Pool in the figure. For input X, first encode each channel along the horizontal and vertical coordinate directions using pooling kernels of size (H, 1) and (1, W). The output expression of the cth channel with height h is:

[0047]

[0048] Similarly, the output expression of the cth channel with width w is:

[0049]

[0050] For the coordinate attention generation operation, first concatenate the two feature maps generated by the previous module, and then use a shared 1x1 convolution transformation F1, the expression is:

[0051] f = δ (F1([z h ,z w ])) (8)

[0052] The generated f ∈ R C / r×(H+W) is the intermediate feature map of spatial information in the horizontal and vertical directions, where r represents the downsampling ratio, which is used to control the size of the module. Along the spatial dimension, f is divided into two separate tensors f h ∈ R C / r×H and f w ∈ R C / r×W Two 1x1 convolutions F h , F w are used to transform the feature maps f h, f w Transforming to the same number of channels as input X, the following results are obtained:

[0053] g h = σ(F h (f h )) (9)

[0054] g w = σ(F w ( fw )) (10)

[0055] Expand g h , g w as attention weights. The CA expression is:

[0056]

[0057] Through this calculation, the accurate position relationship of the target can be captured, so that the coordinated attention can more accurately locate the exact position of the object of interest, thereby helping the model to better identify.

[0058] The improved combined convolution-CA module is applied to the traditional combined convolution a after the first upsampling of feature fusion, the traditional combined convolution b after the second upsampling, the third traditional combined convolution c, and the fourth traditional combined convolution d. The improved combined convolution-CA module is used at ac, bd, and abcd, respectively, that is, the original CBL module at the *5 marked place is replaced by the improved combined convolution-CA module, as shown in Figure 6 , the recognition accuracy of different combinations is compared.

[0059] In this study, the YOLOv4 network is improved using Pytorch. The GPU is NVIDIA Quadro P2200 16G, and the CPU is Intel(R) Xeon(R) Silver 4210R. The YOLOv4 model and the improved model are trained on the Windows 10 operating system. The momentum size of the momentum optimizer in the network is set to 0.9, the initial learning rate of the weight is set to 0.001, the decay coefficient is set to 0.0005, and the training iteration number is 100.

[0060] In this study, the accuracy P, recall rate R, F1 value, AP value, detection speed, and weight size are selected as the model evaluation indicators. Since only the pitaya fruits in the image need to be identified, the pitaya fruits are regarded as positive samples, and all other objects are regarded as negative samples.

[0061]

[0062]

[0063]

[0064]

[0065] Wherein, TP, FP and FN have the following meanings:

[0066] TP: the number of correctly identified positive samples, which is the accuracy.

[0067] FP: the number of false positives, which is the false positive rate.

[0068] FN: the number of missed positive samples, which is the false negative rate.

[0069] In this study, the pitaya image was collected in the greenhouse of the farm, under the conditions of natural light on rainy and sunny days. The image acquisition device was a Canon (Canon EOS 750D) single-lens reflex camera, with an image resolution of 6000x4000 pixels and a format of JPG. To simulate the recognition system of the picking robot, five angles were selected for shooting, including front view, left view, right view, overhead view and downward view. A total of 1280 original pitaya images were collected, and 1100 images were selected to cover various natural environment images of pitaya growth environment, such as Figure 1 As shown. Data augmentation methods such as translation conversion, random rotation, mirror flip, horizontal flip, brightness adjustment and salt and pepper noise were used, and a total of 8800 images were obtained as the data set. According to the ratio of 7:1:2, the training set (6160 images), the validation set (880 images) and the test set (1760 images) were randomly divided. The pitaya target detection network process is shown in Figure 7 As shown. Comparative tests were conducted on different improved models to verify the model effect on the same validation set.

[0070] To prove the superiority of the improved model under the framework of GhostNet network, comparative experiments were conducted on traditional YOLOv4 network and different backbone networks. The accuracy, recall rate, F1 value, average precision, detection speed and weight size of pitaya target detection were detected by using CSPDarknet53, GhostNet, ShuffleNetV2 and EfficientNet as backbone network, respectively. The detection performance comparison results of different backbone networks are shown in Table 1. As shown in Table 1, compared with the CSPDarknet53 backbone network of the traditional YOLOv4 network, although the accuracy, recall rate, F1 value and average precision of the models with GhostNet, ShuffleNetV2 and EfficientNet as backbone network decreased, the detection speed improved and the weight size decreased significantly. Among them, the weight size of the model with GhostNet and ShuffleNetV2 as backbone network is similar, but in terms of detection speed, the model with GhostNet as backbone network is the fastest and the comprehensive effect is the best.

[0071] Table 1 Comparison of detection performance of different backbone networks

[0072]

[0073]

[0074] To prove the superiority of the improved network based on YOLOv4, the detection network performance before and after improvement needs to be compared and analyzed. Under the pitaya data set, YOLOv4 and improved YOLOv4 network were used for pitaya recognition test. The specific improvement methods are as follows: ① backbone network replacement. GhostNet is used as the backbone network. ② Improved combined convolution-CA module is used at feature fusion ac. ③ Improved combined convolution-CA module is used at feature fusion bd. ④ Improved combined convolution-CA module is used at feature fusion abcd. ⑤ Yolo Head refers to Ghost Module. The recognition effect of pitaya fruit in different natural environments under 5 kinds of improved methods is shown in Table 2. Figure 8 As shown in Table 2, the improved algorithm using GhostNet as the backbone network and using improved combined convolution-CA module at feature fusion ac and referring to Ghost Module in Yolo Head has high recognition accuracy when detecting pitaya fruit in rainy, occlusion and backlight conditions, while the other 4 kinds of target detection networks will have missed detection and false detection, and the recognition accuracy is lower than that of YOLOv4+①+②+⑤ network structure model. Therefore, the improved algorithm in this study has strong robustness and can adapt to different conditions in natural environment.

[0075] The detection performance comparison results of the five different improved algorithms are shown in Table 2. As shown in Table 2, the accuracy, recall rate, F1 value, average precision and detection speed of the YOLOv4+①+②+⑤ network structure model are all higher than those of the other four target detection networks, and at IOU = 0.50, they are 99.23%, 95.10%, 98%, 98.94% and 37.2 frames·s -1 respectively, and the weight size is the smallest, which is 59.4 MB, proving that it is obviously superior to the other four network structures. Compared with the traditional YOLOv4 network, at IOU = 0.50, the accuracy is increased by 6.8 percentage points, the recall rate is increased by 2.88 percentage points, the F1 value is increased by 6 percentage points, the average precision is increased by 4.6 percentage points, the detection speed is increased by 9.8 frames·s -1 , and the weight size is reduced by 184.6 MB.

[0076] Table 2 Comparison of detection performance of different improved networks

[0077]

[0078] Compared with the traditional YOLOv4 network, at IOU = 0.50, the improved YOLOv4 target recognition algorithm proposed in this study has an accuracy of 99.23% on the augmented data set, and the weight size is about 1 / 4 of that of the traditional YOLOv4, the average precision is increased by nearly 5 percentage points, and the detection speed is increased by nearly 10 frames·s -1 . The test proves that the YOLOv4 recognition algorithm proposed in this study, which fuses the coordinate attention mechanism and improved combined convolution, has significant advantages. The above description of the specific embodiments is only used to help understand the technical concept and core idea of the present application, and although specific preferred embodiments are used to describe and explain the technical solutions in this paper, it should not be understood as a limitation of the present application itself. Those skilled in the art can make various changes in form and details without departing from the technical concept of the present application. These easily thought of changes or replacements should be covered within the protection scope of the present application.

Claims

1. A YOLOv4 recognition method fusing coordinate attention mechanism and improved combined convolution, characterized in that: The method comprises the following steps: Step 1, selecting fruit and vegetable images in different environments to establish a data set; Step 2, improving the target detection algorithm YOLOv4 to obtain an improved target recognition model comprising an input end, a backbone feature extraction network, a feature fusion unit and an output end, inputting training samples selected from the data set into the established improved target recognition network for training, and the specific improvements comprising: The backbone feature extraction network adopts a GhostNet feature extraction network; The feature fusion unit fuses a coordinate attention mechanism and an improved combined convolution module, the improved combined convolution module is a separable convolution and a residual edge introduced in two traditional combined convolutions, the two traditional combined convolutions are a first traditional combined convolution after the first up-sampling and a third traditional combined convolution, the traditional combined convolution comprises 5 convolution operations executed in sequence, the 2nd and 4th convolution operations of the traditional combined convolution are replaced by the separable convolution, and a residual edge is added to the 1st and 2nd convolution operations of the traditional combined convolution; Step 3, detecting the photographed images by using the obtained improved target recognition model to recognize the fruits in the fruit and vegetable images.

2. The YOLOv4 recognition method of claim 1, wherein: In step 2, the conventional convolution module in the output end is replaced by a Ghost Module.

3. The YOLOv4 recognition method of claim 1, wherein: The specific method of fusing the coordinate attention mechanism in the feature fusion unit is that the coordinate attention mechanism is introduced between the 2nd and 3rd convolution operations of the traditional combined convolution.

Images