Occluded and Overlapped Fruit Recognition Method for Picking Robots

Abstract

The invention relates to an occluded and overlapped fruit recognition method for a picking robot, which belongs to the field of image recognition. A Dense-TRH-YOLO model is proposed. A Denseblock module is integrated into a backbone network on the basis of YOLOv5, a segment path from an early layer to a later layer is created, and a Transfomer module is integrated into the model to improve semantic distinguishability and reduce category confusion, thereby increasing the recognition accuracy of occluded objects. Then, image features of each layer are extracted through a Unet++-PAN neck structure. Finally, an Efficient IOU Loss loss function is used to replace CIOU of an original model to perform border regression to output a detection frame position and classification confidence. On the basis of CIOU, the difference values ​​of width and height are respectively calculated to replace the aspect ratio. Meanwhile, Focal Loss is introduced to solve the problem of imbalance between difficult and easy samples.

Description

Technical Field

[0001] The invention belongs to the field of image recognition and relates to a method for recognizing blocked and overlapping fruits for a picking robot. Background Art

[0002] Harvesting fruits is a very labor-intensive and time-consuming job. With the development of artificial intelligence, most of this work can be replaced by harvesting robots. Harvesting with robots is divided into two steps. First, fruit detection is performed using a computer vision system. Second, the robot is guided to pick the fruit based on the detection results. Among these two steps, fruit detection is the most critical and challenging. It not only determines the subsequent operation of the robot, but also the detection accuracy. Complex conditions and unstructured environments make this task very challenging.

[0003] For the recognition and classification of fruits, people have adopted various traditional visual detection methods to segment or locate fruit images. Nowadays, SSD network models, YOLO network models, and Fast R-CNN network models can be well applied in different platforms and fields. At present, deep learning technology is increasingly widely used and has become the mainstream detection method. Although these algorithms can solve the problem of fruit detection and recognition of picking robots, they often cannot achieve good detection results for fruits with severe occlusion and overlap in unstructured environments, making it impossible for picking robots to correctly identify whether the fruits can be picked, affecting production. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method for recognizing occluded and overlapping fruits for a picking robot, so as to solve the problem that traditional target detection in an unstructured environment is easily affected by complex backgrounds, and misses and misdetects fruits that are severely occluded and overlapped. The improved model Dense-TRU-YOLO based on the latest YOLOv5 network is used to reduce a large number of model parameters while maintaining accuracy, thereby overcoming the shortcomings of general deep learning neural network models such as large parameters, large amount of calculation, long calculation time, high requirements on computer hardware, and insufficient recognition accuracy.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for identifying occluded and overlapping fruits for a picking robot comprises the following steps:

[0007] S1: Improved Yolov5 is used as the main algorithm for fruit object detection. The improved Dense-TR-CSP replaces CSPDarknet-53 as the backbone network. The Denseblock module is integrated into the backbone network to create a segment path from the early layer to the later layer, improve the flow of information and gradients, make the model easier to train, and reduce the occurrence of overfitting problems.

[0008] S2: In order to improve semantic distinguishability, reduce category confusion, and increase the recognition accuracy of occluded objects, the deepest C3 module of the original YOLOv5 is replaced with the C3TR module;

[0009] S3: The Unet++-PAN structure is used to replace the PAnet in the original YOLOv5 as the neck structure of the model, and learnable weights are introduced to learn the importance of different input features when combining shallow information and deep information;

[0010] S4: After deep extraction of image features in step S3, three different feature extraction layers are passed to Yolo head to detect and classify fruit types;

[0011] S5: In the fully connected layer of step S4, a softmax classifier is used to perform target detection and classification, and an Efficient IOU Loss function is used to replace the CIOU of the original model to perform bounding box regression to output the detection box position and classification confidence;

[0012] S6: After the image is located in step S5 is cropped, edge extraction is performed, and a SVM classifier is used to classify whether the located fruit can be directly picked.

[0013] Furthermore, in step S1, the improved Yolov5 is used as the main algorithm for target detection, and its backbone network includes a Focus module and four Bottleneck CSP modules. The Denseblcok module is integrated into the backbone network to form Dense-CSPDarknet53, which effectively reduces the information loss of the image in the deep network. Under the premise of ensuring the detection speed and model complexity, the detection accuracy and feature extraction ability of the occluded objects are greatly increased, and the improved design of the occluded fruit recognition network of the picking robot is realized.

[0014] Furthermore, in order to improve semantic distinguishability and reduce category confusion, and increase the recognition accuracy of occluded objects in step S2, the deepest C3 module of the original YOLOv5 is replaced with a C3TR module, specifically including: firstly, the input image features are divided into two branches, in the first branch, after 1×1 convolution, they are input into the Transfomer module with a multi-head attention mechanism, and in the second branch, only 1×1 convolution operation is used; after the two branches are extracted, the branch information is fused by splicing operation, and the number of channels is restored by 1×1 convolution;

[0015] The calculation formula of the self-attention mechanism in the Transformer module is:

[0016]

[0017] Among them, Q, K, and V represent the query vector, key vector, and value vector, respectively. k is the scaling factor.

[0018] Furthermore, in step S3, firstly, a cross-layer weighted link is added between the original input nodes and output nodes of the same size; the cross-layer link can fuse more shallow semantic information without increasing the network complexity too much, making the network more accurate in regressing the target boundary and improving the recognition accuracy of occluders. Upsampling or downsampling weighted feature fusion is used between nodes of different sizes to obtain the Unet++-PAN structure to extract image features of different sizes;

[0019] The calculation formula for each node in Unet++-PAN is:

[0020]

[0021] Among them, H is convolution, D is downsampling, u is upsampling, let x i,j represents the node output, i represents the number of sampling layers, and j represents the convolutional layer along the skip index dense block.

[0022] Furthermore, in step S4, the obtained image features are input into the anchor-based Yolo head at the end of the Yolov5 framework to extract features of different sizes.

[0023] Furthermore, in step S5, the Efficient IOU Loss loss function is used to replace the CIOU of the original model to perform bounding box regression to output the detection box position and classification confidence. On the basis of CIOU, the difference values ​​of width and height are calculated respectively to replace the aspect ratio. At the same time, Focal Loss is introduced to solve the problem of imbalance between difficult and easy samples.

[0024] The GFL calculation formula is:

[0025]

[0026] Among them, C w and C h is the minimum width and height of the overlay box, L IOU is the IOU loss, L dis is the distance loss, L asp is the aspect loss, b, w, and h represent the center points respectively.

[0027] Furthermore, in step S6, the positioned image is cropped and edge extraction is performed, and a SVM classifier is used to classify whether the positioned fruit can be directly picked;

[0028] The principle of SVM is to find the maximum geometric interval between data:

[0029]

[0030] Sty i (w T x i +b)≧1

[0031] Among them: w, b are the parameters to be obtained, x i ,y i is the number of training samples.

[0032] The beneficial effects of the present invention are as follows: the present invention solves the problems that traditional target detection in unstructured environments is susceptible to the influence of complex backgrounds, missed detection and false detection of fruits with severe occlusion and overlap, and uses an improved model based on the latest YOLOv5 network named Dense-TRU-YOLO, which reduces a large number of model parameters while maintaining accuracy, and overcomes the shortcomings of large parameters, large amount of calculation, long calculation time, high requirements for computer hardware, and insufficient recognition accuracy of general deep learning neural network models. The present invention adopts Dense-TRU-YOLO as the main algorithm for target detection, replaces CSPDarknet-53 with the improved Dense-TR-CSP as the backbone network, and integrates the Denseblock module into the backbone network, effectively reducing the loss of image information in the deep network. Under the premise of ensuring the detection speed and model complexity, the detection accuracy and feature extraction capability of the occluded objects are greatly increased, and the improved design of the occluded fruit recognition network of the picking robot is realized. In order to improve semantic distinguishability and reduce category confusion, and increase the recognition accuracy of occluders, the present invention replaces the deepest C3 module of the original YOLOv5 with a C3TR module. First, the incoming image features are divided into two different branches. In the first branch, the 1×1 convolution is passed to the Transfomer module with a multi-head attention mechanism, and only 1×1 convolution operation is used in the second branch. After the two branches are extracted, the branch information is fused by splicing operation, and the number of channels is restored by 1×1 convolution. The present invention adopts Unet++-PAN structure in the neck structure, and first adds cross-layer weighted links between the original input nodes and output nodes of the same size. Cross-level links can fuse more shallow semantic information without increasing too much network complexity, so that the network's regression of the target boundary is more accurate and the recognition accuracy of occluders is improved. Upsampling or downsampling weighted feature fusion is used between nodes of different sizes, which can more fully fuse image features of various sizes. The present invention uses the Efficient IOU Loss loss function instead of the CIOU of the original model to perform border regression to output the detection box position and classification confidence. On the basis of the CIOU of the original model, the difference values ​​of width and height are calculated respectively to replace the aspect ratio. At the same time, Focal Loss is introduced to solve the problem of imbalance between difficult and easy samples.

[0033] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0035] Figure 1 It is a flow chart of the method for identifying occluded and overlapping fruits for a picking robot according to the present invention;

[0036] Figure 2 The Dense-TRU-YOLO network model structure and principle diagram of the present invention;

[0037] Figure 3 This is a structural diagram of the DenseCSP backbone network of the present invention;

[0038] Figure 4 The schematic diagram of the Transfomer mechanism of the present invention;

[0039] Figure 5 The schematic diagram of the structure of the C3 module and the improved C3TR of the present invention;

[0040] Figure 6 The structure diagram of the original model PANet of the present invention and the Unet++-PAN proposed by the present invention;

[0041] Figure 7 Graph P, R and mAP@0.5 of the Dense-TRU-YOLO model of the present invention trained on the tomato dataset. DETAILED DESCRIPTION

[0042] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0043] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0044] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0045] like Figure 1 As shown, the present invention provides a flowchart of a method for identifying occluded and overlapping fruits for a picking robot, comprising the following specific steps:

[0046] (1) The improved Yolov5 is used as the main algorithm for fruit target detection, Dense-TRU-YOLO Figure 2 As shown in Figure 1, the improved Dense-TR-CSP replaces CSPDarknet-53 as the backbone network, and the Denseblock module is integrated into the backbone network, creating a segment path from the early layer to the later layer, improving the flow of information and gradients, making the model easier to train, and reducing the occurrence of overfitting problems, as shown in Figure 1. Figure 3 As shown in the figure. The backbone network of the YOLOv5s architecture contains a Focus module and four Bottleneck CSP modules. The Denseblcok module is integrated into the backbone network to form Dense-CSPDarknet53, which effectively reduces the information loss of the image in the deep network. Under the premise of ensuring the detection speed and model complexity, the detection accuracy and feature extraction capability of the occluded objects are greatly increased, and the improved design of the occluded fruit recognition network of the picking robot is realized.

[0047] (2) In order to improve semantic distinguishability, reduce category confusion, and increase the recognition accuracy of occluded objects, the deepest C3 module of the original YOLOv5 is replaced with the C3TR module, as shown in Figure 4 , Figure 5 As shown in the figure, the incoming image features are first divided into two different branches. In the first branch, they are passed to the Transformer module with a multi-head attention mechanism after 1×1 convolution, and in the second branch, only 1×1 convolution operation is used. After the two branches are extracted, the branch information is fused by splicing operation, and the number of channels is restored by 1×1 convolution.

[0048] The calculation formula of the self-attention mechanism in the Transformer module is:

[0049]

[0050] Among them, Q, K, and V represent the query vector, key vector, and value vector, respectively. k is the scaling factor.

[0051] (3) The Unet++-PAN structure is used to replace the PAnet in the original model as the neck structure of the model, and when combining shallow information and deep information, learnable weights are introduced to learn the importance of different input features, such as Figure 6 As shown in the figure. First, cross-layer weighted links are added between the original input nodes and output nodes of the same size. Cross-level links can integrate more shallow semantic information without increasing the complexity of the network, making the network more accurate in regressing the target boundary and improving the recognition accuracy of occluders. Upsampling or downsampling weighted feature fusion is used between nodes of different sizes to obtain the Unet++-PAN structure to extract image features of different sizes.

[0052] The calculation formula for each node in Unet++-PAN is:

[0053]

[0054] Among them, H is convolution, D is downsampling, u is upsampling, let x i,j Indicates node output, i indicates the edge

[0055] The number of sampling layers, j represents the convolutional layer along the skip index dense block.

[0056] (4) After deep extraction of image features in step (3), three different feature extraction layers are input into Yolohead to detect and classify the fruit types; the obtained image features are input into the anchor-based Yolo head at the end of the Yolov5 framework to extract features of different sizes.

[0057] (5) In the fully connected layer of step (4), a softmax classifier is used to perform target detection and classification, and the Efficient IOU Loss loss function is used to replace the CIOU of the original model for bounding box regression to output the detection box position and classification confidence; the Efficient IOU Loss function is used to replace the CIOU of the original model for bounding box regression to output the detection box position and classification confidence. On the basis of CIOU, the difference values ​​of width and height are calculated respectively instead of the aspect ratio. At the same time, FocalLoss is introduced to solve the problem of imbalance between difficult and easy samples.

[0058] The GFL calculation formula is:

[0059]

[0060] Among them, C w and C h is the minimum width and height of the overlay box, L IOU is the IOU loss, L dis is the distance loss, L asp is the aspect loss, b, w, and h represent the center points respectively.

[0061] (6) After cropping the positioned image in (5), edge extraction is performed, and the positioned fruit is classified by using a SVM classifier to determine whether it can be directly picked. After cropping the positioned image, edge extraction is performed, and the positioned fruit is classified by using a SVM classifier to determine whether it can be directly picked.

[0062] The principle of SVM is to find the maximum geometric interval between data:

[0063]

[0064] Sty i (w T x i +b)≧1

[0065] Among them: w, b are the parameters we need to find, x i ,y i is the number of training samples.

[0066] like Figure 7 As shown, it is the P, R and mAP@0.5 diagram of the Dense-TRU-YOLO model of the present invention training on the tomato dataset.

[0067] Table 1 shows the Dense-TRU-YOLO ablation experimental data of the present invention.

[0068] Table 1

[0069]

[0070] Table 2 shows the comparison between the present invention and various commonly used network data, using mAP@0.5, F1, fps, and model size as performance evaluation.

[0071] Table 2

[0072]

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for identifying occluded and overlapping fruits for a picking robot, characterized in that: The following steps are involved: S1: The improved Yolov5 is used as the main algorithm for fruit target detection, the improved Dense-TR-CSP replaces CSPDarknet-53 as the backbone network, and the Denseblock module is integrated into the backbone network; S2: Replace the deepest C3 module of the original YOLOv5 with the C3TR module; S3: The Unet++-PAN structure is used to replace the PAnet in the original YOLOv5 as the neck structure of the model, and learnable weights are introduced to learn the importance of different input features when combining shallow information and deep information; first, cross-layer weighted links are added between the original input nodes and output nodes of the same size; upsampling or downsampling weighted feature fusion is used between nodes of different sizes to obtain the Unet++-PAN structure to extract image features of different sizes; The calculation formula for each node in Unet++-PAN is: Among them, H is convolution, D is downsampling, u is upsampling, let x i，j represents the node output, i represents the number of sampling layers, and j represents the convolution layer along the jump index dense block; S4: After deep extraction of image features in step S3, three different feature extraction layers are passed to Yolo head to detect and classify fruit types; S5: In the fully connected layer of step S4, a softmax classifier is used to perform target detection and classification, and the EfficientIOULoss loss function is used instead of the CIOU of the original model to perform bounding box regression to output the detection box position and classification confidence; S6: After the image is located in step S5 is cropped, edge extraction is performed, and a SVM classifier is used to classify whether the located fruit can be directly picked.

2. The method for identifying occluded and overlapping fruits for a picking robot according to claim 1, characterized in that: In the step S1, the improved Yolov5 is used as the main algorithm for target detection, and its backbone network includes a Focus module and four Bottleneck CSP modules. The Denseblcok module is integrated into the backbone network to form Dense-CSPDarknet53.

3. The method for identifying occluded and overlapping fruits for a picking robot according to claim 1, characterized in that: As described in step S2, the deepest C3 module of the original YOLOv5 is replaced with the C3TR module, which specifically includes: firstly, the input image features are divided into two branches. In the first branch, they are input into the Transfomer module with a multi-head attention mechanism after 1×1 convolution, and in the second branch, only 1×1 convolution operation is used; after the two branches are extracted, the branch information is fused by splicing operation, and the number of channels is restored by 1×1 convolution; The calculation formula of the self-attention mechanism in the Transformer module is: Among them, Q, K, and V represent the query vector, key vector, and value vector, respectively. k is the scaling factor.

4. The method for identifying occluded and overlapping fruits for a picking robot according to claim 1, characterized in that: In step S4, the obtained image features are input into the anchor-based Yolo head at the end of the Yolov5 framework to extract features of different sizes.

5. The method for identifying blocked and overlapping fruits for a picking robot according to claim 1, characterized in that: In step S5, the Efficient IOU Loss function is used to replace the CIOU of the original model to perform bounding box regression to output the detection box position and classification confidence. The difference values ​​of width and height are calculated on the basis of CIOU to replace the aspect ratio. At the same time, FocalLoss is introduced to solve the problem of imbalance between difficult and easy samples. The GFL calculation formula is: L EIOU =L IOU +L dis +L asp Among them, C w and C h is the minimum width and height of the overlay box, L IOU is the IOU loss, L dis is the distance loss, L asp is the aspect loss, b, w, and h represent the center points respectively.

6. The method for identifying blocked and overlapping fruits for a picking robot according to claim 1, characterized in that: In step S6, the positioned image is clipped and edge extraction is performed, and a SVM classifier is used to classify whether the positioned fruit can be directly picked; The principle of SVM is to find the maximum geometric interval between data: Among them: w, b are the parameters to be obtained, x i ,y i is the number of training samples.

Images