A fruit recognition and positioning method applied to a fruit picking robot

Abstract

The application discloses a fruit recognition and positioning method applied to a fruit picking robot, relates to the technical field of machine vision recognition, and comprises the following steps: collecting pictures of ripe fruits, storing each picture in an RGB color mode, marking the positions and corresponding types of all ripe fruits in the pictures, and performing data enhancement on a data set; a color focusing module is constructed, the color focusing module enhances the color feature extraction capability through a multi-channel independent learning and feature fusion strategy; a feature scaling module of a delay global average pooling is embedded in a feature pyramid network and a path aggregation network to construct a feature scaling path integration network; a fruit detection model is constructed based on the color focusing module and the feature scaling path integration network and is trained, and the trained fruit detection model is used for recognizing and positioning fruits; and the fruit detection precision and efficiency in a small sample scene are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision recognition, and particularly relates to a fruit recognition and positioning method applied to a fruit picking robot. BACKGROUND

[0002] With the continuous improvement of fruit cultivation and planting technology, the yield of fruits is continuously improved. Every season, a large amount of manpower and material resources are consumed in fruit picking. China is one of the major fruit producing countries, and the picking of fruits accounts for more than two-thirds of the entire fruit production labor. The low level of mechanization leads to high labor costs. With the successful application of artificial intelligence technology in medical treatment, education, industrial production and other industries, it provides a reference for the fruit production industry in the agricultural field. At present, some automatic fruit picking robots have appeared on the market. The application of this technology not only can reduce production cost, but also can improve the production efficiency of orchard. Automatic recognition and positioning of fruits as a key technology of picking robots has become a research hotspot in recent years.

[0003] In the prior art, typical schemes include YOLOv3, YOLOv4, YOLOv5 and their improved versions (such as MangoYOLO, YOLOMuskmelon, etc.). These algorithms improve the precision and speed of fruit detection to a certain extent by optimizing the network structure, introducing attention mechanisms or improving feature fusion modules. However, the prior art still has the following limitations.

[0004] ①High dependence on large-scale labeled data. Existing deep learning algorithms usually require thousands or even tens of thousands of labeled images to achieve high detection accuracy. However, the cost of obtaining and labeling large-scale data in agricultural scenarios is high, especially when targeting specific small classes of fruits (such as chestnuts, strawberries, etc.), the problem of scarce labeled samples is more prominent.

[0005] ②Insufficient color feature extraction capability. Existing algorithms (such as YOLOv5) use standard convolution in the initial convolution layer to uniformly process RGB three-channel information, without fully exploiting the independent features of different color channels. In complex natural environments (such as uneven lighting, fruit surface reflection or color blur scenes), it is difficult for the algorithm to distinguish between targets and backgrounds, leading to missed detection or false detection.

[0006] ③Low efficiency of multi-scale detection. Traditional FPN+PAN structure transmits multi-scale features through bottom-up path aggregation, but each detection head needs to process redundant information across scales, reducing the target positioning efficiency. For example, a small-scale detection head may be forced to process features of large-scale targets, resulting in waste of computing resources and loss of detection accuracy.

[0007] ④ Generalization ability and robustness are insufficient. In the case of limited data enhancement technology, the existing algorithm has poor adaptability to occlusion, dense distribution or light changes. Especially when no dynamic enhancement strategy is introduced in the test stage, the performance of the algorithm fluctuates greatly in complex environments, making it difficult to meet the needs of industrial-level applications.

[0008] The above problems restrict the promotion of fruit picking robots in actual scenes. Therefore, there is an urgent need for a target detection algorithm that can efficiently extract color features under a small amount of labeled data, optimize multi-scale detection efficiency, and have strong generalization ability to improve the accuracy and practicality of multi-class fruit recognition in natural environments. SUMMARY

[0009] To solve the problems existing in the prior art, the purpose of the present application is to provide a fruit identification and positioning method applied to a fruit picking robot, which significantly improves the fruit detection accuracy and efficiency in a small sample scene.

[0010] To achieve the above purpose, the technical scheme adopted by the present application is as follows: a fruit identification and positioning method applied to a fruit picking robot, comprising the following steps:

[0011] Step 1, data set making: collecting pictures of ripe fruits, storing each picture in RGB color mode, marking the positions of all ripe fruits in the picture and the corresponding types, and performing data enhancement on the data set;

[0012] Step 2, constructing a color focusing module CF, which enhances the color feature extraction ability through multi-channel independent learning and feature fusion strategy;

[0013] Step 3, embedding a feature scaling module SE with delayed global average pooling in the feature pyramid network FPN and the path aggregation network PAN to construct a feature scaling path integration network SEPAN;

[0014] Step 4, constructing a fruit detection model Fruits-YOLO based on the color focusing module CF and the feature scaling path integration network SEPAN and training it, and using the trained fruit detection model Fruits-YOLO to identify and position the fruits.

[0015] As a further improvement of the present application, in step 1, the data enhancement of the data set specifically includes: horizontal flip, vertical flip, mosaic enhancement, picture mixing enhancement, random perspective transformation, adaptive scaling, HSV enhancement, image splicing and rotation.

[0016] As a further improvement of the present application, the step 2 specifically comprises the following steps:

[0017] Step 2.1, receiving an RGB three-channel image as the initial input of the color focus module CF;

[0018] Step 2.2, a four-branch parallel processing architecture is constructed, single-channel feature extraction is performed on the red, green and blue three branches of the RGB three-channel image respectively, fine-grained color features are extracted by performing independent convolution operation on each branch, a RGB fusion branch is additionally added for fusion channel feature extraction, cross-channel correlation information is captured by performing convolution operation on the branch, and the output channel number of each branch in the four branches is compressed to 1 / 4 of the total number of channel outputs;

[0019] Step 2.3, after the four-branch output is spliced in the channel dimension, the feature is reorganized through batch normalization and an activation function.

[0020] As a further improvement of the application, in step 3, the feature scaling module SE first adopts a convolution block to fuse cross-channel pixel features, then adopts a global average pooling layer to obtain feature representation of each channel, continues to adopt a convolution block to compress the number of features to half of the original, then uses a convolution block to restore the number of features, and finally excites the original features in a multiplicative manner.

[0021] As a further improvement of the application, in step 4, when training the fruit detection model Fruits-YOLO, in terms of loss function, a binary cross-entropy loss function is used to optimize the class prediction probability; a complete intersection-over-union loss function is used to optimize the target bounding box position, including the overlapping area, center point distance and width-height ratio; a cross-entropy loss function is used to optimize the target confidence, to distinguish the foreground and background areas; in terms of data set division, the data set containing multiple types of fruits is randomly divided according to a preset proportion, part of which is used as a training set for algorithm parameter learning, and the remaining part is used as a test set for performance verification; in terms of optimization strategy, an Adam optimizer is used to dynamically adjust the algorithm parameters, combined with momentum to accelerate convergence; a linear decay learning rate strategy is introduced to balance the training stability and convergence efficiency.

[0022] As a further improvement of the application, in step 4, the trained fruit detection model Fruits-YOLO is also tested and analyzed, the optimal algorithm is tested on an independent test set, the average precision, precision, recall and inference speed indicators are calculated, and the algorithm itself is compared with other algorithms, the ablation research is carried out, and the efficiency of the algorithm is analyzed.

[0023] As a further improvement of the application, in step 4, the fruit is identified and positioned as follows:

[0024] Each output feature map is treated as a grid or data point, that is, the input image is divided into grids of equal size, and the identification and localization of fruits are completed by each grid. In the Fruits-YOLO fruit detection model, each grid of each feature map predicts three pieces of information, including fruit position, predicted class, and confidence score. The position includes the center point coordinates (x, y) and width and height (w, h). The one with the highest probability is selected as the identification result of the predicted class, and the confidence score is a prediction auxiliary value. Finally, the non-maximum suppression algorithm is used to remove the localization that does not meet the conditions.

[0025] The beneficial effects of this invention are:

[0026] This invention significantly improves the accuracy and efficiency of fruit detection in small-sample scenarios by introducing a Color Focus (CF) module and a Squeeze and Excitation Path Aggregation Network (SEPAN) and optimizing feature extraction and multi-scale fusion mechanisms. Attached Figure Description

[0027] Figure 1 This is a flowchart of the algorithm according to an embodiment of the present invention;

[0028] Figure 2 This is a structural diagram of the color focusing module in an embodiment of the present invention;

[0029] Figure 3 This is a structural diagram of the SE module in an embodiment of the present invention;

[0030] Figure 4 This is a network structure diagram of the Fruits-YOLO model in an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram showing a portion of the images in the ContentsFruits dataset in an embodiment of the present invention;

[0032] Figure 6 This is a visual diagram illustrating the recognition results of the Fruits-YOLO algorithm in an embodiment of the present invention. Detailed Implementation

[0033] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0034] Example

[0035] like Figure 1 As shown, a fruit recognition and localization method applied to a fruit-picking robot includes the following steps:

[0036] S01: Dataset construction. 300 pictures of four types of mature fruits (chestnut, orange, strawberry and tomato) were collected, and each picture was stored in RGB color mode. The positions and types of all mature chestnuts, oranges, strawberries and tomatoes in the pictures were marked using VOC annotation mode. Data augmentation was performed on the dataset to enrich the morphology of the dataset, including horizontal flipping, vertical flipping, mosaic enhancement, picture mixing enhancement, random perspective transformation, adaptive scaling technology, HSV enhancement, image stitching and rotation.

[0037] S02: Construction of color focus module (Color Focus, CF). The module enhances the color feature extraction capability through multi-channel independent learning and feature fusion strategy, including five operations: input preprocessing, single-channel feature extraction, cross-channel feature extraction, feature fusion and nonlinear transformation, as shown in Figure 2 The input preprocessing stage receives a RGB three-channel image (input size 1280x1280x3) as the initial input of the module. Single-channel feature extraction includes R, G and B three branches: R branch uses a 7x7 convolution kernel (step 2) to independently extract red channel features, with an output size of 640x640xC / 4, where C is the total number of channel outputs at this layer; G branch uses the same operation to independently process the green channel, with an output size of 640x640xC / 4; B branch independently extracts blue channel features, with an output size of 640x640xC / 4. The fusion channel feature extraction stage performs a 7x7 convolution (step 2) on the RGB three-channel fusion data to capture cross-channel correlation features, with an output size of 640x640xC / 4. The feature fusion operation concatenates the four branches (R, G, B and RGB) along the channel dimension to generate a 640x640xC feature map. Nonlinear transformation is essentially an f function, including BatchNorm and SiLU activation function, which aims to eliminate covariant bias and enhance nonlinear expression capability.

[0038] S03: Construction of SEPAN (Squeeze and Excitation Path Aggregation Network). SE module (Squeeze and Excitation Block) is added to the PAN network structure to solve the problem of processing cross-scale redundant information in the traditional PAN structure. The structure of SE module is as follows Figure 3As shown, X and X' represent input features and scaled features, respectively, 1x1 ConvBlock represents a convolutional block composed of a 1x1 convolution kernel, batch normalization, and SiLU activation function, Conv is a convolution operation, and GAP is a global average pooling layer. In the SE module, a 1x1 convolution block is first used to fuse cross-channel pixel features, then a global average pooling layer is used to obtain feature representation of each channel, and then a 1x1 convolution block is used to compress the number of features to half of the original number, enhancing the semantic representation of each feature. In order to align the original features, a 1x1 convolution is used to restore the number of features, and finally the learned features are used to excite the original features in a multiplicative manner. As shown in Figure 4 As shown, the original PAN structure has four output paths, and the last three output paths all fuse the features of the previous path. In the feature transmission process, the SE module is added to focus on its own features.

[0039] S04: Constructing Fruits-YOLO fruit detection algorithm, Figure 4 The Fruits-YOLO algorithm structure diagram is shown. Based on the color focusing module (CF) and the feature scaling path integration network (SEPAN), the YOLOv5 target detection algorithm is optimized, and through the feature alignment technology, the proposed module and algorithm are perfectly integrated, so that the algorithm has higher accuracy.

[0040] S05: Training Fruits-YOLO algorithm. In terms of loss function, binary cross-entropy loss function (BCELoss) is used to optimize the class prediction probability, and the robustness of multi-class fruit classification is enhanced; complete intersection over union loss function (CIoULoss) is used to optimize the target bounding box position, which considers the overlapping area, center point distance and aspect ratio, and improves the positioning accuracy; cross-entropy loss function (CE Loss) is used to optimize the target confidence, which distinguishes the foreground and background areas and reduces false detection. In terms of data set division, the data set containing four types of fruits such as Chinese chestnut and orange is randomly divided in the ratio of 8:2, of which 80% is used as the training set for algorithm parameter learning, and 20% is used as the test set for performance verification. In terms of optimization strategy, Adam optimizer is used to dynamically adjust algorithm parameters, combined with momentum to accelerate convergence; linear decay learning rate strategy is introduced, and the initial learning rate is set to 10 -3 , which is gradually decayed to 10 -4 , balancing training stability and convergence efficiency.

[0041] S06: Algorithm testing and analysis. The optimal algorithm is tested on an independent test set (20% of the total data) to calculate the average precision (mAP), precision, recall, and inference speed (FPS) and other core indicators. Compared with other algorithms (such as YOLOv3, YOLOv4, and native YOLOv5), the algorithm itself is ablated, and the efficiency of the algorithm is analyzed.

[0042] S07: Fruit recognition and positioning. As shown in Figure 4 , input a 3-channel image with a resolution of 1280x1280 (if the input image is not of this size, it will be automatically scaled), and the Fruits-YOLO algorithm will output four feature maps with sizes of 160x160, 80x80, 40x40, and 20x20, respectively, and the number of channels is 27. Each output feature map can be regarded as a grid (or data point), i.e., the input image is divided into grids of the same size, and the recognition and positioning of fruits are completed by each grid. The 160-size feature map grid is more, which can predict more small fruits, 80 predicts small fruits, 40 predicts medium fruits, and 20 predicts large fruits. In Fruits-YOLO, each grid of each feature map predicts three pieces of information, each piece of information contains one fruit position, four predicted categories, and one confidence, the position contains the center point coordinates (x, y) and the width and height (w, h), the category is four categories (the one with the highest probability will be selected as the recognition result), and the confidence is a prediction auxiliary value, so the output channel number is (4+4+1)x3=27. Thus, it can be calculated that Fruits-YOLO will finally predict 34,000 position information, and finally the non-maximum suppression algorithm is used to remove the positioning that does not meet the conditions.

[0043] The color focusing module (CF) in this embodiment: a four-branch parallel processing architecture is created, which independently performs 7x7 convolution (step 2) on the red, green, and blue single channels of the RGB image, extracts fine-grained color features (the output channel number is compressed to 1 / 4), and adds an RGB fusion branch to capture cross-channel correlation information. After the four branches are output, they are concatenated (Concat) in the channel dimension, and then normalized in batches (BatchNorm) and activated by the SiLU function to realize feature reorganization. This design completely solves the problem of color information loss caused by RGB mixed calculation in traditional convolution.

[0044] The feature scaling path integration network (SEPAN) in this embodiment: embeds a delayed global average pooling (Delayed GAP) feature scaling module in the FPN+PAN structure, and the calculation process is: first learn spatial features through dimension reduction convolution (Conv C / 2) and dimension increasing convolution (Conv C), then perform GAP compression and Sigmoid weighting. This mechanism forces each detection head to focus only on the target of this scale (such as the small scale head focusing on chestnuts, and the large scale head focusing on oranges), and suppresses the interference of cross-scale features.

[0045] The present embodiment is further illustrated by experiments as follows:

[0046] The experiment is based on the ContestsFruits dataset, which is a very challenging dataset released by China Mobile for fruit target detection tasks (available for download at https: / / ecloud.10086.cn / home / ), containing 300 labeled pictures covering 4 types of fruits: chestnuts, oranges, strawberries, and tomatoes. The dataset only labels mature fruits in the pictures, with 320 chestnut labels, 1007 orange labels, 528 strawberry labels, and 221 tomato labels, for a total of 2076 labels. During training, the dataset is randomly divided into a training set and a test set in a ratio of 8:2, Figure 5 is a display of part of the pictures in the dataset. Due to the inclusion of only 300 pictures and the small number of labeled instances, data augmentation methods such as Mosaic, horizontal flipping, vertical flipping, image scaling, brightness, saturation, and hue are used to improve the generalization ability and robustness of the algorithm.

[0047] The experiment uses the Pytorch deep learning framework and accelerates the training on an NVIDIA GeForce RTX 3090 graphics card. The Adam optimization algorithm is used, and the learning rate is in the range of 10 -3 -10 -4The linear learning rate of the model is iterated 300 times on the dataset. In order to reflect the comprehensive performance of Fruits-YOLO, the embodiment evaluates the algorithm from the dimensions of mean average precision (mAP), inference speed, algorithm parameter quantity (Params), and floating point operation quantity (FLOPs). In order to analyze the performance of the algorithm under the condition of setting different overlap joint ratio (IOU) thresholds, the average precision and the mean average precision are respectively evaluated by using thresholds of @0.5, @0.75, and @0.5:0.95, wherein @0.5:0.95 represents the average value of the overlap joint ratio thresholds in the threshold range [0.5, 0.95] with a step size of 0.05. Table 1 shows the performance of the Fruits-YOLO algorithm on four types of fruits, and Table 2 shows the performance of different algorithms and the results of the ablation study of Fruits-YOLO. The ablation study can analyze the effectiveness of the color focus module and the self-attention path integration network. At the same time, in order to make the fruit recognition and positioning effect of Fruits-YOLO more intuitive, the results are visualized in Figure 6 , and the fruit positioning, class, and confidence predicted by the model are marked on the input picture.

[0048] Table 1 Performance of Fruits-YOLO algorithm in four types of fruits

[0049]

[0050] Table 2 Performance comparison of different algorithms and ablation study

[0051]

[0052] From the experimental results, it can be seen that:

[0053] 1) The color focus deep learning module based on RGB images can learn more color features. This module reduces the floating point operation quantity by 0.3B while improving the average precision (mAP@0.5) by 0.6%, which reflects the effectiveness of the color focus module.

[0054] 2) The built feature scaling path aggregation network (SEPAN) fully utilizes the self-attention mechanism, so that each path prediction can focus on the target belonging to the current detection head scale. The average precision of fruit detection is improved from 83.2% to 85.3%, and the inference speed is still maintained at 41.9ms / frame.

[0055] 3) Fruits-YOLO fruit detection algorithm is constructed, and on a data set with only 300 labeled pictures, the average precision is as high as 85.3%, which is improved by 6.1% compared with YOLOv3, 7.9% compared with YOLOv4, 5.4% and 2.7% respectively compared with YOLOv5s and YOLOv5s6, and the average precision can reach 86.6% in combination with the enhanced test. Fruits-YOLO is an efficient end-to-end detection algorithm, which can also show good detection results on a small amount of labeled data set.

[0056] The above embodiments only express the specific implementation of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A fruit recognition and positioning method applied to a fruit picking robot, characterized in that, The method comprises the following steps: Step 1, dataset making: collecting pictures of mature fruits, each picture is stored in RGB color mode, marking the positions and corresponding types of all mature fruits in the picture, and performing data enhancement on the dataset; Step 2, constructing a color focusing module CF, which enhances the color feature extraction capability through multi-channel independent learning and feature fusion strategy; The step 2 specifically comprises the following steps: Step 2.1, receiving an RGB three-channel image as the initial input of the color focusing module CF; Step 2.2, constructing a four-branch parallel processing architecture, respectively performing single-channel feature extraction on the red, green and blue three branches of the RGB three-channel image, extracting fine-grained color features through independent convolution operation on each branch, and additionally adding an RGB fusion branch to perform fusion channel feature extraction, capturing cross-channel correlation information through convolution operation on the branch, and compressing the output channel number of each branch in the four branches to 1 / 4 of the total number of channel outputs; Step 2.3, after the four-branch output is spliced in the channel dimension, the feature is reorganized through batch normalization and activation function; Step 3, embedding a feature scaling module SE with delayed global average pooling in a feature pyramid network FPN and a path aggregation network PAN to construct a feature scaling path integration network SEPAN; Step 4, constructing a fruit detection model Fruits-YOLO based on the color focusing module CF and the feature scaling path integration network SEPAN and training the fruit detection model Fruits-YOLO, and using the trained fruit detection model Fruits-YOLO to identify and locate the fruits.

2. The fruit recognition and positioning method for a fruit picking robot according to claim 1, wherein, In step 1, the data enhancement of the dataset specifically includes: horizontal flip, vertical flip, mosaic enhancement, picture mixing enhancement, random perspective transformation, adaptive scaling, HSV enhancement, image splicing and rotation. 3.The fruit recognition and positioning method applied to the fruit picking robot according to claim 2, characterized in that, In step 3, the feature scaling module SE first fuses cross-channel pixel features using a convolution block, then obtains feature representation of each channel using a global average pooling layer, continues to compress the number of features to half of the original number using a convolution block, then restores the number of features using a convolution block, and finally excites the original features in a multiplicative manner.

4. The fruit recognition and positioning method for a fruit picking robot according to claim 3, wherein, In step 4, when training the fruit detection model Fruits-YOLO, in terms of loss function, a binary cross-entropy loss function is used to optimize the class prediction probability; a complete intersection-over-union loss function is used to optimize the target bounding box position, including the overlapping area, center point distance and width-height ratio; a cross-entropy loss function is used to optimize the target confidence to distinguish foreground and background areas; in terms of dataset division, the dataset containing multiple types of fruits is randomly divided according to a predetermined proportion, wherein a part is used as a training set for algorithm parameter learning, and the remaining part is used as a test set for performance verification; in terms of optimization strategy, an Adam optimizer is used to dynamically adjust the algorithm parameters to accelerate convergence; a linear decay learning rate strategy is introduced to balance training stability and convergence efficiency.

5. The fruit recognition and positioning method for a fruit picking robot according to claim 4, wherein, In step 4, the trained fruit detection model Fruits-YOLO is also tested and analyzed, the optimal algorithm is tested on an independent test set, the average precision, precision, recall and inference speed indicators are calculated, and the algorithm itself is compared with other algorithms, the ablation study is carried out, and the efficiency of the algorithm is analyzed. 6.The fruit recognition and positioning method for a fruit picking robot according to claim 4 or 5, characterized in that, In step 4, the fruit identification and positioning is as follows: Each output feature map is regarded as a grid or a data point, that is, the input picture is divided into grids of the same size, and the identification and positioning of the fruit are completed by each grid; in the fruit detection model Fruits-YOLO, each grid of each feature map predicts 3 pieces of information, including fruit position, predicted category and confidence, the position includes center point coordinates (x, y) and width and height (w, h), the highest probability is selected as the identification result of the predicted category, the confidence is a predicted auxiliary value, and finally the non-maximum suppression algorithm is used to remove the positioning that does not meet the conditions.

Images