DSE-YOLO v3-based fruit target identification method
By introducing a deep separation convolution and channel attention mechanism in the YOLO v3 network, optimizing the network structure, the problem of insufficient recognition accuracy and rate in identifying multiple fruit targets, overlapping or occlusion scenarios is solved, and accurate identification and efficient sorting of fruit targets are achieved.
Patent Information
- Application Number
- CN202411630180.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-02-28
AI Technical Summary
The existing fruit sorting equipment has problems with low equipment usage rate, high labor costs, and low identification accuracy and rate in identifying multiple fruit targets, overlap or occlusion scenarios, and poor robustness.
A fruit target recognition method based on DSE-YOLO v3 was designed. By introducing a depth-separable convolution and channel attention mechanism, the YOLO v3 network structure is optimized, the number of cycles of residual units is reduced, and the number of Anchor boxes is optimized, so as to improve the recognition accuracy and rate of fruit targets.
Accurate identification and reliable sorting of fruit targets are achieved, the identification robustness in overlapping and occlusion scenarios is improved, and the utilization rate and work efficiency of equipment are improved.
Smart Images
Figure CN119540945A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a fruit target recognition method based on DSE-YOLO v3, and belongs to the technical field of artificial intelligence. Background Art
[0002] my country is an important fruit-producing country in the world, and post-harvest processing is a huge operation link in the fruit processing industry. Sorting technology is a key technology in the post-harvest processing of fruits. Traditional fruit sorting equipment generally adopts a mechanical sorting method of fixed aperture screening and weight weighing, which has problems such as low equipment utilization rate, easy damage to the fruit surface, and high labor cost, which directly affects the development of the fruit industry. With the rapid development of machine vision, intelligent sorting robots have gradually replaced mechanical sorting, and target recognition methods based on deep learning have also been widely used in the field of fruit picking and sorting, providing an effective means for intelligent post-harvest processing of fruits. In order to solve the problem that sorting equipment can be used for sorting different types of fruits in different seasons, improve equipment utilization and work efficiency, realize accurate and rapid fruit recognition and efficient and reliable sorting, and improve the robustness of overlapping and occluded fruit target recognition, it is of great application value and practical significance to study a fast and accurate fruit target recognition strategy.
[0003] Mainstream networks for target recognition:
[0004] The models are roughly divided into one-stage models and two-stage models. The two-stage model is represented by the R-CNN series. This type of method first extracts the feature information of the candidate area, and then predicts the target position and category of the candidate area through the detection network. The computational complexity is relatively high, but the detection accuracy is greatly improved compared to the performance of traditional algorithms; the one-stage model is represented by the YOLO series. This type of method is based on the idea of regression, does not generate candidate boxes, and directly outputs the target position and category through the network model in an end-to-end manner (end-to-end means that in the entire process from inputting raw data to outputting the final result, there is no need for multiple stages of independent processing or manual intervention, and all tasks are automatically completed by a unified model). Therefore, the recognition speed is faster. Compared with the two-stage detection model, the detection accuracy needs to be further improved.
[0005] In the post-harvest processing of fruits, in order to achieve the goal of automatic robot sorting, it is first necessary to accurately identify and locate the type and position of the fruit. The deep learning algorithm has a strong feature extraction capability, breaking through the limitation of traditional detection methods relying on artificially designed features. Therefore, it often has better generalization ability in practical applications and is suitable for target recognition tasks of various categories of fruits. With the requirements for real-time performance of neural network detection, REDMON et al. proposed YOLO, YOLO9000 and YOLO v3 target recognition network models. YOLO v3 is the third generation version of the YOLO series. It has great advantages over other target recognition methods in terms of target recognition accuracy and speed, and is applied to multi-field scenarios. Therefore, the present invention selects YOLO v3 in the one-stage model as the basic architecture for fruit target recognition.
[0006] The speed and accuracy of fruit target recognition is an important factor affecting the efficient operation of the sorting robot. The YOLO v3 model still has room for improvement in terms of network structure and parameters. On the one hand, the YOLO v3 network is complex, with a large number of parameters and high computer operation costs; on the other hand, when identifying multiple fruit targets, or when some targets overlap or are blocked, it is easy to miss or misdetect, resulting in poor recognition of fruit images in complex scenarios.
[0007] In view of the above problems (single fruit sorting target after harvest, recognition accuracy and rate need to be improved, and the robustness of overlapping and occluded fruit target recognition is improved), in order to achieve accurate and rapid recognition and efficient and reliable sorting of fruits after harvest, the present invention designs a DSE-YOLO v3 network, establishes a mixed data set with three common fruits: apples, oranges and peaches, and amplifies the samples. By introducing the SENet module in the deep separable convolution and channel attention mechanism, and reducing the number of cycles of the residual unit, the number of anchor boxes is optimized. This method effectively reduces the number of model parameters and calculations, and at the same time integrates shallow and deep information of different scales to improve the robustness of overlapping and occluded fruit target recognition. Summary of the invention
[0008] The present invention aims to solve the problems that the post-harvest sorting target of fruits is single, the recognition accuracy and rate need to be improved, and the robustness of the recognition of overlapping and occluded fruit targets is improved. A fruit target recognition method based on DSE-YOLO v3 is proposed. The method optimizes the YOLO v3 network structure through a self-made mixed data set and is named DSE-YOLO v3 network. The DSE-YOLO v3 network introduces deep separable convolution to replace standard convolution, embeds a channel attention mechanism in the residual unit, reduces the number of cycles of the residual unit, and optimizes the number of anchor boxes. The method realizes accurate recognition and reliable sorting of fruit targets, and can provide necessary technical support for intelligent post-harvest sorting in the fruit industry.
[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0010] A fruit target recognition method based on DSE-YOLO v3, comprising the following steps:
[0011] Step 1, constructing the dataset of fruit targets;
[0012] Step 2: Establish the DSE-YOLO v3 network model, use the deep separable convolution DSC and channel attention SENet modules, reduce the number of cycles of the residual unit, and optimize the number of anchor boxes;
[0013] Step 3, DSE-YOLO v3 network model construction: The DSE-YOLO v3 network model includes the backbone network DSE-Darknet and the feature extraction architecture New-FPN; the backbone network DSE-ResN is constructed with 5 attention residual modules, which are composed of DPBL and DES-Res_unit, and N is the number of cycles of the attention residual unit; in the feature extraction architecture New-FPN, the 3×3CBL in ConvSet is replaced with DPBL to obtain a new convolution set named New-Conv Set;
[0014] Step 4: Identify fruit targets of large, medium, and small scales: The fruit image data is input into the DSE-YOLO v3 network model. After the backbone network DSE-Darknet and New-FPN architecture, feature maps of 13×13, 26×26, and 52×52 are obtained to identify fruit targets of large, medium, and small scales respectively.
[0015] The feature map output by the fifth residual attention module of the backbone network DSE-Darknet undergoes a series of convolutions and is used for the first target recognition at a scale of 13×13, and then is upsampled and fused with the feature map output by the fourth residual attention module. After a series of convolutions, the second target recognition is performed at a scale of 26×26 at y2, and then it is upsampled and fused with the feature map output by the third residual attention module. After a series of convolutions, the third target recognition is performed at a scale of 52×52 at y3;
[0016] Step 5: Train the DSE-YOLO v3 network model;
[0017] Step 6, determining the evaluation index of the fruit target recognition model;
[0018] Step 7: Test effect verification.
[0019] Preferably, step 1 selects apples, oranges and peaches as target recognition objects;
[0020] The dataset construction method for fruit targets includes the following:
[0021] a, self-made mixed data set, online enhanced expansion data set sample:
[0022] Construct a mixed dataset of fruits from network and real-life images, randomly divide the dataset into 70%, 20%, and 10% parts, and the ratio of training set, validation set, and test set is 7:2:1.
[0023] Perform online enhancement on the training set, randomly perform translation, rotation, and salt and pepper noise operations on the image to obtain the processed dataset image;
[0024] b. Use the LabelImg tool to label the processed dataset images:
[0025] The annotated data information is saved as a txt format file. Different categories of fruits are marked by drawing an external rectangular box in the fruit image. The annotated data contains the category and location information of the fruit target. The location information includes the width, height and horizontal and vertical coordinates of the center point of the rectangular box.
[0026] Preferably, the depthwise separable convolution DSC is decomposed into two operations: depthwise convolution DC and pointwise convolution PC;
[0027] DC performs a convolution operation on each input channel separately.
[0028] PC uses a 1×1 convolution kernel to perform channel adjustment on the output of DC.
[0029] Preferably, a channel attention SENet module is embedded in the residual unit to obtain an attention residual unit, and the convolution forward channel consists of 1×1 CBL, DPBL and SENet modules, named DSE-Res_unit;
[0030] In the DSE-YOLO v3 network model, the IOU formula is:
[0031] d(box,centroid)=1-IOU(box,centroid)
[0032] Where: d(box, centroid) represents the distance from the anchor box to the cluster center; IOU(box, centroid) represents the intersection-union ratio between the anchor box and the object target box;
[0033] The average overlap Avg IOU is used as the metric for cluster analysis, and the objective function f Avg IOU It is expressed as:
[0034]
[0035] In the formula: n represents the total number of samples in the data set; k is the number of categories; n k Represents the number of samples in the kth cluster center; IOU (box, centroid) represents the intersection-over-union ratio of the Anchor box and the groundtruth box; the optimal number of Anchor boxes is obtained.
[0036] Preferably, in step 5, the model training is performed in an environment of AMD Ryzen7 2700CPU, 16GB RAM, and NVIDIA RTX 2070, using the PyTorch deep learning framework under the Linux Ubuntu 16.04 operating system for model acceleration training;
[0037] In the training phase, batch processing was used, with an initial learning rate of 0.001, a momentum factor of 0.937, a weight decay rate of 0.0001, and 200 training rounds;
[0038] The DSE-YOLO v3 network model was trained for 200 rounds, and evaluation and verification were performed on the validation set after each round of training.
[0039] Preferably, in step 6, the average precision AP and the mean average precision mAP are used as evaluation indicators of the fruit target recognition accuracy, and the number of frames per second FPS of the fruit image is used as the evaluation indicator of the fruit image recognition speed.
[0040] Preferably, in step 7, the AP of the DSE-YOLO v3 network model for three typical fruits, apple, orange, and peach, is as high as 93% or more, the average precision mean AP is as high as 94% or more, and the frame rate FPS of recognizing fruit images per second is as high as 64 frames / s or more.
[0041] A fruit target recognition system using the method of the invention.
[0042] The fruit sorting device using the fruit target recognition system of the present invention comprises an intelligent sorting robot.
[0043] The method of the invention is applied in sorting fruits of different categories in different seasons.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] The present invention aims to solve the problems that the post-harvest sorting target of fruits is single, the recognition accuracy and rate need to be improved, and the recognition ability of overlapping and occluded fruit targets needs to be improved. A fruit target recognition method based on DSE-YOLO v3 is proposed. The method optimizes the YOLO v3 network structure through a self-made mixed data set and names it as the DSE-YOLO v3 network. The DSE-YOLO v3 network replaces the standard convolution by introducing a deep separable convolution, embeds a channel attention mechanism in the residual unit, reduces the number of cycles of the residual unit, and optimizes the number of anchor boxes. The method realizes accurate recognition and reliable sorting of fruit targets, and can provide necessary technical support for intelligent post-harvest sorting in the fruit industry.
[0046] The DSE-YOLO v3 network of the present invention compresses the amount of network parameter calculation as a whole, and pays more attention to the target features of fruits of different categories in the process of generating feature maps, enhances the semantic information of fruit features, effectively suppresses complex background information, and can solve the problem of poor recognition accuracy when fruits are overlapped, occluded, and densely distributed. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a fruit target identification flow chart of the present invention;
[0048] Figure 2 It is the YOLO v3 network model;
[0049] Figure 3 It is the convolution module CBL;
[0050] Figure 4 Conv Set is the convolution set;
[0051] Figure 5 Comparison between standard convolution and depth-wise separable convolution structures;
[0052] Figure 6 It is the depth-wise separable convolution module DPBL;
[0053] Figure 7 Res_unit is the residual unit in Darknet-53;
[0054] Figure 8 is the attention residual unit DSE-Res_unit;
[0055] Fig. 9 This is the result of K-means clustering analysis;
[0056] Fig.10 The backbone network is DSE-Darknet;
[0057] Fig.11 It is the attention residual module DSE-ResN;
[0058] Fig.12 New-Conv Set is the convolution set;
[0059] Fig.13 is the training loss curve;
[0060] Fig.14 Comparison of recognition results between DSE-YOLO v3 and YOLO v3. DETAILED DESCRIPTION
[0061] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.
[0062] A fruit target recognition method based on DSE-YOLO v3 includes the following processes:
[0063] The process of the fruit target recognition method based on DSE-YOLO v3 proposed in this invention is as follows Figure 1 As shown in the figure. The fruit images were preprocessed by unifying the size and data annotation, and the input image scale was unified to 416×416. After preprocessing, the fruit images were divided into three sub-datasets: training set, validation set, and test set. The training set image data was input into the DSE-YOLO v3 network for model training, and the validation set was used for evaluation and verification. The optimal model for fruit target recognition was output, and finally the performance of the fruit recognition model was tested on the test set.
[0064] Step 1: Dataset construction for fruit targets
[0065] The present invention selects three common fruits, namely apples, oranges and peaches, as target recognition objects. The three types of fruits have obvious differences in shape, color, size, texture and other characteristics, which can reflect the recognition model's ability to extract and classify features of various types of fruits.
[0066] (1) Self-made mixed dataset, online enhancement and expansion of dataset samples
[0067] The present invention constructs a mixed fruit dataset of network and real-shot images, totaling 5,000 images, which contains a large number of target images with overlapping and occluded scenes. As shown in Table 1, 1,102 images in the mixed dataset come from network images, and 3,898 images come from real-shot camera images. Using Matlab software, the dataset is randomly divided into 70%, 20%, and 10% ratios, and the ratio of training set, validation set, and test set is 7:2:1, as shown in Table 1.
[0068] Table 1. Mixed dataset division
[0069] Dataset Partitioning Real shot images Web Image total Training set 800 2700 3500 Validation set 200 800 1000 Test Set 102 398 500 total 1102 3898 5000
[0070] In order to improve the quality of the dataset samples and enrich the sample diversity, the training set is enhanced online. Before entering the network model, the images are randomly translated, rotated, and subjected to salt and pepper noise. Online enhancement makes the training set images used in each round of training different, the dataset samples are expanded, and the model is more robust and has better generalization capabilities.
[0071] (2) Use the LabelImg tool to label the processed dataset images and save the labeled data information as a txt file. By drawing an external rectangular box in the fruit image to label different categories of fruits, the manual labeling of the fruit image is realized. The labeled data contains the category and location information of the fruit target. The location information is the width, height and horizontal and vertical coordinates of the center point of the rectangular box.
[0072] Step 2: Build the DSE-YOLO v3 network model
[0073] YOLO v3 network model:
[0074] The overall network model of YOLO v3 consists of the backbone network Darknet-53 and the feature pyramid network FPN (FeaturePyramid Networks), such as Figure 2 As shown in Figure 1, the convolutional module CBL is the basic component of the YOLO v3 network ( Figure 3 ), which consists of a convolutional layer Conv, a BN layer, and a LeakyRelu activation function layer; the convolution set Conv Set consists of continuous 1×1 and 3×3 CBL convolution modules ( Figure 4); Concat is a tensor concatenation operation to expand the tensor dimension; Upsample is an upsampling operation to achieve multi-scale feature fusion.
[0075] DSE-YOLO v3 network model:
[0076] (1) Lightweight convolution module
[0077] The YOLO v3 network uses a large number of 1×1 and 3×3 standard convolutions to construct CBL modules, resulting in too many parameters in the entire model and a large amount of computation, which affects the improvement of model performance. In order to improve the recognition rate of fruit targets, the present invention uses a depthwise separable convolution (DSC) with a small number of parameters and low computational complexity, which can be decomposed into two operations: depthwise convolution (DC) and pointwise convolution (PC). DC performs convolution operations on each input channel separately, and PC performs channel adjustment work on the 1×1 convolution kernel used by the output of DC, so the depthwise separable convolution can achieve the overall effect of standard convolution.
[0078] Figure 5 It is a network structure of standard convolution and depth-separable convolution. The input image size is C F ×C F ×M, the output feature map size is C F ×C F ×N. Where M and N refer to the number of channels of the input and output images. H ×C H The convolution kernel of size , the parameter quantity of standard convolution is W SC And the amount of calculation Q SC They are:
[0079] W SC =C H ×C H ×M×N (1)
[0080] Q SC =C H ×C H ×M×N×C F ×C F (2)
[0081] The number of parameters W of depthwise separable convolution DSC And the amount of calculation Q DSC They are:
[0082]
[0083]
[0084] Where W DC , W PC are the parameters of DC and PC respectively; Q DC , Q PC are the calculation amounts of DC and PC respectively.
[0085] In summary, the parameter and calculation ratios of DSC and SC are:
[0086]
[0087]
[0088] From formula (6), it can be calculated that the number of parameters and computation of the standard convolution with a convolution kernel size of 3×3 and an output channel number of 16 is nearly 6 times that of the depthwise separable convolution.
[0089] Therefore, the present invention uses a depth-separable convolution consisting of 3×3 DC and 1×1 PC to replace the 3×3 standard convolution in the Darknet-53 residual module and FPN structure to reduce the number of model parameters and computational complexity. After the replacement, a new convolution module is obtained, named DPBL, such as Figure 6 shown.
[0090] (2) Strengthening fruit target characteristics
[0091] Post-harvest sorting scenes have problems such as complex backgrounds, overlapping occlusions, and dense distribution. In order to reduce missed detection and false detection rates and better adapt to actual needs, it is particularly important to fully extract fruit target features. The channel attention mechanism SENet module can be applied in convolutional neural networks due to its own advantages of simplicity and ease of integration, which can improve the neural network feature extraction ability, strengthen important channel features, and suppress redundant features, thereby improving network performance.
[0092] SENet includes three processes: compression (Squeeze), excitation (Excitation) and feature recalibration (Scale).
[0093] There are a large number of residual layer jump connections in Darknet-53, and the residual unit Res_unit ( Figure 7 ) Two convolution modules CBL perform residual operations. The present invention embeds the SENet module in the residual unit to enhance the key features of the fruit target, thereby reducing the missed detection rate and the false detection rate and improving the recognition accuracy. The SENet module is embedded to obtain the attention residual unit. The convolution forward channel consists of 1×1 CBL, DPBL and SENet modules, named DSE-Res_unit.
[0094] (3) Reduce the number of cycles of the residual unit
[0095] Backbone network DSE-Darknet( Fig.10 ) is used as the basic network to extract features, and the residual units DES-Res_unit ( Fig.11 ) The number of cycles is adjusted to 6, 6, and 3 times respectively, and the number of backbone network layers is further reduced while ensuring network stability.
[0096] (4) Optimize the number of anchor boxes
[0097] The idea of anchor box was first proposed in Faster R-CNN. In the YOLO v3 network, the K-means clustering algorithm is used to cluster the target boxes of the data set to find the most suitable prior box. The anchor box selects the prior box with the largest intersection-union ratio with the real target box to calculate the loss, in order to make the anchor box and the object target box have a larger IOU and a smaller distance. However, the commonly used distance formulas such as Manhattan distance and Euclidean distance often cannot achieve the desired effect here, so the IOU definition distance measurement formula of the present invention is:
[0098] d(box,centroid)=1-IOU(box,centroid) (7)
[0099] Where: d(box, centroid) represents the distance from the anchor box to the cluster center; IOU(box, centroid) represents the intersection-union ratio between the anchor box and the object target box.
[0100] The number of anchor boxes is obtained by clustering the candidate boxes of the data set. Generally speaking, the more categories of anchor boxes there are, the closer the predicted box is to the real box, but it will also bring more calculation parameters. Therefore, the selection of values is critical to balance accuracy and speed. This paper adopts the average overlap Avg IOU (Average Intersection Union) as the metric for cluster analysis, and the objective function f Avg IOU It can be expressed as:
[0101]
[0102] In the formula: n represents the total number of samples in the data set; k is the number of categories; n krepresents the number of samples of the kth cluster center; IOU (box, centroid) represents the intersection-over-union ratio of the anchor box and the groundtruth box. Cluster analysis was performed on the target fruit data set of the present invention, and after multiple experiments, the optimal number of anchor boxes was obtained. Select k = 1, 2, 3, ..., 9, and obtain the functional relationship curve between k value and Avg IOU as shown in the figure. Fig. 9 shown.
[0103] Fig. 9 In, f Avg IOU As the number of k increases, f increases; after k = 3, f Avg IOU It tends to be stable, and the value is about 89.8%. Since the network structure outputs 3 scales in total, the best choice for the k value is a multiple of 3. Considering that the predicted box shapes of fruit targets in the image are relatively close, setting k = 6, that is, the number of anchor boxes is 6, can reduce the amount of model calculation, reduce network loss, and improve the recognition speed of fruit targets.
[0104] (5) DSE-YOLO v3 network model construction
[0105] The present invention integrates the deep separable convolution and channel attention SENet modules into the YOLO v3 network, and constructs the DSE-YOLO v3 network by reducing the number of cycles of the residual unit and optimizing the number of anchor boxes. The DSE-YOLO v3 network includes the backbone network DSE-Darknet and the feature extraction architecture New-FPN. The backbone network DSE-Darknet is as follows Fig.10 As shown, DSE-ResN( Fig.11 ) are the five attention residual modules constructed, which are composed of DPBL and DES-Res_unit, and N is the number of cycles of the attention residual unit. In the feature extraction architecture New-FPN, the 3×3CBL in Conv Set is replaced with DPBL to obtain a new convolution set named New-Conv Set ( Fig.12 ).
[0106] The fruit image data is input into the DSE-YOLO v3 network, and through the backbone network DSE-Darknet and New-FPN architecture, feature maps of three scales of 13×13, 26×26, and 52×52 are obtained to identify large, medium, and small fruit targets respectively. The FPN feature fusion method of the original YOLO v3 network is still retained during prediction. The feature map output by the fifth residual module of DSE-Darknet undergoes a series of convolutions and performs the first target recognition with a scale of 13×13 y1, and then is upsampled and fused with the feature map output by the fourth residual module. After a series of convolutions, the second target recognition is performed with a scale of 26×26 y2, and then upsampled and fused with the feature map output by the third residual module. After a series of convolutions, the third target recognition is performed with a scale of 52×52 y3.
[0107] The DSE-YOLO v3 network as a whole compresses the amount of network parameter calculation, and pays more attention to the target features of different categories of fruits in the process of generating feature maps, enhances the semantic information of fruit features, effectively suppresses complex background information, and can solve the problem of poor recognition accuracy when fruits are overlapped, occluded, and densely distributed.
[0108] Step 3: Train the DSE-YOLO v3 network model
[0109] (1) Parameter settings
[0110] Model training was performed using the PyTorch deep learning framework under the Linux Ubuntu 16.04 operating system under the configuration environment of AMD Ryzen 7 2700 CPU, 16GB RAM, and NVIDIA RTX 2070. During the training phase, batch processing was used, with an initial learning rate of 0.001, a momentum factor of 0.937, a weight decay rate of 0.0001, and 200 training rounds.
[0111] (2) Determine the optimal model for fruit target recognition
[0112] The DSE-YOLO v3 network model was trained for 200 rounds. To avoid overfitting and underfitting, evaluation and verification were performed on the validation set after each round of training. Fig.13 It is the loss curve under the training set and the validation set. When the network is trained for the first 25 rounds, the training loss decreases rapidly; when the number of training rounds reaches around 125, the training loss gradually tends to be stable. From the perspective of loss convergence, the convergence effect of the training set and the validation set is ideal and meets the test requirements. Therefore, the optimal model output after 125 rounds of iteration is determined as the optimal model for fruit target recognition.
[0113] Step 4: Determine the evaluation index of the fruit object recognition model
[0114] The present invention uses the average precision AP and the mean average precision mAP as evaluation indicators of the fruit target recognition accuracy, and the number of frames per second of fruit image recognition FPS as the evaluation indicator of the fruit image recognition speed. The calculation formulas of the average precision AP and the mean average precision mAP are as follows:
[0115]
[0116]
[0117]
[0118]
[0119] In the formula, TP means predicting positive examples as positive; FP means predicting negative examples as positive; FN means predicting the original positive examples as negative; P is the accuracy, which represents the prediction accuracy in the positive results; R is the recall, which represents the overall prediction accuracy; P(R) is the accuracy P corresponding to different recall rates R; AP i The mean progress of the i-th category; N is the number of categories.
[0120] Step 5: Test effect verification
[0121] (1) Comparison of recognition effects under overlapping and occluded environments
[0122] In the sorting operation after fruit harvesting, the target fruits to be sorted are usually in an overlapping, occluded and interfered environment. In order to simulate the actual sorting environment, three kinds of fruit images with overlapping, occluded and interfered images of apples, oranges and peaches were selected from the test set, and a comparative experiment between the DSE-YOLO v3 and YOLO v3 networks was designed. Some recognition results are shown in the figure. Fig.14 shown.
[0123] from Fig.14 It can be seen from the figure that under the interference of branches and leaves blocking and fruit overlapping, Fig.14 (b) There are obvious omissions. Fig.14 (a) The target fruit can be accurately identified, and the confidence of the DSE-YOLO v3 network in identifying the target is significantly higher than that of the YOLOv3 network. The Des-YOLO v3 network is embedded with the channel attention SEnet module, which has more advantages in the accuracy and confidence of target recognition.
[0124] (2) Verification of the recognition performance of the network model
[0125] In order to verify the recognition performance of the DSE-YOLO v3 fruit target recognition method proposed in this invention, the average precision AP, the mean average precision mAP and the recognition speed PFS are used as the model performance evaluation indicators. After training, the performance comparison experiments of four networks are carried out on the test set for three kinds of fruits: apple, orange and peach. The test results are shown in Table 2. Among them, YOLO v3 (1) represents the YOLO v3 network improved by using the depthwise separable convolution alone; YOLO v3 (2) represents the YOLO v3 network improved by using the channel attention SEnet module alone; DSE-YOLO v3 represents the DSE-YOLO v3 network proposed in this invention.
[0126] Table 2 Target recognition performance analysis
[0127]
[0128] As shown in Table 2, the AP of DSE-YOLO v3 for three typical fruits, apple, orange, and peach, is as high as 93%, which is higher than YOLO v3, YOLO v3(1), and YOLO v3(2). Compared with YOLO v3, the mAP of DSE-YOLO v3 is improved by 3.83% and the PFS is improved by 13.84%. It can be seen that DSE-YOLO v3 can achieve faster and more accurate recognition of different types of fruits compared with the original version.
[0129] The mAP of YOLO v3(1) is 91.27%, slightly higher than YOLO v3, and the PFS is increased to 65.72 frames / s, which is 16.34% higher than YOLOv3. This shows that the introduction of deep separable convolution has obvious advantages in efficient recognition of fruit targets while ensuring recognition accuracy. The PFS of YOLO v3(2) is 54.68%, slightly lower than YOLO v3, and the mAP is increased to 94.31%, which is 3.5% higher than YOLO v3. This shows that the embedded SENet module can achieve higher-precision recognition of fruit targets within an acceptable range of recognition speed. DSE-YOLO v3 combines the advantages of deep separable convolution and SENet modules on the basis of YOLO v3, and reduces the number of cycles of residual units and optimizes the number of anchor boxes. The network calculation pressure is reduced while the sensitivity of fruit targets in the feature map is enhanced. Therefore, it is better than YOLO v3 in the three evaluation indicators of AP, mAP and PFS.
[0130] (2) Comprehensive performance verification of different recognition methods
[0131] In order to further verify the comprehensive performance of the DSE-YOLO v3 fruit target recognition method proposed in this paper, a performance comparison test was conducted between DSE-YOLO v3 and the current mainstream target recognition network. The mean average precision (mAP) and the recognition speed (PFS) were used as the model performance evaluation indicators. After training, DSE-YOLO v3 was compared with the SSD, FasterR-CNN and YOLO v3 networks on the test set. The test results are shown in Table 3.
[0132] Table 3. Comprehensive performance comparison with other networks
[0133] network mAP(%) FPS (frames / s) SSD 85.14 64.27 Faster R-CNN 90.49 13.68 YOLO v3 91.14 56.49 DSE-YOLO v3 94.63 64.31
[0134] From Table 3, we can see that: SSD has a faster recognition speed, but the lowest mAP; Faster R-CNN has a better mAP than SSD, but the recognition speed is the slowest; YOLO v3 takes into account both recognition accuracy and speed, but its performance can be further improved in the specific fruit target recognition. The DSE-YOLO v3 network proposed in this paper is better than SSD, Faster R-CNN and YOLOv3 in both mAP and PFS indicators.
[0135] In summary, through three groups of comparative experiments, it can be concluded that the DSE-YOLO v3 fruit target recognition method proposed in this invention has obvious advantages over the mainstream recognition models in terms of recognition accuracy and speed.
[0136] A fruit target recognition system using the method of the invention.
[0137] The fruit sorting device using the fruit target recognition system of the present invention comprises an intelligent sorting robot.
[0138] The method of the invention is applied in sorting fruits of different categories in different seasons.
[0139] It should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all of the features of the previously disclosed embodiments. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.
[0140] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.
[0141] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A fruit target recognition method based on DSE-YOLO v3, characterized in that: The following steps are involved: Step 1, constructing the dataset of fruit targets; Step 2: Establish the DSE-YOLO v3 network model, use the deep separable convolution DSC and channel attention SENet modules, reduce the number of cycles of the residual unit, and optimize the number of anchor boxes; Step 3, DSE-YOLO v3 network model construction: The DSE-YOLO v3 network model includes the backbone network DSE-Darknet and the feature extraction architecture New-FPN; the backbone network DSE-ResN is constructed with 5 attention residual modules, which are composed of DPBL and DES-Res_unit, and N is the number of cycles of the attention residual unit; in the feature extraction architecture New-FPN, the 3×3CBL in Conv Set is replaced with DPBL to obtain a new convolution set named New-Conv Set; Step 4: Identify fruit targets of large, medium, and small scales: The fruit image data is input into the DSE-YOLO v3 network model. After the backbone network DSE-Darknet and New-FPN architecture, feature maps of 13×13, 26×26, and 52×52 are obtained to identify fruit targets of large, medium, and small scales respectively. The feature map output by the fifth residual attention module of the backbone network DSE-Darknet undergoes a series of convolutions and is used for the first target recognition at a scale of 13×13, and then is upsampled and fused with the feature map output by the fourth residual attention module. After a series of convolutions, the second target recognition is performed at a scale of 26×26 at y2, and then it is upsampled and fused with the feature map output by the third residual attention module. After a series of convolutions, the third target recognition is performed at a scale of 52×52 at y3; Step 5: Train the DSE-YOLO v3 network model; Step 6, determining the evaluation index of the fruit target recognition model; Step 7: Test effect verification.
2. The method according to claim 1, characterized in that: Step 1: Select apples, oranges, and peaches as target objects for recognition; The dataset construction method for fruit targets includes the following: a, self-made mixed data set, online enhanced expansion data set sample: Construct a mixed dataset of fruits from network and real-life images, randomly divide the dataset into 70%, 20%, and 10% parts, and the ratio of training set, validation set, and test set is 7:2:
1. Perform online enhancement on the training set, randomly perform translation, rotation, and salt and pepper noise operations on the image to obtain the processed dataset image; b. Use the LabelImg tool to label the processed dataset images: The annotated data information is saved as a txt format file. Different categories of fruits are marked by drawing an external rectangular box in the fruit image. The annotated data contains the category and location information of the fruit target. The location information includes the width, height and horizontal and vertical coordinates of the center point of the rectangular box.
3. The method according to claim 2, characterized in that Depth-wise separable convolution DSC is decomposed into two operations: depth-wise convolution DC and point-wise convolution PC. DC performs a convolution operation on each input channel separately. PC uses a 1×1 convolution kernel to perform channel adjustment on the output of DC.
4. The method according to claim 3, characterized in that The channel attention SENet module is embedded in the residual unit to obtain the attention residual unit. The convolution forward channel consists of 1×1 CBL, DPBL and SENet modules, named DSE-Res_unit; In the DSE-YOLO v3 network model, the IOU formula is: d(box,centroid)=1-IOU(box,centroid) Where: d(box, centroid) represents the distance from the anchor box to the cluster center; IOU(box, centroid) represents the intersection-union ratio between the anchor box and the object target box; The average overlap Avg IOU is used as the metric for cluster analysis, and the objective function f AvgIOU It is expressed as: In the formula: n represents the total number of samples in the data set; k is the number of categories; n k Represents the number of samples in the kth cluster center; IOU (box, centroid) represents the intersection-over-union ratio of the Anchor box and the groundtruth box; the optimal number of Anchor boxes is obtained.
5. The method according to claim 4, characterized in that In step 5, the model training is performed under the configuration environment of AMD Ryzen72700CPU, 16GB RAM, and NVIDIA RTX 2070, using the PyTorch deep learning framework under the Linux Ubuntu16.04 operating system for model acceleration training; In the training phase, batch processing was used, with an initial learning rate of 0.001, a momentum factor of 0.937, a weight decay rate of 0.0001, and 200 training rounds; The DSE-YOLO v3 network model was trained for 200 rounds, and evaluation and verification were performed on the validation set after each round of training.
6. The method according to claim 5, characterized in that In step 6, the average precision AP and the mean average precision mAP are used as evaluation indicators of the fruit target recognition accuracy, and the number of frames per second (FPS) of fruit images recognized is used as the evaluation indicator of the fruit image recognition speed.
7. The method according to claim 6, characterized in that In step 7, the AP of the DSE-YOLO v3 network model for three typical fruits, apple, orange, and peach, is over 93%, the average precision mAP is over 94%, and the FPS of recognizing fruit images per second is over 64 frames / s.
8. A fruit target recognition system using the method described in any one of claims 1 to 7.
9. A fruit sorting device using the fruit object recognition system according to claim 8, characterized in that: Fruit sorting equipment includes intelligent sorting robots.
10. Application of the method according to any one of claims 1 to 7 in sorting different types of fruits in different seasons.