Citrus fruit picking positioning method, system and device based on deep learning

Abstract

The application discloses a citrus fruit picking positioning method, system and device based on deep learning, which comprises the following steps: using a target detection model to detect fruits in a scene shot by a camera; determining the close-up and long shot scenes of the shot scene based on the fruit detection result; using a clustering algorithm to process the fruits in the long shot scene, so that the lens is converted from the long shot scene to the close-up scene; using a prior method to obtain the relative position of the fruits and branches; using an improved instance segmentation algorithm to segment the branches connected with the fruits in the close-up scene; according to the fruit detection result, the relative position of the fruits and the branches and the branch segmentation result, using a picking point positioning algorithm to obtain the coordinates of the fruit picking point, and controlling a mechanical arm to complete the automatic picking of the fruits. The IAM part of the SparseInst algorithm is improved by adding the image brightness and the fruit leaf growth prior, so that the position of the aggregated instance feature is accurately focused on the target branch, and the accuracy and real-time performance of the segmentation algorithm are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method, system, and device for citrus fruit harvesting and positioning based on deep learning. Background Technology

[0002] Automated fruit harvesting technology is one of the key technologies in smart agriculture. With the rapid development of deep learning, the transformation from traditional agriculture to smart agriculture is becoming increasingly urgent. In recent years, many farms have faced difficulties in recruiting workers during the harvest season, and fruits are at risk of dying on the trees due to untimely harvesting. This has created a huge market demand for research on automated fruit harvesting. Currently, most automated fruit harvesting point positioning technologies are limited by conditions such as vertical fruit growth and clear exposed branch backgrounds, making them difficult to apply to many complex fruit growth situations. Therefore, this invention utilizes target detection and instance segmentation algorithms combined with agronomic knowledge to research and design an automated fruit harvesting point positioning method applicable to various complex environments and conditions. Because the growth environment of Wogan mandarin oranges is complex and their growth habits are unpredictable, the automated harvesting of Wogan mandarin oranges will be used as a specific example to introduce the designed automated fruit harvesting point positioning method. Summary of the Invention

[0003] The main objective of this invention is to overcome the shortcomings and deficiencies of the existing technology and provide a deep learning-based method for citrus fruit picking and localization. By adding image brightness priors and fruit and leaf growth priors, the IAM part of the SparseInst algorithm is improved, allowing the location of aggregated instance features to be more accurately focused on the target branch, thereby improving the accuracy and real-time performance of the entire segmentation algorithm.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] This invention provides a deep learning-based method for citrus fruit harvesting and localization, comprising the following steps:

[0006] An improved YOLOv5 object detection model is used to detect fruit in a scene captured by a depth camera, yielding fruit detection results. The improved YOLOv5 object detection model replaces the original convolution operation with large kernel convolution. The large kernel convolution includes spatial local convolution, spatial long-range convolution, and channel convolution. Spatial local convolution saves on the number of parameters by replacing traditional convolution with separate convolution and point-wise convolution. Spatial long-range convolution incorporates the characteristics of dilated convolution on top of spatial local convolution, increasing the long-range dependency of the convolution. Channel convolution is used to obtain the feature associations between different channels of each pixel.

[0007] Based on the fruit detection results, the scene being photographed is determined to be either a close-up or a distant scene.

[0008] A density clustering algorithm is applied to the fruits in the distant scene to achieve the transition of the shot from a distant scene to a close-up scene;

[0009] Using a priori method, the relative positions of the fruit and branches are obtained; based on the relative positions of the fruit and branches, the branches connected to the fruit in the foreground scene are segmented using an improved instance segmentation algorithm to obtain the branch segmentation results.

[0010] Based on the fruit detection results, the relative position of the fruit and the branch, and the branch segmentation results, the coordinates of the fruit picking point are obtained through the picking point positioning algorithm, and the robotic arm is controlled to complete the automated picking of the fruit.

[0011] As a preferred technical solution, the method of using the improved YOLOv5 object detection model to detect fruit in the scene captured by the depth camera is due to the complex occlusion problem in the fruit growth scene. Therefore, the improved YOLOv5 object detection model is used to process the features of the occluded area to highlight the feature differences between the fruit and the background, while distinguishing overlapping fruits. Specifically:

[0012] First, data augmentation is performed on the input images, that is, each input image is enhanced by color and brightness mixing. Then, the original image and the enhanced image are extracted separately and input into the improved YOLOv5 object detection model. The different detection results output by the improved YOLOv5 object detection model are mutually corrected. Finally, the corrected detection result is output and the subsequent fruit point counting operation is performed.

[0013] The improved YOLOv5 object detection model replaces the original convolution operation with large kernel convolution. Large kernel convolution (LKA) combines the advantages of convolution and self-attention, obtaining local contextual information of the feature map through spatial local convolution and achieving a large receptive field and long-range dependencies through spatial dilated convolution. Simultaneously, it gains adaptability in channel dimensions through channel convolution, preventing excessive computation from severely impacting the algorithm's real-time performance. The calculation formula is as follows:

[0014] LKA = Con 1x1 (DW-D-Conv(DW-Conv(X

[0015] Output = LKA(X)

[0016] Where LKA represents large kernel convolution, X represents the input feature map, DW-D-Conv represents spatial long-range convolution, DW-Conv represents spatial local convolution, and Con represents... 1x1 This is channel convolution.

[0017] As a preferred technical solution, it also includes performing a point counting operation on the fruits in the input image based on the fruit detection results to help determine the distance of the scene. The fruit point counting operation is performed before the input image is input into the improved YOLOv5 object detection model. The image is first cropped into multiple images of the same size, and then input into the improved YOLOv5 object detection model for detection. The points of the multiple images are counted separately, and finally the sum is used as the number of fruits in each scene.

[0018] As a preferred technical solution, the density clustering algorithm is applied to the fruits in the distant scene to achieve the transition of the shot from a distant scene to a close-up scene. The specific process is as follows:

[0019] Based on the fruit detection results, the improved YOLOv5 target model will return four parameters (x, y, h, w) for each target, where x and y represent the center coordinates of the detection box, and h and w represent the height and width of the detection box, respectively. Then, the detection box is drawn using the four parameters, and the center coordinates of each fruit are obtained through the returned target parameters. Clustering is performed based on the center coordinates of each fruit, and fruits whose center coordinates are within a preset range are grouped into the same cluster, thereby obtaining different fruit clusters. The camera is then controlled to move towards the cluster with more fruits to complete the transition from a distant scene to a close-up scene.

[0020] As a preferred technical solution, the prior method involves adding image brightness priors and fruit and leaf growth priors before branch segmentation to locate the relative position of the target branch, thereby improving the performance of the target detection model in low-light scenarios and enhancing the robustness of the target detection model and the accuracy of instance segmentation.

[0021] As a preferred technical solution, the improved instance segmentation algorithm used to segment branches connected to the fruit in the foreground scene employs the SparseInst instance segmentation model. The SparseInst model uses Instance Activation Map (IAM) to highlight foreground instance features in the image and suppresses irrelevant pixels, avoiding the incorrect instance feature localization problem in center / region-based methods. Furthermore, it aggregates instance features from the entire image to achieve instance segmentation. Specifically:

[0022] Given an input feature map X, IAM is represented as f iam (X, Based on research on fruit growth, the IAM parameters are initialized by adding image brightness priors and fruit and leaf growth priors. This allows more activation weights to focus on locations where the image brightness prior is darker and the fruit and leaf growth points, enabling IAM to more accurately locate branch instances and aggregate branch instance features more quickly based on these locations. Therefore, the corrected IAM representation is obtained as follows:) Where T represents the priors for image brightness and fruit growth. This indicates that T is normalized.

[0023] As a preferred technical solution, the coordinates of the fruit picking point are obtained through a picking point positioning algorithm based on the fruit detection results, the relative position of the fruit and the branch, and the branch segmentation results. Specifically:

[0024] First, based on the results of the picking point localization, the detection box and coordinate parameters of the cluster of fruits are calculated according to the (x,y,h,w) parameters of the fruits in the foreground scene. That is, the four extreme values ​​in the x and y directions of all fruits in the cluster are taken. Then, the image edge is extended along the relative direction between the fruit and the branch. The pixel point segmented by the target branch instance is searched at 1 / 3 of the extension line, and the pixel point is used as the candidate picking point coordinates.

[0025] As a preferred technical solution, it also includes constructing a picking point scoring module to score all candidate picking points. The coordinate values ​​(x, y, z) of the pixel positions of each candidate picking point in the scene are obtained through a depth camera. Since each cut in the picking process is from shallow to deep and from the outside to the inside, even if the picking is not completed after each cut, the occluded parts in the scene can be removed. Therefore, by weighting the coordinate values, the candidate picking point with the smaller depth and the closer to the cluster center point scores higher, and the point with the highest final score is taken as the picking point.

[0026] In another aspect, the present invention provides a deep learning-based citrus fruit picking and positioning system, which is applied to the deep learning-based citrus fruit picking and positioning method, including a fruit detection module, a scene determination module, a scene transformation module, a branch segmentation module, and a positioning module;

[0027] The fruit detection module utilizes an improved YOLOv5 object detection model to detect fruits in a scene captured by a depth camera, obtaining fruit detection results. The improved YOLOv5 object detection model replaces the original convolution operation with large kernel convolution. The large kernel convolution includes spatial local convolution, spatial long-range convolution, and channel convolution. Spatial local convolution saves on the number of parameters by replacing traditional convolution with separate convolution and point-wise convolution. Spatial long-range convolution incorporates the characteristics of dilated convolution on top of spatial local convolution, increasing the long-range dependency of the convolution. Channel convolution is used to obtain feature associations between different channels of each pixel.

[0028] The scene determination module is used to determine the distant scene and the distant view scene based on the fruit detection results.

[0029] The scene transition module is used to perform a density clustering algorithm on the fruits in the distant scene to realize the transition of the shot from a distant scene to a close-up scene;

[0030] The branch segmentation module is used to obtain the relative position of the fruit and the branch using a priori method; based on the relative position of the fruit and the branch, the branches connected to the fruit in the foreground scene are segmented using an improved instance segmentation algorithm to obtain the branch segmentation result.

[0031] The positioning module is used to obtain the coordinates of the fruit picking point based on the fruit detection results, the relative position of the fruit and the branch, and the branch segmentation results, using a picking point positioning algorithm, and to control the robotic arm to complete the automated picking of the fruit.

[0032] In another aspect, the present invention provides an electronic device, characterized in that the electronic device comprises:

[0033] At least one processor; and,

[0034] A memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to execute the deep learning-based citrus fruit picking and positioning method.

[0036] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0037] 1. This invention adopts a cluster harvesting strategy; in order to solve the current harvesting needs, a cluster harvesting strategy is used to improve the efficiency of automated harvesting, and the harvesting position is located on the branch connected to the cluster of fruits rather than in a single fruit;

[0038] 2. This invention adds image brightness prior and fruit and leaf growth prior to initialize the IAM module in the SparseInst instance segmentation model, so as to help the model locate the position of the target branch instance faster and more accurately, and improve the real-time performance and accuracy of the branch instance segmentation algorithm.

[0039] 3. The fruit picking point positioning algorithm proposed in this invention has wide applicability. Based on the fruit detection results, branch segmentation results, and the relative position of the fruit and the branch, the picking point positioning algorithm and the picking point scoring module are designed. The picking point positioning method is basically applicable to all fruit picking.

[0040] 4. The fruit picking point positioning algorithm proposed in this invention locates picking points from branches connected to fruit clusters. Therefore, fruit picking can be achieved by cutting branches, thus achieving non-destructive fruit picking. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a deep learning-based method for citrus fruit harvesting and positioning according to an embodiment of the present invention;

[0043] Figure 2 This is a flowchart illustrating the interaction between the software and hardware algorithms for automated fruit harvesting according to an embodiment of the present invention.

[0044] Figure 3 This is a model diagram of the improved SparseInst according to an embodiment of the present invention;

[0045] Figure 4 (a) is a diagram showing the results of fruit detection in an embodiment of the present invention;

[0046] Figure 4 (b) is a diagram showing the results of cluster fruit detection in an embodiment of the present invention;

[0047] Figure 5 This is a diagram illustrating the method for counting the test results of Wogan oranges.

[0048] Figure 6 This is a segmented diagram of a branch connected to a fruit cluster;

[0049] Figure 7 This is a block diagram of a deep learning-based citrus fruit harvesting and positioning system according to an embodiment of the present invention.

[0050] Figure 8 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0052] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0053] This application provides a fruit picking point location method applicable to various complex environmental conditions. It uses a target detection algorithm to locate Wogan mandarin oranges and predict their yield, a lens transfer based on Wogan mandarin orange fruit density clustering algorithm, and a related branch segmentation based on an instance segmentation algorithm. By adding image brightness priors and fruit and leaf growth priors, the IAM part of the SparseInst algorithm is improved, allowing the location of aggregated instance features to be more accurately focused on the target branch, thus improving the accuracy and real-time performance of the entire segmentation algorithm. A picking point location algorithm and a picking point scoring module are designed to locate the picking point using the information obtained above, and a robotic arm is controlled to cut the picking point to complete the fruit picking.

[0054] Example 1: This example uses the automated harvesting of Wogan oranges as an example to describe in detail the fruit harvesting point positioning scheme proposed in this invention.

[0055] Please see Figure 1 , Figure 2 In one embodiment of this application, a deep learning-based method for citrus fruit harvesting and localization is provided, comprising the following steps:

[0056] S1, please refer to Figure 4 (a) Fruit detection is performed on the scene captured by the depth camera using an improved YOLOv5 object detection model to obtain fruit detection results. The improved YOLOv5 object detection model uses large kernel convolution to replace the original convolution operation. The large kernel convolution includes spatial local convolution, spatial long-range convolution, and channel convolution. The spatial local convolution (Depth-wise Conv) is used to save the number of parameters of the convolution by replacing the traditional convolution with separate Conv and point-wise Conv. The spatial long-range convolution is used to integrate the characteristics of dilated convolution on the basis of spatial local convolution, increasing the long-range dependency of the convolution. The channel convolution is used to achieve the function of channel convolution through 1X1 convolution to obtain the feature association between different channels of each pixel.

[0057] S11. Furthermore, using the improved YOLOv5 object detection model to detect fruit in the scene captured by the depth camera is necessary because the fruit growth scene has complex occlusion problems. Therefore, the improved YOLOv5 object detection model is used to process the features of the occluded areas to highlight the feature differences between the fruit and the background, while also distinguishing overlapping fruits. Specifically:

[0058] First, data augmentation is performed on the input images, specifically by applying a color and brightness blending enhancement to each image. Features are then extracted from both the original and enhanced images before being fed into the improved YOLOv5 object detection model. The two different detection results from the improved YOLOv5 model are cross-corrected, and the corrected detection result is output for subsequent fruit point counting. In the improved YOLOv5 object detection model, large-kernel convolutions replace the original convolution operations. While ordinary convolutions can extract contextual feature information, ... The size of the convolutional kernel limits the range of feature extraction. While self-attention mechanisms are often considered to enhance the model's global modeling ability by leveraging long-range dependencies, the computationally intensive nature of self-attention can negatively impact the real-time performance of the detection algorithm. Therefore, large-kernel convolution (LKA) is used instead of ordinary convolution. LKA combines the advantages of convolution and self-attention, obtaining local contextual information from the feature map through spatial local convolution and achieving a large receptive field and long-range dependencies through spatial dilated convolution. Simultaneously, channel convolution provides adaptability to channel dimensions, preventing excessive computation from severely impacting the algorithm's real-time performance. The calculation formula is as follows:

[0059] LKA = Con 1x1 (DW-D-Conv(DW-Conv(X

[0060] Output = LKA(X)

[0061] Where LKA represents large kernel convolution, X represents the input feature map, DW-D-Conv represents spatial long-range convolution, DW-Conv represents spatial local convolution, and Con represents... 1x1 Channel convolution;

[0062] S12, please refer to Figure 5 The results of detecting Wogan oranges based on the improved YOLOv5 object detection model require counting the number of points on the fruits in the input image to help determine the distance of the scene. Specifically:

[0063] To achieve more accurate fruit detection and counting, the image is cropped into four equal parts before being input into the improved YOLOv5 object detection model. Each of these four parts is then input into the improved YOLOv5 object detection model for detection. The points in each part are counted, and the sum is used to determine the number of fruits in each scene.

[0064] S2. Based on the fruit detection results, determine the distant scene and the distant scene in the captured scene;

[0065] S3. Perform a density clustering algorithm on the fruits in the distant scene to realize the transition of the shot from the distant scene to the close-up scene;

[0066] Furthermore, the density clustering algorithm is applied to the fruits in the distant scene to achieve the transition of the shot from a distant scene to a close-up scene. The specific process is as follows:

[0067] Please see Figure 4 (b) Based on the fruit detection results, the improved YOLOv5 target detection model will return four parameters (x, y, h, w) for each target, where x and y represent the center coordinates of the detection box, and h and w represent the height and width of the detection box, respectively. The detection box can be drawn using these four parameters. The center coordinates (x, y) of each Wogan fruit are obtained through the returned target parameters. Clustering is performed based on the center coordinates of each fruit, and Wogan fruits whose center coordinates (x, y) are within a certain preset range are classified into the same cluster. Different fruit clusters are obtained, and the camera is controlled to move towards the cluster with the most fruits to complete the transfer from the distant scene to the near scene, thereby performing the automated fruit picking operation.

[0068] S4. Using a priori methods, obtain the relative positions of the fruit and branches; based on the relative positions of the fruit and branches, perform instance segmentation on the branches connected to the fruit in the foreground scene using an improved instance segmentation algorithm to obtain the branch segmentation results. Please refer to [link to relevant documentation]. Figure 6 .

[0069] S41. Furthermore, the prior method involves adding image brightness priors and fruit and leaf growth priors before branch segmentation, which improves the performance of the target detection model in low-light scenarios, thereby enhancing the robustness of the target detection model and the accuracy of instance segmentation.

[0070] S411. In the observation and research of Wogan mandarin fruit, combined with its morphology and growth habits, if the fruit tree is divided into inner and outer layers from the inside out, it is found that the fruit often grows on the outer layer, while the branches are all in the inner layer. Therefore, clarifying the inside-outside direction of the fruit helps to determine the relative position of the branch and the fruit, and to determine the approximate position of the branch to be divided by the position of the fruit. Specifically:

[0071] During the data collection process, it was found that each image has certain differences in brightness, and the direction from light to dark often corresponds to the direction from the outside to the inside of the fruit tree. Therefore, by adding a brightness prior to the image before branch segmentation, the relative position of the target branch can be determined more quickly. Adding a brightness prior can improve the accuracy and real-time performance of branch instance segmentation.

[0072] S412. Clarifying the internal and external orientation of the fruit tree only provides a general direction for the relative position of the fruit and the branch. Therefore, it is necessary to further clarify the position of the branch to be segmented. Specifically, through the study of crop growth patterns, it has been found that the growth of fruit leaves always points to the branch connected to the fruit leaves. Therefore, it is only necessary to obtain the growth of fruit leaves within the fruit cluster frame to roughly determine the position of the target branch. Therefore, before performing branch instance segmentation and picking point positioning, the prior of fruit leaf growth within the fruit tree cluster is added to help the model to further accurately locate the target branch.

[0073] S42. To meet the needs of efficient Wogan mandarin fruit harvesting, a cluster fruit harvesting strategy is adopted instead of a single fruit harvesting strategy. Therefore, the pruning target for fruit harvesting is focused on the branches connected to the fruit tree clusters. However, within the entire camera scene, there are a large number of irrelevant branches, and the entire segmentation process faces complex background obfuscation and occlusion problems. Therefore, accurately segmenting the target branches within the scene is a key step in the automated Wogan mandarin fruit harvesting process. The entire instance segmentation process must not only ensure accuracy but also have sufficiently high real-time performance to guarantee the smooth operation of the entire automated harvesting process. Therefore, for branches connected to the fruit in the foreground scene, an improved instance segmentation algorithm is used for instance segmentation. The SparseInst segmentation model is selected to achieve branch segmentation, which meets the requirements of segmentation characteristics and real-time performance. The SparseInst instance segmentation model uses Instance Activation Map (IAM) to highlight foreground instance features in the image and suppress irrelevant pixels, avoiding the incorrect instance feature localization problem in center / region-based methods. It also aggregates instance features from the entire image to achieve instance segmentation. Please refer to [link to relevant documentation]. Figure 3 The specific steps for implementing branch segmentation using the SparseInst instance segmentation model are as follows:

[0074] Given an input feature map X, IAM can be represented as f iam(X, Based on research on the growth of Wogan mandarin oranges, the IAM is initialized by adding a brightness prior and a fruit and leaf growth prior. This allows it to focus more activation weights on locations where the brightness prior is darker and the fruit and leaf growth points, enabling the IAM to more accurately locate branch instances and aggregate branch instance features more quickly and effectively based on these locations. Therefore, the modified IAM can be represented as:) Where T represents the priors for image brightness and fruit growth. This indicates that T is normalized; the formula is:

[0075] X∈R Dx(HxW)

[0076]

[0077] A = F iam (X∈R Nx(HxW)

[0078] S5. Based on the fruit detection results, the relative position of the fruit and the branch, and the branch segmentation results, the coordinates of the fruit picking point are obtained through the picking point positioning algorithm, and the robotic arm is controlled to complete the automated picking of the fruit.

[0079] S51. First, based on the results of the picking point localization, and according to the (x, y, h, w) parameters of the fruit in the scene, where x and y represent the center coordinates of the detection box, and h and w represent the height and width of the detection box, calculate the detection box and its coordinate parameters for the cluster of fruits. This involves taking the four extreme values ​​in the x and y directions of all fruits within the cluster. Then, extend the line along the relative direction between the fruit and the branch to the image edge, searching for pixels segmented from the branch instances at 1 / 3 of the extension line, and returning these pixels as candidate picking point coordinates. The formula is:

[0080] g i =(x i ,y i ,h i ,w i )g i ∈G

[0081]

[0082] T = (X, Y, H, W)

[0083]

[0084] Where G represents the coordinate set of the detection frame for Wogan mandarin oranges, M and N represent the boundary coordinates of the detection frame in the x and y directions, respectively, and T represents the coordinates of the fruit tree cluster frame.

[0085]

[0086]

[0087] Where θ represents the relative position and direction of the fruit cluster and the branch, α represents the four values ​​of the picking point positioning algorithm, after determining the α direction, extend along the direction by 1 / 3 distance to vertically search and segment the target branch instance pixels as picking candidate points, B represents the set of pixels of the segmented branch, D represents the set of picking candidate points, and the depth camera captures 640*480 pixels.

[0088] S52. Several candidate picking points obtained through the picking point localization algorithm need to be further filtered. Therefore, a picking point scoring module is constructed to score all candidate picking points. The (x, y, z) coordinate values ​​of the pixel position of each candidate picking point in the scene are obtained through a depth camera. Since each cut in the picking process is from shallow to deep and from the outside to the inside, even if the picking is not completed after each cut, the occluded parts in the scene can be removed. Therefore, the coordinate values ​​are weighted so that the point with the smaller depth and the closer to the cluster center point gets a higher score, and the point with the highest final score is returned as the picking point.

[0089] This application uses the SparseInst network structure for the branch instance segmentation algorithm. The biggest advantage of this network structure is that it has sufficient real-time segmentation to support automated harvesting, while also ensuring that the branch segmentation results meet the requirements for harvesting point positioning.

[0090] In order to improve the efficiency of fruit harvesting, this invention targets the branches connected to fruit clusters rather than individual fruits. Therefore, how to segment the branches connected to fruit clusters is the most critical part of the entire automated fruit harvesting method. It is necessary not only to segment the required branch instances from the complex branches in the image, but also to ensure the real-time performance and accuracy of the segmentation algorithm. Finally, the location of the harvesting point is calculated based on the branch segmentation results and the cluster fruit detection results.

[0091] Example 2

[0092] This invention can also be used for automated harvesting of apple trees.

[0093] Apple fruits have a vertically downward growth habit, and the point where the fruit stem separates is often directly above the fruit. Therefore, the picking point of apple fruits is usually located directly above the fruit. However, even apple trees still conform to the light and dark distribution of the fruit and the branches connecting them, as well as the growth habit of the fruit and leaves pointing towards the branches. The fruit picking point positioning method of the present invention can still be used for the automated picking of apple trees. Moreover, the method used is a cluster picking strategy for branches, which has higher picking efficiency than conventional picking methods. The specific steps for the automated picking of apple trees in Example 2 are the same as those in Example 1, and will not be repeated here.

[0094] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0095] Based on the same idea as the deep learning-based citrus fruit picking and positioning method in the above embodiments, the present invention also provides a deep learning-based citrus fruit picking and positioning system, which can be used to execute the above-described deep learning-based citrus fruit picking and positioning method. For ease of explanation, the structural diagram of the deep learning-based citrus fruit picking and positioning system embodiment only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0096] Please see Figure 7 In another embodiment of this application, a deep learning-based citrus fruit picking and positioning system 100 is provided. The system includes a fruit detection module 101, a scene determination module 102, a scene conversion module 103, a branch segmentation module 104, and a positioning module 105.

[0097] The fruit detection module 101 is used to detect fruits in the scene captured by the depth camera using the YOLOv5 object detection model to obtain fruit detection results. The improved YOLOv5 object detection model uses large kernel convolution to replace the original convolution operation. The large kernel convolution includes spatial local convolution, spatial long-range convolution, and channel convolution. The spatial local convolution is used to save the number of parameters of convolution by replacing traditional convolution with separate convolution and point-wise convolution. The spatial long-range convolution is used to combine the characteristics of dilated convolution on the basis of spatial local convolution, increasing the long-range dependency of convolution. The channel convolution is used to obtain the feature association between different channels of each pixel.

[0098] The scene determination module 102 is used to determine the distant scene and the distant scene based on the fruit detection result.

[0099] The scene transition module 103 is used to perform a density clustering algorithm on the fruits in the distant scene to realize the transition of the shot from a distant scene to a close-up scene.

[0100] The branch segmentation module 104 is used to obtain the relative position of the fruit and the branch using a priori method; based on the relative position of the fruit and the branch, the branches connected to the fruit in the near scene are segmented using an improved instance segmentation algorithm to obtain the branch segmentation result.

[0101] The positioning module 105 is used to obtain the coordinates of the fruit picking point based on the fruit detection results, the relative position of the fruit and the branch, and the branch segmentation results, through a picking point positioning algorithm, and control the robotic arm to complete the automated picking of the fruit.

[0102] It should be noted that the deep learning-based citrus fruit picking and positioning system of the present invention corresponds one-to-one with the deep learning-based citrus fruit picking and positioning method of the present invention. The technical features and beneficial effects described in the embodiments of the deep learning-based citrus fruit picking and positioning method are applicable to the embodiments of the deep learning-based citrus fruit picking and positioning system. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0103] Furthermore, in the embodiments of the deep learning-based citrus fruit picking and positioning system described above, the logical division of each program module is merely an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the deep learning-based citrus fruit picking and positioning system can be divided into different program modules to complete all or part of the functions described above.

[0104] Please see Figure 8 In one embodiment, an electronic device is provided for implementing a deep learning-based method for citrus fruit picking and positioning. The electronic device 200 may include a first processor 201, a first memory 202, and a bus. It may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a deep learning-based citrus fruit picking and positioning program 203.

[0105] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal and external storage units of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of the deep learning citrus fruit picking and positioning program 203, but also to temporarily store data that has been output or will be output.

[0106] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0107] Figure 8 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 8 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0108] The deep learning-based citrus fruit picking and positioning program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0109] An improved YOLOv5 object detection model is used to detect fruit in a scene captured by a depth camera, yielding fruit detection results. The improved YOLOv5 object detection model replaces the original convolution operations with large kernel convolutions, including spatial local convolution, spatial long-range convolution, and channel convolution. Spatial local convolution (Depth-wise Conv) saves on the number of parameters by replacing traditional convolution with separate Conv and point-wise Conv. Spatial long-range convolution, based on spatial local convolution, incorporates the characteristics of dilated convolution, increasing the long-range dependency of the convolution. Channel convolution achieves the function of channel convolution through 1x1 convolution, obtaining the feature correlation between different channels of each pixel.

[0110] Based on the fruit detection results, the scene being photographed is determined to be a distant scene or a distant scene.

[0111] A density clustering algorithm is applied to the fruits in the distant scene to achieve the transition of the shot from a distant scene to a close-up scene;

[0112] Using a priori method, the relative positions of the fruit and branches are obtained; based on the relative positions of the fruit and branches, the branches connected to the fruit in the foreground scene are segmented using an improved instance segmentation algorithm to obtain the branch segmentation results.

[0113] Based on the fruit detection results, the relative position of the fruit and the branch, and the branch segmentation results, the coordinates of the fruit picking point are obtained through the picking point positioning algorithm, and the robotic arm is controlled to complete the automated picking of the fruit.

[0114] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0115] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A deep learning-based method for citrus fruit harvesting and localization, characterized in that, Includes the following steps: An improved YOLOv5 object detection model is used to detect fruit in a scene captured by a depth camera, yielding fruit detection results. The improved YOLOv5 object detection model replaces the original convolution operation with large kernel convolutions. These large kernel convolutions include spatial local convolution, spatial long-range convolution, and channel convolution. The spatial local convolution is used to transform traditional convolutions using separateable convolutions and point-wise convolutions. Conv replacement saves on the number of parameters in convolution; the spatial long-range convolution is used to integrate the characteristics of dilated convolution on the basis of spatial local convolution, increasing the long-range dependency of convolution; the channel convolution is used to obtain the feature association between different channels of each pixel; the improved YOLOv5 object detection model uses large kernel convolution to replace the original convolution operation. Large kernel convolution (LKA) combines the advantages of convolution and self-attention, obtains local context information of feature maps through spatial local convolution, and obtains a large receptive field and long-range dependency through spatial dilated convolution. At the same time, it obtains the adaptability of channel dimension through channel convolution, so that the algorithm's real-time performance will not be seriously affected by excessive computation. The calculation formula is: LKA=Con 1x1 (DW-D-Conv(DW-Conv(X)))Output=LKA(X) Where LKA represents large kernel convolution, X represents the input feature map, DW-D-Conv represents spatial long-range convolution, DW-Conv represents spatial local convolution, and Con represents... 1x1 Channel convolution; Based on the fruit detection results, the scene being photographed is determined to be either a close-up or a distant scene. A density clustering algorithm is applied to the fruits in the distant scene to achieve the transition of the shot from a distant scene to a close-up scene; Using a priori method, the relative positions of the fruit and branches are obtained; based on the relative positions of the fruit and branches, the branches connected to the fruit in the foreground scene are segmented using an improved instance segmentation algorithm to obtain the branch segmentation results. Based on the fruit detection results, the relative position of the fruit and the branch, and the branch segmentation results, the coordinates of the fruit picking point are obtained through the picking point positioning algorithm, and the robotic arm is controlled to complete the automated picking of the fruit. The improved instance segmentation algorithm used for segmenting branches connected to fruits in the foreground scene employs the SparseInst instance segmentation model. The SparseInst model utilizes Instance Activation Map (IAM) to highlight foreground instance features and suppress irrelevant pixels, avoiding the incorrect instance feature localization problem found in center / region-based methods. Furthermore, it aggregates instance features from the entire image to achieve instance segmentation. Specifically: Given an input feature map X, IAM is represented as f iam (X), based on research on fruit growth, the IAM parameters are initialized by adding image brightness priors and fruit and leaf growth priors. This focuses more activation weights on locations where the image brightness prior is darker and the fruit and leaf growth points, allowing IAM to more accurately locate branch instances and aggregate branch instance features more quickly based on these locations. Therefore, the corrected IAM representation is obtained as follows: Where T represents the priors for image brightness and fruit growth. This indicates that T is normalized.

2. The deep learning-based citrus fruit harvesting and positioning method according to claim 1, characterized in that, The use of an improved YOLOv5 object detection model for fruit detection in scenes captured by a depth camera is due to the complex occlusion issues in fruit growth scenes. Therefore, the improved YOLOv5 object detection model processes the features of occluded areas to highlight the feature differences between the fruit and the background, while also distinguishing overlapping fruits. Specifically: First, data augmentation is performed on the input images, that is, each input image is enhanced by color and brightness mixing. Then, the original image and the enhanced image are processed by feature extraction and input into the improved YOLOv5 object detection model. The different detection results output by the improved YOLOv5 object detection model are mutually corrected. Finally, the corrected detection result is output and the subsequent fruit point counting operation is performed.

3. The deep learning-based citrus fruit harvesting and positioning method according to claim 2, characterized in that, It also includes counting fruits in the input image based on the fruit detection results to help determine the distance of the scene. The counting operation is performed before the input image is input into the improved YOLOv5 object detection model. The image is first cropped into multiple images of the same size, and then input into the improved YOLOv5 object detection model for detection. Counting is performed on multiple images, and the sum is finally used as the number of fruits in each scene.

4. The deep learning-based citrus fruit harvesting and positioning method according to claim 1, characterized in that, The density clustering algorithm is applied to the fruits in the distant scene to achieve the transition of the shot from a distant scene to a close-up scene. The specific process is as follows: Based on the fruit detection results, the improved YOLOv5 target model will return four parameters (x, y, h, w) for each target, where x and y represent the center coordinates of the detection box, and h and w represent the height and width of the detection box, respectively. Then, the detection box is drawn using the four parameters, and the center coordinates of each fruit are obtained through the returned target parameters. Clustering is performed based on the center coordinates of each fruit, and fruits whose center coordinates are within a preset range are grouped into the same cluster, thereby obtaining different fruit clusters. The camera is then controlled to move towards the cluster with more fruits to complete the transition from a distant scene to a close-up scene.

5. The deep learning-based citrus fruit harvesting and positioning method according to claim 1, characterized in that, The aforementioned prior method involves adding image brightness priors and fruit and leaf growth priors before branch segmentation to locate the relative position of the target branches. This improves the performance of the target detection model in low-light conditions, enhancing its robustness and the accuracy of instance segmentation.

6. The deep learning-based method for citrus fruit harvesting and positioning according to claim 1, characterized in that, Based on the fruit detection results, the relative position of the fruit and the branch, and the branch segmentation results, the coordinates of the fruit picking point are obtained through a picking point positioning algorithm, specifically as follows: First, based on the results of the picking point localization, the detection box and coordinate parameters of the cluster of fruits are calculated according to the (x,y,h,w) parameters of the fruits in the foreground scene. That is, the four extreme values ​​in the x and y directions of all fruits in the cluster are taken. Then, the image edge is extended along the relative direction between the fruit and the branch. The pixel point segmented by the target branch instance is searched at 1 / 3 of the extension line, and the pixel point is used as the candidate picking point coordinates.

7. The deep learning-based citrus fruit harvesting and positioning method according to claim 6, characterized in that, It also includes a picking point scoring module to score all candidate picking points. The coordinate values ​​(x, y, z) of the pixel positions of each candidate picking point in the scene are obtained through a depth camera. Since each cut in the picking process is from shallow to deep and from the outside to the inside, even if the picking is not completed after each cut, the occluded parts in the scene can be removed. Therefore, by weighting the coordinate values, the candidate picking point with the smaller depth and the closer to the cluster center point scores higher, and the point with the highest final score is taken as the picking point.

8. A deep learning-based citrus fruit harvesting and positioning system, characterized in that, The deep learning-based citrus fruit picking and positioning method applied to any one of claims 1-7 includes a fruit detection module, a scene determination module, a scene transformation module, a branch segmentation module, and a positioning module; The fruit detection module utilizes an improved YOLOv5 object detection model to detect fruits in a scene captured by a depth camera, obtaining fruit detection results. The improved YOLOv5 object detection model replaces the original convolution operation with large kernel convolution. The large kernel convolution includes spatial local convolution, spatial long-range convolution, and channel convolution. Spatial local convolution saves on the number of parameters by replacing traditional convolution with separate convolution and point-wise convolution. Spatial long-range convolution incorporates the characteristics of dilated convolution on top of spatial local convolution, increasing the long-range dependency of the convolution. Channel convolution is used to obtain feature associations between different channels of each pixel. The scene determination module is used to determine the distant scene and the distant view scene based on the fruit detection results. The scene transition module is used to perform a density clustering algorithm on the fruits in the distant scene to realize the transition of the shot from a distant scene to a close-up scene; The branch segmentation module is used to obtain the relative position of the fruit and the branch using a priori method; based on the relative position of the fruit and the branch, the branches connected to the fruit in the foreground scene are segmented using an improved instance segmentation algorithm to obtain the branch segmentation result. The positioning module is used to obtain the coordinates of the fruit picking point based on the fruit detection results, the relative position of the fruit and the branch, and the branch segmentation results, using a picking point positioning algorithm, and to control the robotic arm to complete the automated picking of the fruit.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the deep learning-based citrus fruit picking and positioning method as described in any one of claims 1-7.

Images