Fruit positioning method and device for picking robots

Abstract

The present invention provides a fruit positioning method and device for a picking robot, which relates to the field of artificial intelligence technology. The method collects RGB images and depth images of fruits through a visual device carried by the picking robot, and inputs the RGB images of the fruits into a trained MT-YOLO model to obtain a two-dimensional bounding box and a segmentation mask of the fruits output by the MT-YOLO model, and then maps the two-dimensional bounding box of the fruits to a three-dimensional space to obtain a three-dimensional view cone, and based on the three-dimensional view cone, the segmentation mask and the depth image, generates a point cloud of the exposed part of the fruit, thereby determining the radius and the center of mass position of the fruit. The fruit positioning method for a picking robot provided by the present invention realizes the automatic and accurate detection of the obscured fruits, and improves the positioning accuracy of the picking robot for the fruits.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a fruit positioning method and device for a picking robot. Background Art

[0002] The primary task of a fruit picking robot (such as apples, oranges, pears and other spherical fruits) is to use information perception technology to accurately identify and locate the fruit, which is the key to ensuring the robot's picking performance.

[0003] However, in complex orchard environments, fruits are often blocked by branches and leaves, and the fruits are closely distributed. Traditional single-task-based target detection models are difficult to meet the requirements of real-time and accuracy. In addition, existing technologies mostly use expensive depth sensors or multi-algorithm cascade methods to improve fruit positioning accuracy, but the computational complexity is high and the cost is too high, making it difficult to actually apply to large-scale orchard automation operations. Summary of the invention

[0004] The present invention provides a fruit positioning method and device for a picking robot, which are used to solve the technical problems in the prior art that the picking robot has high cost and low positioning accuracy for fruits.

[0005] The present invention provides a fruit positioning method for a picking robot, comprising the following steps: The RGB image and depth image of the fruit are collected through the visual equipment of the picking robot; The RGB image of the fruit is input into the trained MT-YOLO model to obtain the two-dimensional bounding box and segmentation mask of the fruit output by the MT-YOLO model; the MT-YOLO model is trained based on the historical fruit data set collected in the real orchard; the MT-YOLO model includes a backbone network, a first neck network, a second neck network, a detection network and a segmentation network; the backbone network is used to obtain feature maps of different scales through multi-scale feature extraction; the first neck network is used to weightedly fuse feature maps of different scales; the second neck network is used to step-by-step fuse feature maps of different scales; the detection network is used to output the two-dimensional bounding box of the fruit; the segmentation network is used to output the segmentation mask of the fruit; Mapping the two-dimensional bounding box of the fruit into a three-dimensional space to obtain a three-dimensional viewing cone; Based on the three-dimensional viewing cone, the segmentation mask and the depth image, a point cloud of the exposed part of the fruit is generated; the point cloud is distributed on the surface of the fruit within the three-dimensional viewing cone; Determine the radius and center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit; Based on the radius and the center of mass position of the fruit, a three-dimensional model of the fruit is obtained.

[0006] According to a fruit positioning method for a picking robot provided by the present invention, the RGB image of the fruit is input into a trained MT-YOLO model to obtain a two-dimensional bounding box and a segmentation mask of the fruit output by the MT-YOLO model, including: Based on the RGB image of the fruit, obtaining feature maps of different scales through a backbone network; Based on the feature maps of different scales, the feature maps of different scales are weightedly fused through a first neck network to obtain a first feature map; and based on the feature maps of different scales, the feature maps of different scales are stepwise fused through a second neck network to obtain a second feature map; Based on the first feature map, a two-dimensional bounding box of the fruit is obtained through a detection network; and based on the second feature map, a segmentation mask of the fruit is obtained through a segmentation network.

[0007] According to a fruit positioning method for a picking robot provided by the present invention, the backbone network is a convolutional neural network constructed based on wavelet transform.

[0008] According to a fruit positioning method for a picking robot provided by the present invention, mapping the two-dimensional bounding box of the fruit into a three-dimensional space to obtain a three-dimensional viewing cone includes: Based on the two-dimensional bounding box of the fruit, determine the center point of the fruit in the two-dimensional plane; Based on the center point of the fruit in the two-dimensional plane, obtaining a fruit center line; Based on the center line of the fruit and the depth image of the fruit, the two-dimensional bounding box of the fruit is mapped to the three-dimensional space to obtain a three-dimensional viewing cone.

[0009] According to a fruit positioning method for a picking robot provided by the present invention, the radius and the center of mass position of the fruit are determined based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit, including: Based on the point cloud of the exposed part of the fruit, determine the centroid position of the point cloud; and based on the two-dimensional bounding box of the fruit, determine the radius of the fruit; Constructing a three-dimensional sphere with the centroid position of the point cloud as the sphere center and the radius of the fruit as the radius; The farthest point of the two intersection points between the three-dimensional sphere surface and the center line of the fruit is taken as the center of mass position of the fruit.

[0010] According to a fruit positioning method for a picking robot provided by the present invention, the far point is an intersection point of two intersection points of the three-dimensional spherical surface and the fruit center line that is far away from a visual device of the picking robot.

[0011] The present invention also provides a fruit positioning device for a picking robot, comprising the following modules: The acquisition module is used to acquire RGB images and depth images of the fruit through the visual device of the picking robot; The detection module is used to input the RGB image of the fruit into the trained MT-YOLO model to obtain the two-dimensional bounding box and segmentation mask of the fruit output by the MT-YOLO model; the MT-YOLO model is trained based on the historical fruit data set collected in the real orchard; the MT-YOLO model includes a backbone network, a first neck network, a second neck network, a detection network and a segmentation network; the backbone network is used to obtain feature maps of different scales through multi-scale feature extraction; the first neck network is used to weightedly fuse feature maps of different scales; the second neck network is used to step-by-step fuse feature maps of different scales; the detection network is used to output the two-dimensional bounding box of the fruit; the segmentation network is used to output the segmentation mask of the fruit; A three-dimensional module, used for mapping the two-dimensional bounding box of the fruit into a three-dimensional space to obtain a three-dimensional viewing cone; A generating module, used for generating a point cloud of the exposed part of the fruit based on the three-dimensional viewing cone, the segmentation mask and the depth image; the point cloud is distributed on the surface of the fruit within the three-dimensional viewing cone; A positioning module, for determining the radius and the center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit; The simulation module is used to obtain a three-dimensional model of the fruit based on the radius and the center of mass position of the fruit.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements any of the above-mentioned fruit positioning methods for a picking robot.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the fruit positioning method for a picking robot as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned fruit positioning methods for a picking robot.

[0015] The fruit positioning method for a picking robot provided by the present invention collects RGB images and depth images of the fruit through a visual device carried by the picking robot, and inputs the RGB image of the fruit into a trained MT-YOLO model to obtain a two-dimensional bounding box and a segmentation mask of the fruit output by the MT-YOLO model, thereby realizing automatic and accurate detection of blocked fruits, and then mapping the two-dimensional bounding box of the fruit to a three-dimensional space to obtain a three-dimensional viewing cone, and generating a point cloud of the exposed part of the fruit based on the three-dimensional viewing cone, the segmentation mask and the depth image, thereby determining the radius and the center of mass position of the fruit, and realizing real-time, accurate and automatic three-dimensional positioning of the fruit without the need for an additional point cloud processing network and without using an expensive depth sensor, thereby reducing costs and improving the positioning accuracy of the picking robot for the fruit. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of a fruit positioning method for a picking robot provided by the present invention.

[0018] Figure 2 It is a schematic diagram of image acquisition in a fruit positioning method for a picking robot provided by the present invention.

[0019] Figure 3 It is one of the structural schematic diagrams of the MT-YOLO model in the fruit positioning method for a picking robot provided by the present invention.

[0020] Figure 4 This is the second structural schematic diagram of the MT-YOLO model in the fruit positioning method for a picking robot provided by the present invention.

[0021] Figure 5 It is a schematic diagram of cone positioning of a fruit positioning method for a picking robot provided by the present invention.

[0022] Figure 6 This is the second flow chart of a fruit positioning method for a picking robot provided by the present invention.

[0023] Figure 7 It is a structural schematic diagram of a fruit positioning device for a picking robot provided by the present invention.

[0024] Figure 8It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] With the aging of the population and the continuous increase in labor costs around the world, orchards are facing a serious labor shortage problem. The traditional manual picking method is not only time-consuming and labor-intensive, but also difficult to meet the needs of modern orchards for efficient and refined management. Therefore, harvesting robots have been widely studied in the past few decades.

[0026] The primary task of a harvesting robot is to use information perception technology to accurately identify and locate the fruit, which is the key to ensuring the robot's picking performance. However, in a complex orchard environment, the fruit is often blocked by branches and leaves, and the fruits are closely distributed. Traditional single-task-based target detection models are difficult to meet the requirements of real-time and accuracy. In addition, existing technologies often use expensive depth sensors or multi-algorithm cascade methods to improve the accuracy of fruit positioning, but the computational complexity is high and the cost is too high, making it difficult to actually apply to large-scale orchard automation operations.

[0027] Currently, most fruit recognition research focuses on analyzing single tasks (such as classification, target detection and segmentation) or multiple task cascades in the complex working environment of orchards. There are few studies on simultaneously obtaining multi-dimensional information of occluded fruits.

[0028] However, during robot operation, fruits are often hidden in the overlapping and messy branches and leaves, and the picking efficiency will be affected by these obstacles. In order to accurately identify and segment the contours of the target fruit while taking into account the recognition rate, multi-dimensional information integration is required. In addition, in the actual orchard environment, fruits usually grow closely adjacent to each other, and the fruits are easily damaged by external forces or hard objects, which reduces the quality of the fruits after picking. Therefore, accurately locating the three-dimensional spatial coordinates of the fruit is an important prerequisite for robot non-destructive picking.

[0029] At present, a lot of research has been done on fruit positioning, but the occlusion of obstacles such as branches, leaves, and fruits makes the generated point cloud incomplete, which makes it still challenging to accurately locate fruits in complex orchard environments.

[0030] In the prior art, there are two main methods for using RGBD sensors to obtain target 3D point clouds in harvesting robot operations: 1. Using expensive depth sensors; 2. Using multiple algorithms to cascade the obtained point clouds. However, with the rapid growth in the application demand for harvesting robots in agricultural scenarios, efficient and real-time positioning methods suitable for consumer-grade sensors have become an urgent problem to be solved.

[0031] In view of the above problems faced by the existing picking robot vision system, the present invention provides a fruit positioning method for the picking robot to solve the multi-inspection and re-inspection problems of the traditional instance segmentation network under occlusion. On this basis, a point cloud processing method based on a three-dimensional cone is proposed to determine the spatial position of the fruit and estimate its size, so as to achieve accurate positioning of the fruit and improve the fruit picking efficiency and success rate of the picking robot.

[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Combine the following Figures 1 to 8 The present invention describes a fruit positioning method and device for a picking robot.

[0034] Figure 1 This is one of the flow charts of a fruit positioning method for a picking robot provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 101: Collect the RGB image and depth image of the fruit through the visual device of the picking robot.

[0035] Specifically, harvesting robots usually use stereo vision systems to locate fruits, such as binocular vision cameras based on optical geometry principles and RGBD sensors based on structured light (SL) and time of flight (ToF) technologies.

[0036] However, sensors based on structured light technology are more sensitive to changes in ambient light, and their accuracy decreases significantly at longer distances, making them unsuitable for robot operations. Therefore, in order to simulate the visual system of a picking robot and apply the fruit positioning method for a picking robot proposed in the present invention, in an embodiment of the present invention, seven different sensors, namely, cameras Azure, M4 pro, Realsense L515 based on ToF imaging, binocular cameras ZED 2i, Realsense d455, Realsense d435i, and Mid-70 lidar, are used, and are controlled by a portable laptop to conduct tests in a real orchard.

[0037] Figure 2 Schematic diagram of image acquisition in a fruit positioning method for a picking robot provided by the present invention, such as Figure 2 As shown, the above 7 sensors are mounted on the mobile collection platform of the picking robot. The mobile collection platform consists of an automatic navigation vehicle and a collection platform. The collection platform is fixed at a distance of 1.5m from the bottom of the automatic navigation vehicle. At the same time, the embodiment of the present invention also sets up a reference system consisting of an RGBD camera and a laser radar. Since the laser radar has high distance resolution and robustness to sunlight, the embodiment of the present invention uses the measurement value of the laser radar as the true value of the surface position of the fruit.

[0038] The relative position and direction of each camera and lidar on the mobile acquisition platform are determined, so the rigid body transformation parameters (rotation matrix and translation vectors ), and complete the unified transformation of multi-sensor coordinates through the following formula:

[0039] in, represents the point cloud coordinates in the camera coordinate system, Represents the point cloud coordinates after conversion to the laser radar coordinate system. is the rotation matrix, is the translation vector. After the coordinate transformation is completed, the data collected by all RGBD cameras are uniformly mapped to the reference coordinate system of the lidar.

[0040] In order to improve the controllability and repeatability of the test, the embodiment of the present invention defines a fixed collection point in front of each fruit tree to ensure that the collection platform takes the same point at different distances, thereby improving the accuracy of the captured image. The collection platform simulates the RGB image and depth image of the fruit taken by the picking robot, so that the subsequent model can identify and detect the fruit.

[0041] Step 102: input the RGB image of the fruit into the trained MT-YOLO model to obtain a two-dimensional bounding box and a segmentation mask of the fruit output by the MT-YOLO model; the MT-YOLO model is trained based on a historical fruit data set collected in a real orchard; the MT-YOLO model includes a backbone network, a first neck network, a second neck network, a detection network and a segmentation network; the backbone network is used to obtain feature maps of different scales through multi-scale feature extraction; the first neck network is used to weightedly fuse feature maps of different scales; the second neck network is used to step-by-step fuse feature maps of different scales; the detection network is used to output a two-dimensional bounding box of the fruit; and the segmentation network is used to output a segmentation mask of the fruit.

[0042] Specifically, during robot operation, fruits are often hidden in the overlapping and messy branches and leaves, and the picking efficiency will be affected by these obstacles, making it difficult to accurately identify and segment the target fruit outline while taking into account the recognition rate. In order to achieve real-time, accurate, and automated detection of obscured fruits, the present invention designs a lightweight multi-task learning network, namely the MT-YOLO model, which performs target detection and semantic segmentation through the MT-YOLO model, integrates multi-dimensional information, and thus provides more detailed fruit information for robot picking.

[0043] Figure 3 1 is one of the structural schematic diagrams of the MT-YOLO model in the fruit positioning method for a picking robot provided by the present invention, such as Figure 3 shown.

[0044] The MT-YOLO model can be divided into five parts: the backbone network (Backbone), the first neck network (Detect Neck), the second neck network (Segment Neck), the detection network (Detect), and the segmentation network (Segment). The backbone network can be regarded as a shared feature extraction network, which is used to obtain feature maps of different scales through multi-scale feature extraction.

[0045] Figure 4 This is the second structural diagram of the MT-YOLO model in the fruit positioning method for a picking robot provided by the present invention, such as Figure 4 shown.

[0046] The backbone network consists of a series of convolutional layers. Due to the excellent performance of YOLOv8 in detection tasks, the embodiment of the present invention selects the backbone network of YOLOv8 as the feature extraction network of the MT-YOLO model. In order to better extract features of different levels and abstraction levels in the input data, the present invention also proposes a new feature fusion module, namely C2f_WTConv. The embodiment of the present invention adds C2f_WTConv to the feature extraction network of the MT-YOLO model to form the backbone network of the MT-YOLO model.

[0047] Traditional convolutional layers can capture a wider range of contextual information by increasing the size of the convolution kernel, but this method will cause the number of parameters to grow at a quadratic level, and the performance will saturate or even decrease after reaching a certain receptive field. In addition, larger convolution kernels are prone to over-parameterization, which affects the generalization ability of the network. In order to solve this problem, the embodiment of the present invention adds wavelet convolution to Bottleneck in C2f_WTConv, and decomposes the input signal into different frequency components through wavelet transform, thereby operating on multiple frequency scales, and realizing multi-scale feature extraction and enhancement. The backbone network of the MT-YOLO model designed in the embodiment of the present invention significantly improves the performance and receptive field of the model, and enhances the feature extraction ability of the model.

[0048] In the MT-YOLO model, two neck networks are also designed: the first neck network corresponds to the fruit object detection task, and the second neck network corresponds to the fruit semantic segmentation task. The two neck networks enhance the feature representation capability by fusing feature maps of different scales from the backbone network.

[0049] The first neck network adopts a feature fusion path weighted bidirectional feature pyramid network (BiFPN) structure based on the path aggregation network (PAN) (see Figure 3 ), the nodes with only one input are deleted, and a new aggregation path is added between the original input node and the output node. Through multiple superpositions, efficient bidirectional cross-scale connection and weighted feature fusion are achieved, thereby obtaining more feature information of the occluded fruit.

[0050] In the second neck network, the Feature Pyramid Networks (FPN) fuses features of different scales step by step (see Figure 3 ), which effectively realizes the integration of multi-scale information.

[0051] The detection network and segmentation network process the feature maps from the two neck networks respectively and make predictions for each task, including the corresponding 2D bounding box and the label information of the segmented object (i.e., segmentation mask).

[0052] The detection network takes the feature map output by the first neck network as input and uses the decoupled head structure of the YOLOv8 series model. This structure adopts the anchor-free target assigner Anchor-Free Assigner to classify positive and negative samples, does not rely on predefined anchor boxes to predict the target position, and outputs the two-dimensional bounding box of the fruit.

[0053] The segmentation network uses a lightweight segmentation head to process the feature map output by the second neck network and output the segmentation mask of the fruit. The structure is as follows Figure 3 The segmentation network only contains three Conv modules for extracting context information and one DeConv module for restoring the resolution to the original image size. Combined with the good feature extraction capability of FPN in the backbone network and the second neck network, the computational burden of the decoder can be effectively reduced, the overall structure can be lightweight, and the model performance can be improved.

[0054] In the object detection task, the detection network recognizes fruits at the image level, while the segmentation network segments fruits at the pixel level.

[0055] In the MT-YOLO model, the backbone network and the two neck networks together form the encoder, while the detection network and the segmentation network can be regarded as decoders. There are no redundant shared blocks between different decoders to reduce computational consumption and maintain the one-stage structure of the network. By combining the results of target detection and semantic segmentation, accurate detection of occluded fruits is achieved.

[0056] In order to improve the performance of the MT-YOLO model, the embodiment of the present invention pre-trains the model using a historical fruit dataset collected in a real orchard.

[0057] The MT-YOLO model trains the entire network by combining the loss of the detection network and the loss of the segmentation network. The formula is as follows:

[0058] represents the loss of the MT-YOLO model, represents the loss of the detection network, represents the loss of the segmentation network.

[0059] The detection network includes a classification branch and a bounding box prediction branch. The classification branch and the bounding box branch have different loss functions. For classification tasks, the Binary Cross Entropy (BCE) loss function is used. For the bounding box prediction task, the Distribution Focal Loss (DFL) loss function and the Complete IoU (CIoU) loss function are used to better solve the category imbalance problem and improve the detection performance of small objects. The loss function expression in the detection network is as follows:

[0060] represents the loss of the detection network, represents the BCE loss, represents the CIoU loss, represents the DFL loss, represents the CIoU loss coefficient, represents the DFL loss coefficient, Represents the BCE loss coefficient.

[0061]

[0062] represents the corresponding true label, Represents the predicted classification of each object.

[0063]

[0064]

[0065]

[0066] represents the DFL loss, represents the i-th value, represents the i+1th value, Indicates the upper integer of the true value y of the bounding box coordinate, represents the true value of the coordinates of the i-th bounding box, The true value (a decimal) representing the bounding box coordinates.

[0067]

[0068] represents the comprehensive intersection-over-union ratio, represents the intersection-and-union ratio, represents the Euclidean distance between the predicted point and the true point, represents the center point of the prediction box, represents the center point of the ground-truth box, Indicates the diagonal length of the minimum circumscribed rectangle of the two boxes.

[0069] in, and Are the control proportional coefficient values, respectively determined by the following formula:

[0070]

[0071] Indicates the width of the real box, Indicates the height of the real frame, Indicates width, Indicates height, Represents the intersection and union ratio.

[0072] In the segmentation network, a method combining the BCE loss function and the IoU loss function is used. Among them, the BCE loss performs fine-grained supervision on the classification at the pixel level to ensure that each pixel of the model is classified correctly as much as possible; while the IoU loss provides a global area supervision method to ensure the shape and spatial consistency between the predicted area and the real area. This combination can improve the performance of the model at both the pixel level and the overall target area.

[0073] The loss function expression of the segmentation network is as follows:

[0074] represents the segmentation network loss, represents the BCE loss coefficient, represents the BCE loss, represents the IoU loss coefficient, represents the IoU loss.

[0075] After the MT-YOLO model is trained, the RGB image of the fruit collected in real time is input into the MT-YOLO model to obtain the two-dimensional bounding box of the fruit output by the detection network in the MT-YOLO model and the segmentation mask of the fruit output by the segmentation network in the MT-YOLO model.

[0076] The embodiment of the present invention obtains a two-dimensional bounding box and a segmentation mask of the fruit through the MT-YOLO model, thereby achieving accurate, real-time, and automatic detection of the obscured fruit.

[0077] Step 103: Map the two-dimensional bounding box of the fruit to a three-dimensional space to obtain a three-dimensional viewing cone.

[0078] Specifically, the two-dimensional bounding box output by the MT-YOLO model is used to determine the approximate position of the fruit in the RGB image, that is, the center point of the fruit in the two-dimensional plane. Due to the quasi-symmetry of spherical fruits, the center point of the bounding box can be approximated to the central projection point of the fruit, thereby generating the center line of the fruit. Since the two-dimensional bounding box output by the MT-YOLO model can estimate the full size of the occluded fruit, based on the camera coordinate system x Axis and y The center information of the axis can more accurately determine the initial position of the fruit in the image coordinate system.

[0079] Figure 5 : is a schematic diagram of a cone positioning method for a fruit picking robot provided by the present invention, such as Figure 5 shown.

[0080] In order to further determine the fruit zThe embodiment of the present invention maps the two-dimensional bounding box of the fruit into three-dimensional space to form a three-dimensional cone structure. Specifically, the near plane and far plane of the three-dimensional cone are defined by the depth range of the two-dimensional bounding box in the depth image, and the center line is formed by the ray connecting the origin of the camera coordinate system to the center of the far plane of the cone.

[0081] The construction of the 3D viewing cone not only provides the spatial area where the fruit may exist, but also provides geometric constraints for the subsequent screening of point clouds and accurate estimation of the 3D centroid.

[0082] Step 104: Generate a point cloud of the exposed part of the fruit based on the three-dimensional viewing cone, the segmentation mask and the depth image; the point cloud is distributed on the surface of the fruit within the three-dimensional viewing cone.

[0083] Specifically, according to the segmentation mask output by the MT-YOLO model combined with its corresponding depth image, a point cloud within the mask can be generated, and the generated point cloud is distributed on the surface of the fruit sphere within the three-dimensional viewing cone.

[0084] The MT-YOLO model can accurately segment the exposed part of the fruit (i.e., the segmentation mask) and estimate the 2D size of the complete fruit (i.e., the 2D bounding box). On this basis, the embodiment of the present invention converts the 2D information obtained by the MT-YOLO model into 3D centroid coordinates through the cone method and fits a point cloud on the surface of the fruit sphere. At the same time, the centroid position of the point cloud is obtained, providing a basis for subsequent fruit positioning.

[0085] Step 105: Determine the radius and center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit.

[0086] Specifically, first, the radius of the fruit is calculated using the 2D bounding box and the camera intrinsics. :

[0087] In the formula, is the focal length scaling factor of the camera, is the width of the 2D bounding box in pixel coordinates, is the centroid of the point cloud exist z Coordinates on the axis.

[0088] Then, Construct a sphere with a radius of The three-dimensional sphere is obtained, and two intersection points of the three-dimensional sphere surface and the center line of the three-dimensional viewing cone (i.e. the center line of the fruit) are obtained.

[0089] Finally, the farthest point among the two intersection points is taken as the centroid of the fruit. .

[0090] The embodiment of the present invention calculates the radius of the fruit and determines the center of mass of the fruit through the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit. It achieves accurate three-dimensional positioning of the fruit without the need for an additional point cloud processing network or expensive depth sensors, thereby improving the efficiency of fruit picking by the picking robot.

[0091] Step 106: Obtain a three-dimensional model of the fruit based on the radius and the center of mass position of the fruit.

[0092] Specifically, the embodiment of the present invention fits a three-dimensional model of the fruit according to the radius and center of mass position of the fruit, provides the robot with more detailed fruit information, facilitates the precise picking of the picking robot, and further improves the efficiency of the picking robot in picking fruits.

[0093] The present invention provides a fruit positioning method for a picking robot. The method collects RGB images and depth images of fruits through a visual device carried by the picking robot, inputs the RGB images of fruits into a trained MT-YOLO model, obtains a two-dimensional bounding box and a segmentation mask of the fruits output by the MT-YOLO model, realizes automatic and accurate detection of blocked fruits, and then maps the two-dimensional bounding box of the fruits into a three-dimensional space to obtain a three-dimensional viewing cone. Based on the three-dimensional viewing cone, the segmentation mask and the depth image, a point cloud of the exposed part of the fruits is generated, thereby determining the radius and the center of mass position of the fruits. Without the need for an additional point cloud processing network and without the use of expensive depth sensors, the method realizes real-time, accurate and automatic three-dimensional positioning of the fruits, thereby reducing costs and improving the positioning accuracy of the picking robots for the fruits.

[0094] Optionally, inputting the RGB image of the fruit into a trained MT-YOLO model to obtain a two-dimensional bounding box and a segmentation mask of the fruit output by the MT-YOLO model comprises: Based on the RGB image of the fruit, obtaining feature maps of different scales through a backbone network; Based on the feature maps of different scales, the feature maps of different scales are weightedly fused through a first neck network to obtain a first feature map; and based on the feature maps of different scales, the feature maps of different scales are stepwise fused through a second neck network to obtain a second feature map; Based on the first feature map, a two-dimensional bounding box of the fruit is obtained through a detection network; and based on the second feature map, a segmentation mask of the fruit is obtained through a segmentation network.

[0095] Specifically, Figure 3As shown in the figure, the MT-YOLO model can be divided into five parts: the backbone network (Backbone), the first neck network (Detect Neck), the second neck network (Segment Neck), the detection network (Detect), and the segmentation network (Segment). The steps of the MT-YOLO model for fruit target detection are as follows: First, the RGB image of the fruit is input into the trained MT-YOLO model, and multi-scale feature extraction is performed through the backbone network to obtain feature maps of different scales.

[0096] Then, in order to better fuse the features extracted by the backbone network, the first neck network and the second neck network were designed respectively. The first neck network weightedly fuses the feature maps of different scales to obtain the first feature map, and the second neck network gradually fuses the feature maps of different scales to obtain the second feature map. The two neck networks fuse the feature maps of different scales from the backbone network to enhance the feature representation capability, significantly improve the backbone network's ability to extract and fuse multi-scale features, and reduce the computational complexity of the backbone network.

[0097] Finally, the first feature map is input into the detection network, which outputs the two-dimensional bounding box of the fruit; and the second feature map is input into the segmentation network, which outputs the segmentation mask of the fruit.

[0098] The embodiment of the present invention fully utilizes the multi-scale features of the RGB image of the fruit through the trained MT-YOLO model to accurately identify and segment the outline of the fruit. At the same time, due to the reduced computational complexity, the model recognition rate is improved, thereby achieving accurate and efficient detection of the fruit.

[0099] Optionally, the backbone network is a convolutional neural network constructed based on wavelet transform.

[0100] Specifically, Figure 4 As shown, the backbone network consists of a series of convolutional layers. Due to the excellent performance of YOLOv8 in detection tasks, the embodiment of the present invention selects the backbone network of YOLOv8 as the feature extraction network of the MT-YOLO model. In order to better extract features of different levels and abstraction levels in the input data, the present invention also proposes a new feature fusion module, namely C2f_WTConv.

[0101] The embodiment of the present invention adds C2f_WTConv to the feature extraction network of the MT-YOLO model to form the backbone network of the MT-YOLO model. The traditional convolution layer can capture a wider range of context information by increasing the size of the convolution kernel, but this method will cause the number of parameters to grow at a quadratic level, and the performance will saturate or even decrease after reaching a certain receptive field. In addition, larger convolution kernels are prone to over-parameterization, which affects the generalization ability of the network.

[0102] To solve this problem, the embodiment of the present invention adds wavelet convolution in Bottleneck in C2f_WTConv, that is, decomposes the input signal into different frequency components through wavelet transform, thereby operating on multiple frequency scales, realizing multi-scale feature extraction and enhancement.

[0103] The backbone network of the MT-YOLO model designed in the embodiment of the present invention significantly improves the performance and receptive field of the model, enhances the feature extraction capability of the model, and thereby improves the accuracy of model detection and recognition.

[0104] Optionally, mapping the two-dimensional bounding box of the fruit into a three-dimensional space to obtain a three-dimensional viewing cone comprises: Based on the two-dimensional bounding box of the fruit, determine the center point of the fruit in the two-dimensional plane; Based on the center point of the fruit in the two-dimensional plane, obtaining a fruit center line; Based on the center line of the fruit and the depth image of the fruit, the two-dimensional bounding box of the fruit is mapped to the three-dimensional space to obtain a three-dimensional viewing cone.

[0105] Specifically, after the MT-YOLO model outputs the two-dimensional bounding box of the fruit, the approximate position of the fruit in the RGB image is determined according to the two-dimensional bounding box of the fruit, that is, the center point of the fruit in the two-dimensional plane. Due to the quasi-symmetry of the spherical fruit, the center point of the bounding box can be approximated to the central projection point of the fruit, thereby generating the center line of the fruit.

[0106] Since the two-dimensional bounding box output by the MT-YOLO model can estimate the full size of the occluded fruit, based on the camera coordinate system x Axis and y The center information of the axis can more accurately determine the initial position of the fruit in the image coordinate system.

[0107] like Figure 5 As shown, in order to further determine the fruit zThe embodiment of the present invention maps the two-dimensional bounding box of the fruit into three-dimensional space to form a three-dimensional cone structure. Specifically, the near plane and far plane of the three-dimensional cone are defined by the depth range of the two-dimensional bounding box in the depth image, and the center line is formed by the ray connecting the origin of the camera coordinate system to the center of the far plane of the cone.

[0108] The embodiment of the present invention constructs a three-dimensional viewing cone, which not only provides a spatial area where fruits may exist, but also provides geometric constraints for subsequent point cloud screening and accurate estimation of the three-dimensional center of mass.

[0109] Optionally, determining the radius and center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit includes: Based on the point cloud of the exposed part of the fruit, determine the centroid position of the point cloud; and based on the two-dimensional bounding box of the fruit, determine the radius of the fruit; Constructing a three-dimensional sphere with the centroid position of the point cloud as the sphere center and the radius of the fruit as the radius; The farthest point of the two intersection points between the three-dimensional sphere surface and the center line of the fruit is taken as the center of mass position of the fruit.

[0110] Specifically, first, according to the point cloud of the exposed part of the fruit, the centroid position of the point cloud is obtained. , and the radius of the fruit is calculated using the fruit's 2D bounding box and the camera's intrinsic parameters :

[0111] In the formula, is the focal length scaling factor of the camera, is the width of the 2D bounding box in pixel coordinates, is the centroid of the point cloud exist z Coordinates on the axis.

[0112] Then, Construct a sphere with a radius of The three-dimensional sphere is obtained, and two intersection points of the three-dimensional sphere surface and the center line of the three-dimensional viewing cone (i.e. the center line of the fruit) are obtained.

[0113] Finally, the farthest point among the two intersection points is taken as the centroid of the fruit. .

[0114] The embodiment of the present invention calculates the radius of the fruit and determines the center of mass of the fruit through the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit. It achieves accurate three-dimensional positioning of the fruit without the need for an additional point cloud processing network or expensive depth sensors, thereby improving the efficiency of fruit picking by the picking robot.

[0115] Optionally, the far point is an intersection point of two intersection points of the three-dimensional spherical surface and the center line of the fruit, which is farthest from the visual device of the picking robot.

[0116] Specifically, since the RGB image of the fruit captured by the visual device of the picking robot is the part of the fruit close to or facing the visual device, and the back of the fruit cannot be captured, the point cloud generated based on the model's detection results of the part of the fruit close to or facing the visual device is also distributed on the front surface of the fruit close to the visual device. Therefore, the three-dimensional sphere constructed based on the centroid of the front surface point cloud, the intersection point far from the visual device of the picking robot is closer to or even the same as the real centroid position of the fruit.

[0117] The embodiment of the present invention uses the intersection point of the two intersection points of the three-dimensional sphere surface and the center line of the fruit, which is farthest from the visual device of the picking robot, as the center of mass position of the fruit. This can achieve accurate positioning of the fruit in three-dimensional space, provide a basis for subsequent fitting of the three-dimensional model of the fruit, realize the integration of multi-dimensional information, and further provide the picking robot with more detailed fruit information, thereby improving the efficiency of fruit picking.

[0118] Based on the above embodiment, the embodiment of the present invention applies the above fruit positioning method for a picking robot to a picking robot, and conducts an example of actual operation test for apple picking. Figure 6 FIG. 2 is a flow chart of a fruit positioning method for a picking robot provided by the present invention. Figure 6 The specific steps are as follows: (1) Data collection. RGBD images were collected by camera. A Realsense D435i camera was used at a distance of 160 cm from the trunk of the 13th tree to collect the RGB image and depth image of the apple.

[0119] (2) Obtain two-dimensional information through the MT-YOLO model. The RGB image of the fruit is processed through the trained MT-YOLO model to obtain the MT-YOLO model output of the apple's two-dimensional bounding box and segmentation mask (i.e., segmented image).

[0120] (3) Apple centerline and 3D viewing cone. The apple centerline is obtained based on the apple’s 2D bounding box, and the apple’s 2D bounding box is mapped to 3D space to generate a 3D viewing cone, so as to generate a point cloud of the apple surface within the 3D viewing cone.

[0121] (4) Determine the center of mass. Determine the center of mass of the apple based on the point cloud that blocks the visible part of the apple.

[0122] In the embodiment of the present invention, apples are often hidden in the overlapping and messy branches and leaves during the operation of the robot. The picking efficiency will be affected by these obstacles, making it difficult to accurately identify and segment the target apple contour, resulting in a low recognition rate. The MT-YOLO model is used for target detection and semantic segmentation, and the exposed part of the apple is accurately segmented and the two-dimensional size of the complete fruit is estimated. The cone method is used for precise positioning, and multi-dimensional information is integrated. The two-dimensional information is converted into three-dimensional centroid coordinates and a three-dimensional model of the apple is fitted, which facilitates the robot's precise picking and improves the operation efficiency of the picking robot.

[0123] A fruit positioning device for a picking robot provided by the present invention is described below. The fruit positioning device for a picking robot described below and the fruit positioning method for a picking robot described above can be referenced to each other.

[0124] Based on any of the above embodiments, Figure 7 Schematic diagram of the structure of a fruit positioning device for a picking robot provided by the present invention. Figure 7 The embodiment of the present invention provides a fruit positioning device for a picking robot, including a collection module 701, a detection module 702, a three-dimensional module 703, a generation module 704, a positioning module 705 and a simulation module 706, wherein: The acquisition module 701 is used to collect the RGB image and depth image of the fruit through the visual equipment of the picking robot; the detection module 702 is used to input the RGB image of the fruit into the trained MT-YOLO model to obtain the two-dimensional bounding box and segmentation mask of the fruit output by the MT-YOLO model; the MT-YOLO model is trained based on the historical fruit data set collected in the real orchard; the MT-YOLO model includes a backbone network, a first neck network, a second neck network, a detection network and a segmentation network; the backbone network is used to obtain feature maps of different scales through multi-scale feature extraction; the first neck network is used to weightedly fuse feature maps of different scales; the second neck network is used to weightedly fuse feature maps of different scales; The local network is used to fuse feature maps of different scales step by step; the detection network is used to output a two-dimensional bounding box of the fruit; the segmentation network is used to output a segmentation mask of the fruit; the three-dimensional module 703 is used to map the two-dimensional bounding box of the fruit to a three-dimensional space to obtain a three-dimensional viewing cone; the generation module 704 is used to generate a point cloud of the exposed part of the fruit based on the three-dimensional viewing cone, the segmentation mask and the depth image; the point cloud is distributed on the surface of the fruit within the three-dimensional viewing cone; the positioning module 705 is used to determine the radius and the center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit; the simulation module 706 is used to obtain a three-dimensional model of the fruit based on the radius and the center of mass position of the fruit.

[0125] The present invention provides a fruit positioning device for a picking robot. The device collects RGB images and depth images of fruits through a visual device carried by the picking robot, and inputs the RGB images of fruits into a trained MT-YOLO model to obtain a two-dimensional bounding box and a segmentation mask of the fruits output by the MT-YOLO model, thereby realizing automatic and accurate detection of blocked fruits. The two-dimensional bounding box of the fruits is then mapped to a three-dimensional space to obtain a three-dimensional viewing cone. Based on the three-dimensional viewing cone, the segmentation mask and the depth image, a point cloud of the exposed part of the fruits is generated, thereby determining the radius and the center of mass position of the fruits. Without the need for an additional point cloud processing network and without the need for expensive depth sensors, the device realizes real-time, accurate and automatic three-dimensional positioning of the fruits, thereby reducing costs and improving the positioning accuracy of the picking robots for the fruits.

[0126] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the fruit positioning method for the picking robot, and the method includes: The RGB image and depth image of the fruit are collected through the visual equipment of the picking robot; The RGB image of the fruit is input into the trained MT-YOLO model to obtain the two-dimensional bounding box and segmentation mask of the fruit output by the MT-YOLO model; the MT-YOLO model is trained based on the historical fruit data set collected in the real orchard; the MT-YOLO model includes a backbone network, a first neck network, a second neck network, a detection network and a segmentation network; the backbone network is used to obtain feature maps of different scales through multi-scale feature extraction; the first neck network is used to weightedly fuse feature maps of different scales; the second neck network is used to step-by-step fuse feature maps of different scales; the detection network is used to output the two-dimensional bounding box of the fruit; the segmentation network is used to output the segmentation mask of the fruit; Mapping the two-dimensional bounding box of the fruit into a three-dimensional space to obtain a three-dimensional viewing cone; Based on the three-dimensional viewing cone, the segmentation mask and the depth image, a point cloud of the exposed part of the fruit is generated; the point cloud is distributed on the surface of the fruit within the three-dimensional viewing cone; Determine the radius and center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit; Based on the radius and the center of mass position of the fruit, a three-dimensional model of the fruit is obtained.

[0127] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0128] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the fruit positioning method for the picking robot provided by the above methods, the method includes: The RGB image and depth image of the fruit are collected through the visual equipment of the picking robot; The RGB image of the fruit is input into the trained MT-YOLO model to obtain the two-dimensional bounding box and segmentation mask of the fruit output by the MT-YOLO model; the MT-YOLO model is trained based on the historical fruit data set collected in the real orchard; the MT-YOLO model includes a backbone network, a first neck network, a second neck network, a detection network and a segmentation network; the backbone network is used to obtain feature maps of different scales through multi-scale feature extraction; the first neck network is used to weightedly fuse feature maps of different scales; the second neck network is used to step-by-step fuse feature maps of different scales; the detection network is used to output the two-dimensional bounding box of the fruit; the segmentation network is used to output the segmentation mask of the fruit; Mapping the two-dimensional bounding box of the fruit into a three-dimensional space to obtain a three-dimensional viewing cone; Based on the three-dimensional viewing cone, the segmentation mask and the depth image, a point cloud of the exposed part of the fruit is generated; the point cloud is distributed on the surface of the fruit within the three-dimensional viewing cone; Determine the radius and center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit; Based on the radius and the center of mass position of the fruit, a three-dimensional model of the fruit is obtained.

[0129] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the fruit positioning method for a picking robot provided by the above methods, the method comprising: The RGB image and depth image of the fruit are collected through the visual equipment of the picking robot; The RGB image of the fruit is input into the trained MT-YOLO model to obtain the two-dimensional bounding box and segmentation mask of the fruit output by the MT-YOLO model; the MT-YOLO model is trained based on the historical fruit data set collected in the real orchard; the MT-YOLO model includes a backbone network, a first neck network, a second neck network, a detection network and a segmentation network; the backbone network is used to obtain feature maps of different scales through multi-scale feature extraction; the first neck network is used to weightedly fuse feature maps of different scales; the second neck network is used to step-by-step fuse feature maps of different scales; the detection network is used to output the two-dimensional bounding box of the fruit; the segmentation network is used to output the segmentation mask of the fruit; Mapping the two-dimensional bounding box of the fruit into a three-dimensional space to obtain a three-dimensional viewing cone; Based on the three-dimensional viewing cone, the segmentation mask and the depth image, a point cloud of the exposed part of the fruit is generated; the point cloud is distributed on the surface of the fruit within the three-dimensional viewing cone; Determine the radius and center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit; Based on the radius and the center of mass position of the fruit, a three-dimensional model of the fruit is obtained.

[0130] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0132] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0133] It should also be noted that the terms "target", "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0134] "Determine B based on A" in the embodiments of the present application means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "Determine B based on A and C", "Determine B based on A, C and E", "Determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "When A meets the first condition, use the first method to determine B"; for another example, "When A meets the second condition, determine B", etc.; for another example, "When A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "When A meets the first condition, use the first method to determine C, and further determine B based on C", etc.

[0135] In the present invention, the term "plurality" refers to two or more than two, and other quantifiers are similar to them.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fruit positioning method for a picking robot, characterized in that: include: The RGB image and depth image of the fruit are collected through the visual equipment of the picking robot; The RGB image of the fruit is input into the trained MT-YOLO model to obtain the two-dimensional bounding box and segmentation mask of the fruit output by the MT-YOLO model; the MT-YOLO model is trained based on the historical fruit data set collected in the real orchard; the MT-YOLO model includes a backbone network, a first neck network, a second neck network, a detection network and a segmentation network; the backbone network is used to obtain feature maps of different scales through multi-scale feature extraction; the first neck network is used to weightedly fuse feature maps of different scales; the second neck network is used to step-by-step fuse feature maps of different scales; the detection network is used to output the two-dimensional bounding box of the fruit; the segmentation network is used to output the segmentation mask of the fruit; Mapping the two-dimensional bounding box of the fruit into a three-dimensional space to obtain a three-dimensional viewing cone; Generate a point cloud of the exposed part of the fruit based on the three-dimensional viewing cone, the segmentation mask and the depth image; The point cloud is distributed on the surface of the fruit within the three-dimensional viewing cone; Determine the radius and center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit; Based on the radius and the center of mass position of the fruit, a three-dimensional model of the fruit is obtained.

2. The fruit positioning method for a picking robot according to claim 1, characterized in that: The step of inputting the RGB image of the fruit into the trained MT-YOLO model to obtain a two-dimensional bounding box and a segmentation mask of the fruit output by the MT-YOLO model comprises: Based on the RGB image of the fruit, obtaining feature maps of different scales through a backbone network; Based on the feature maps of different scales, the feature maps of different scales are weightedly fused through a first neck network to obtain a first feature map; and based on the feature maps of different scales, the feature maps of different scales are stepwise fused through a second neck network to obtain a second feature map; Based on the first feature map, a two-dimensional bounding box of the fruit is obtained through a detection network; and based on the second feature map, a segmentation mask of the fruit is obtained through a segmentation network.

3. The fruit positioning method for a picking robot according to claim 2, characterized in that: The backbone network is a convolutional neural network constructed based on wavelet transform.

4. The fruit positioning method for a picking robot according to claim 1, characterized in that: Mapping the two-dimensional bounding box of the fruit to a three-dimensional space to obtain a three-dimensional viewing cone includes: Based on the two-dimensional bounding box of the fruit, determine the center point of the fruit in the two-dimensional plane; Based on the center point of the fruit in the two-dimensional plane, obtaining a fruit center line; Based on the center line of the fruit and the depth image of the fruit, the two-dimensional bounding box of the fruit is mapped to the three-dimensional space to obtain a three-dimensional viewing cone.

5. The fruit positioning method for a picking robot according to claim 1, characterized in that: The step of determining the radius and the center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit comprises: Based on the point cloud of the exposed part of the fruit, determine the centroid position of the point cloud; and based on the two-dimensional bounding box of the fruit, determine the radius of the fruit; Constructing a three-dimensional sphere with the centroid position of the point cloud as the sphere center and the radius of the fruit as the radius; The farthest point of the two intersection points between the three-dimensional sphere surface and the center line of the fruit is taken as the center of mass position of the fruit.

6. The fruit positioning method for a picking robot according to claim 5, characterized in that: The far point is the intersection point of the two intersection points of the three-dimensional spherical surface and the center line of the fruit, which is farthest from the visual device of the picking robot.

7. A fruit positioning device for a picking robot, characterized in that: include: The acquisition module is used to acquire RGB images and depth images of the fruit through the visual device of the picking robot; The detection module is used to input the RGB image of the fruit into the trained MT-YOLO model to obtain the two-dimensional bounding box and segmentation mask of the fruit output by the MT-YOLO model; the MT-YOLO model is trained based on the historical fruit data set collected in the real orchard; the MT-YOLO model includes a backbone network, a first neck network, a second neck network, a detection network and a segmentation network; the backbone network is used to obtain feature maps of different scales through multi-scale feature extraction; the first neck network is used to weightedly fuse feature maps of different scales; the second neck network is used to step-by-step fuse feature maps of different scales; the detection network is used to output the two-dimensional bounding box of the fruit; the segmentation network is used to output the segmentation mask of the fruit; A three-dimensional module, used for mapping the two-dimensional bounding box of the fruit into a three-dimensional space to obtain a three-dimensional viewing cone; A generating module, for generating a point cloud of the exposed part of the fruit based on the three-dimensional viewing cone, the segmentation mask and the depth image; The point cloud is distributed on the surface of the fruit within the three-dimensional viewing cone; A positioning module, for determining the radius and the center of mass position of the fruit based on the point cloud of the exposed part of the fruit and the two-dimensional bounding box of the fruit; The simulation module is used to obtain a three-dimensional model of the fruit based on the radius and the center of mass position of the fruit.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the fruit positioning method for a picking robot is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fruit positioning method for a picking robot is implemented as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the fruit positioning method for a picking robot is implemented as described in any one of claims 1 to 6.

Images