Fruit pose real-time estimation method based on multi-frame information fusion

Abstract

The invention discloses a fruit pose real-time estimation method based on multi-frame information fusion. The method comprises the following steps: acquiring observed image data and a sensor pose; performing fruit semantic instance segmentation on the image data to obtain a current frame fruit instance and a depth point cloud, performing data association on a fruit semantic instance segmentation result and a fruit map representation constructed based on a truncated symbol distance field in combination with a sensor pose, and determining a data association relationship between the current frame and a historical frame; according to the data association relationship, the current frame fruit instance and the depth point cloud, updating the fruit map representation; and extracting effective point clouds on the surfaces of the fruit instances from the updated map representation, determining the attitude information of the fruits through filtering and principal component analysis, estimating the omnibearing size information of the fruits, and finally determining the real-time poses of the fruits. Real-time and high-precision estimation of the fruit poses can be achieved, lightweight deployment of the fruit picking robot in a facility agriculture scene is achieved, and the operation success rate and the intelligent level are remarkably improved.

Description

A Real-Time Fruit Pose Estimation Method Based on Multi-Frame Information Fusion Technical Field

[0001] This invention belongs to the field of perception technology for agricultural harvesting robots, specifically involving a real-time fruit pose estimation method based on multi-frame information fusion. Background Technology

[0002] With the continuous rise in labor costs, smart agriculture, with intelligence at its core, has become an inevitable trend in modern agricultural development. In the development of smart agriculture, the level of automation and intelligence of fruit-harvesting robots directly affects agricultural production efficiency. Accurate real-time estimation of fruit posture is the core prerequisite for automated robotic harvesting, and its accuracy directly determines the success rate of the harvesting robotic arm.

[0003] Currently, fruit pose estimation methods are mainly divided into two categories: single-frame observation methods and multi-frame fusion methods. Single-frame observation methods directly calculate fruit pose through semantic segmentation and point cloud processing of a single-frame RGBD image. These methods (e.g., patent document CN118941630A provides a method and device for berry fruit picking and positioning, and document titled "2D pose estimation of multiple tomato fruit-bearing systems for robotic harvesting" provides a two-dimensional pose estimation method for multi-tomato fruit-bearing systems for robotic harvesting) have the advantage of high computational speed. However, limited by the perspective of single-frame observation and the influence of environmental noise, they are prone to problems such as missing information in fruit occlusion areas and large pose estimation errors, making it difficult to meet the requirements of high-precision harvesting. Multi-frame fusion methods improve the robustness of pose estimation by fusing information from multiple frames of observation. Existing technologies mostly employ point cloud stitching or feature matching fusion methods. However, traditional point cloud stitching methods are prone to point cloud misalignment when dealing with occlusion and dynamic scenes, and have extremely high requirements for observation and localization, which can easily lead to a decrease in fusion accuracy. Feature matching-based methods are highly dependent on fruit surface texture; for fruits with smooth surfaces or simple textures, feature extraction is difficult, data association accuracy is low, and this affects the stability of pose estimation. Recently, some novel multi-frame fusion methods have attempted to use deep learning for multi-frame fusion fruit completion to determine fruit pose information and harvesting location (for example, the paper titled "Panoptic mapping with fruit completion and pose estimation for horticultural robots" provides a panoramic mapping method for horticultural robots with fruit completion and pose estimation functions, and the paper titled "Improving robotic fruitharvesting within cluttered environments through 3D shape completion" provides a method to improve the efficiency of robots harvesting fruit in cluttered environments through 3D shape completion). However, these methods rely on highly accurate training data, cannot be reused in multiple application scenarios, and cannot achieve lightweight edge deployment, limiting the widespread application of agricultural harvesting robots.

[0004] In summary, existing fruit pose estimation methods suffer from problems such as incomplete single-frame observation information, inaccurate multi-frame data association, high complexity of fusion models, and difficulty in implementing edge deployment, making it difficult to achieve real-time and accurate fruit pose estimation in complex agricultural scenarios. Therefore, designing a method that can effectively fuse multi-frame information, improve the accuracy of data association, and achieve complete pose reconstruction under partial observations has become an urgent technical challenge to be solved in the field of agricultural robot perception. Summary of the Invention

[0005] In view of the above, the purpose of this invention is to address the problems existing in current fruit pose estimation methods, such as incomplete single-frame observation information, inaccurate multi-frame data association, low fusion accuracy, and difficulty in size restoration under some observations. This invention provides a real-time fruit pose estimation method based on multi-frame information fusion. By constructing a fruit map representation using a truncated symbolic distance field (TSDF), optimizing the multi-frame data association logic, and combining principal component analysis to achieve accurate solutions for pose and size, this method enables lightweight deployment of fruit-harvesting robots in facility agriculture scenarios. Ultimately, it achieves real-time, high-precision estimation of the six-degree-of-freedom pose of the fruit, significantly improving the success rate and intelligence level of operations, and providing reliable technical support for the automated operation of agricultural harvesting robots.

[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution: Firstly, an embodiment of the present invention provides a real-time fruit pose estimation method based on multi-frame information fusion, comprising the following steps: synchronously acquiring image data observed by a sensor and the corresponding sensor pose; performing fruit semantic instance segmentation on the image data to obtain the fruit instance and depth point cloud of the current frame, and then combining the sensor pose to perform data association between the fruit semantic instance segmentation result and the fruit map representation constructed based on the truncated symbolic distance field, determining the data association relationship between the current frame and historical frames; updating the truncated symbolic distance field map representation of the fruit according to the data association relationship and the fruit instance and depth point cloud of the current frame, realizing the fusion and updating of multi-frame observation information; extracting the effective point cloud of the surface of each fruit instance from the updated map representation, determining the fruit's pose information through filtering and principal component analysis, and estimating the fruit's size information including height, left and right width, and front and back width, and obtaining the real-time pose of the fruit by combining the pose information and size information.

[0007] Preferably, the synchronous acquisition of image data observed by the sensor and the corresponding sensor pose includes: acquiring color and depth images synchronously collected by a depth camera mounted on a robotic arm, and calculating the camera sensor pose based on the joint angles of the robotic arm using a forward kinematics model; or, acquiring color and depth images synchronously collected by a depth camera mounted on a mobile platform, and running the RGBD-SLAM algorithm to calculate the camera sensor pose in real time.

[0008] Preferably, the step of performing fruit semantic instance segmentation on the image data to obtain the fruit instance and depth point cloud of the current frame includes: performing fruit semantic instance segmentation on a single-frame color image using a semantic segmentation model to obtain the pixel-level mask region of each fruit instance; and combining the depth image to convert the pixel-level mask region of each segmented fruit instance into the corresponding three-dimensional depth point cloud of the current single-frame fruit instance.

[0009] Preferably, the step of combining the sensor pose to segment the fruit semantic instances and data association with the fruit map representation constructed based on the truncated symbol distance field to determine the data association relationship between the current frame and historical frames includes: transforming the depth point cloud of each fruit instance in the current frame to the world coordinate system according to the sensor pose; calculating the distance from the transformed point cloud to the truncated symbol distance field surface of each historical fruit instance in the fruit map representation; if the overlapping area of ​​the current fruit instance point cloud and a certain historical fruit instance satisfies the association condition, then it is determined that the two are the same fruit and a data association relationship is established; otherwise, the current fruit instance point cloud is initialized as a new fruit instance and a corresponding fruit map representation is established.

[0010] Preferably, the step of updating the truncated symbolic distance field map representation of the fruit based on the data association relationship and the fruit instance and depth point cloud of the current frame to realize the fusion and update of observation information of multiple frames includes: for fruit instances with established data association relationships, fusing the current fruit point cloud with the historical truncated symbolic distance field map, updating the symbolic distance value and weight of voxels in the truncated symbolic distance field by weighted averaging, realizing the cumulative fusion of information of multiple frames, and improving the structural representation of the fruit.

[0011] Preferably, principal component analysis is achieved by calculating the covariance matrix of the filtered point cloud and performing eigenvalue decomposition. The direction of the eigenvector corresponding to the largest eigenvalue is taken as the height axis of the fruit, and the directions of the other two mutually orthogonal eigenvectors are taken as the left and right width and front and back width axes of the fruit, respectively. A reference coordinate system is established in combination with the centroid of the point cloud to determine the posture information of the fruit.

[0012] Preferably, the estimation of the fruit's dimensions, including height, left and right width, and front and back width, includes: projecting the effective point cloud onto the fruit's reference coordinate system determined by principal component analysis to obtain the coordinates of the point cloud in three axes, which are respectively used as the initial height, initial left and right width, and initial front and back width of the fruit; if the reliability of the initial front and back width is lower than the preset requirement due to the limitation of the observation angle, the value of the initial left and right width is assigned to the initial front and back width as the final front and back width dimension used.

[0013] Secondly, embodiments of the present invention also provide a real-time fruit pose estimation device based on multi-frame information fusion, implemented using the aforementioned real-time fruit pose estimation method based on multi-frame information fusion, comprising: an information input module, a data association module, a multi-frame fusion module, and a pose generation module; the information input module is used to synchronously acquire image data observed by the sensor and the corresponding sensor pose; the data association module is used to perform fruit semantic instance segmentation on the image data to obtain the fruit instance and depth point cloud of the current frame, and then combine the sensor pose to perform data association between the fruit semantic instance segmentation result and the fruit map representation constructed based on the truncated symbolic distance field to determine the data association relationship between the current frame and historical frames; the multi-frame fusion module is used to update the truncated symbolic distance field map representation of the fruit according to the data association relationship and the fruit instance and depth point cloud of the current frame, realizing the fusion and updating of multi-frame observation information; the pose generation module is used to extract the effective point cloud of the surface of each fruit instance from the updated map representation, determine the pose information of the fruit through filtering and principal component analysis, and estimate the size information of the fruit including height, left and right width and front and back width, and obtain the real-time pose of the fruit by combining the pose information and size information.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described real-time fruit pose estimation method based on multi-frame information fusion when executing the computer program.

[0015] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for real-time estimation of fruit pose based on multi-frame information fusion.

[0016] Compared with the prior art, the beneficial effects of the present invention include at least: (1) improving the integrity and accuracy of pose estimation: the present invention uses a truncated symbolic distance field to construct a fruit map representation, which can effectively fuse point cloud information from multiple frames of observation, make up for the viewpoint occlusion and information loss problems of single frame observation, and ensure the accuracy of the fused map through a weighted update strategy, providing a complete three-dimensional structural basis for pose estimation.

[0017] (2) Improve the reliability of multi-frame data association: This invention associates single-frame fruit point cloud with truncated symbol distance field map, and combines spatial location and symbol distance matching degree for dual judgment, avoiding the dependence of traditional feature matching methods on fruit surface texture, significantly improving the accuracy of data association in complex scenes, and ensuring the effective fusion of multi-frame information.

[0018] (3) Achieving complete pose restoration under partial observation: In agricultural scenarios, fruits are often partially obscured by leaves and branches. This invention assumes that the left and right widths of the fully observable fruits are equal to the front and back widths. Combined with the pose and size information obtained from principal component analysis, it can accurately restore the complete three-dimensional size and six-degree-of-freedom pose of the fruits, meeting the actual needs of harvesting operations.

[0019] (4) Real-time performance and engineering applicability: The algorithm computation complexity of each step of the present invention is moderate, the process of information input, data association, multi-frame fusion and pose generation is efficiently connected, the computation efficiency of neighborhood filtering and principal component analysis is high, which can meet the response requirements of real-time operation of agricultural harvesting robots, and can be adapted to different types of RGBD sensor mounting methods, with a wide range of engineering application scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 is a flowchart of the real-time fruit pose estimation method based on multi-frame information fusion provided in an embodiment of the present invention; Figure 2 is a schematic diagram of the agricultural dual-arm harvesting robot platform used in experimental verification provided in an embodiment of the present invention; Figure 3 is a schematic diagram of the three-dimensional representation of the truncated symbolic distance field map after multi-frame fusion provided in an embodiment of the present invention; Figure 4 is a schematic diagram of the fruit pose information generated in real time based on the fused point cloud provided in an embodiment of the present invention; Figure 5 is a schematic diagram of the structure of the real-time fruit pose estimation device based on multi-frame information fusion provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0023] Hardware Platform: A differential-drive agricultural mobile robot serves as the platform. A laptop (Intel Core i7-11800H @2.3GHz) is mounted on the upper layer of the robot for data acquisition and algorithm processing. It employs an RGBD sensor (Intel RealSense D405) with a frame rate of 30fps, a color image resolution of 848×480, and a depth image resolution of 848×480. The robot moves with its dual arms during harvesting, and the robotic arms are equipped with high-precision encoders to acquire pose information.

[0024] Software platform: The operating system is Ubuntu 20.04 LTS, and the robot software framework is ROS Noetic (Robot Operating System). The deep learning framework used is PyTorch 2.0, the semantic segmentation model is YOLOv8, and the pose information corresponding to the observations is combined with the robot arm pose and extrinsic parameters to obtain the results.

[0025] Based on the above-mentioned software and hardware platform deployment, as shown in Figure 1, this embodiment of the invention provides a real-time fruit pose estimation method based on multi-frame information fusion, which specifically includes the following steps: S1, information input.

[0026] In this embodiment, as shown in Figure 2, an agricultural robot platform is harvesting strawberries in a laboratory environment. In the ROS environment, the RGBD sensor is activated via the sensor ROS driver node to collect color and depth images of the facility agriculture scene in real time, and the data is transmitted to the data processing unit via ROS topics.

[0027] When a servo-driven robotic arm is used to mount the sensor, the angles of each joint are obtained through the robotic arm's ROS driver package. A TensorFlow transformation is then published using the robotic arm's forward kinematics model, and the sensor's pose information (X, Y, Z, Roll, Pitch, Yaw) is calculated using extrinsic parameters. Furthermore, if a mobile platform is used, the RGBD-SLAM algorithm is run to receive RGBD sensor data and output the sensor's pose information in the world coordinate system in real time (including position coordinates and attitude angle parameters), thus completing the coordinate system unification of the observed data.

[0028] S2, Data Association.

[0029] In this embodiment, considering that during actual harvesting, due to foliage obstruction and the limited field of view of the RGBD sensor, the current viewpoint cannot effectively observe all the fruits in the harvesting area. This easily leads to low accuracy and poor reliability in size and pose estimation based on single-frame data, failing to meet the requirements of robust robot grasping. Therefore, this invention innovatively introduces a fruit map representation based on truncated symbolic distance field (TSDF) into the fruit harvesting operation scenario. It maintains global fruit map information obtained from historical observations in real time and associates fruit instances that can be effectively observed from the current viewpoint with global fruit instances as information input for subsequent multi-frame information fusion. This is specifically achieved through the following steps.

[0030] S2.1 Load the YOLOv8 semantic segmentation model obtained by pre-training on fruit sample images, perform semantic instance segmentation of fruit on a single frame color image, detect and identify fruit instances, and publish the corresponding fruit instance mask image and category information.

[0031] S2.2, Match the depth image of the corresponding frame according to the acquisition time, convert the pixel coordinates of the segmented fruit instance mask region into three-dimensional point cloud coordinates through the intrinsic parameters of the RGBD sensor, and map the depth image to the three-dimensional point cloud coordinate system to obtain the three-dimensional depth point cloud of each fruit instance.

[0032] S2.3 Construct a fruit map representation based on a truncated symbolic distance field. This representation stores the symbolic distance information of the fruit in units of voxels, where the symbolic distance is the signed distance from the voxel to the fruit surface (negative inside, positive outside, and zero on the surface).

[0033] S2.4 After transforming the 3D depth point cloud of each fruit instance in the current frame to the world coordinate system according to the sensor pose, calculate the absolute value of the distance from the truncation symbol of each historical fruit instance in the fruit map representation to the field surface.

[0034] If the IoU (Intersection over Union) threshold between the current fruit instance point cloud and a certain historical fruit instance in the image plane is greater than or equal to 0.15, then the overlapping area of ​​the current fruit instance to which the point cloud belongs and the historical fruit instance satisfies the association condition, and the two are the same fruit, and a data association relationship is established; otherwise, the current fruit instance point cloud (satisfying a certain size and not associated with any historical fruit instance) is initialized as a new fruit instance, and a corresponding fruit map representation is established.

[0035] S3, multi-frame fusion.

[0036] In this embodiment, to overcome the inherent defect of incomplete single-frame observation information, the present invention fuses the current and corresponding historical fruit objects, where the historical fruit objects are global fruit instances obtained based on previous multi-frame observations. By fusing and updating multi-frame information, a more complete and accurate 3D representation of the fruit is obtained, thereby significantly improving the accuracy and robustness of subsequent pose and size estimation.

[0037] Specifically, based on the data association results, the unique historical fruit object corresponding to the current fruit point cloud is identified, and the current fruit point cloud is fused with the historical truncated symbol distance field map. The symbol distance values ​​and weights of voxels in the truncated symbol distance field are updated by weighted averaging, thereby realizing the accumulation and fusion of information from multiple frames and improving the structural representation of the fruit.

[0038] Figure 3 shows the three-dimensional representation of the truncated symbolic distance field map after multi-frame fusion. The three sub-images on the left correspond to the original color image, fruit instance segmentation, and depth image from top to bottom, respectively. The right side shows the fruit grid map information generated and updated by the truncated symbolic distance field.

[0039] S4, pose generation.

[0040] In this embodiment, addressing the problem that in practical applications, only a portion of the fruit object is observable, resulting in only partial surface point clouds of the fruit, which prevents existing methods from accurately acquiring fruit pose information, and that deep learning-based shape completion methods are insufficient to meet the stringent requirements of real-time harvesting, this invention proposes estimating the fruit's principal inertial axis through principal component analysis. Based on this, assuming that the left-right width equals the front-back width, the missing dimensional information is completed, ultimately achieving real-time and accurate fruit pose generation under partial observation conditions.

[0041] S4.1, traverse the truncated symbol distance field map voxels of each fruit, and filter out voxels whose absolute symbol distance is less than a certain threshold (set to 0.875 times the voxel size) and whose updated weights meet the requirements (in the example, the voxel has been updated) based on the voxel's symbol distance information and weights. The corresponding point cloud information is taken as the valid point cloud.

[0042] S4.2, the K-Nearest Neighbor (KNN) neighborhood filtering algorithm is used to process the effective point cloud, calculate the number of effective point clouds in the neighborhood range of each point, and adaptively adjust the filtering radius according to the density of the fruit point cloud. Outliers with fewer than a certain threshold (set to 50 in the example) in the neighborhood are removed to ensure the consistency of the point cloud.

[0043] S4.3, the filtered point cloud is centered using principal component analysis algorithm, the covariance matrix of the point cloud is calculated, and the eigenvalues ​​and eigenvectors of the covariance matrix are solved by singular value decomposition. The direction of the eigenvector corresponding to the largest eigenvalue is taken as the height axis of the fruit, and the directions of the other two mutually orthogonal eigenvectors are taken as the left and right width and front and back width axes of the fruit, respectively. A reference coordinate system is established in combination with the centroid of the point cloud to determine the posture information of the fruit.

[0044] S4.4 Project the fruit point cloud onto the reference coordinate system, calculate the fruit's height, left and right width and other size parameters, and based on the limitations of some observation scenarios, assume that the fully observable left and right width is equal to the front and back width, supplement the front and back width parameters as the final front and back width dimensions used.

[0045] S4.5, combining the pose information obtained from principal component analysis, and considering the current fruit size parameters, the six-degree-of-freedom pose of the fruit is finally generated. Figure 4 shows the real-time generated fruit pose information, where the red point cloud represents the progressively generated fruit surface point cloud information, and the arrows represent the estimated pose information.

[0046] S5, self-harvesting operation.

[0047] In actual operation, real-time estimated fruit pose information is published via ROS topics. After receiving the six-DOF pose information of the fruit, the picking robot arm plans its motion path and performs the picking operation. At the same time, the RGBD sensor continuously collects new observation data, repeating the above steps to achieve real-time updates and estimations of the poses of all fruits in the scene until the entire picking task is completed.

[0048] In summary, the real-time fruit pose estimation method based on multi-frame information fusion provided by the embodiments of the present invention can effectively solve the fundamental problems of inaccurate fruit pose estimation and non-lightweight calculation process in the prior art. By fusing multi-frame observation information through truncated symbolic distance field, it can realize lightweight deployment of fruit picking robots in facility agriculture scenarios, thereby significantly improving the success rate and intelligence level of operations.

[0049] Based on the same inventive concept, as shown in Figure 5, this embodiment of the invention also provides a real-time fruit pose estimation device 500 based on multi-frame information fusion, including: an information input module 510, a data association module 520, a multi-frame fusion module 530, and a pose generation module 540.

[0050] The information input module 510 is used to synchronously acquire the image data observed by the sensor and the corresponding sensor pose.

[0051] The data association module 520 is used to perform fruit semantic instance segmentation on image data to obtain fruit instances and depth point clouds in the current frame. Then, it combines the sensor pose to perform data association between the fruit semantic instance segmentation results and the fruit map representation constructed based on the truncated symbolic distance field, and determines the data association relationship between the current frame and historical frames.

[0052] The multi-frame fusion module 530 is used to update the truncated symbol distance field map representation of the fruit based on the data correlation and the fruit instance and depth point cloud of the current frame, so as to realize the fusion and update of multi-frame observation information.

[0053] The pose generation module 540 is used to extract the effective point cloud of the surface of each fruit instance from the updated map representation, determine the pose information of the fruit through filtering and principal component analysis, and estimate the size information of the fruit, including height, left and right width and front and back width. The pose information and size information are combined to obtain the real-time pose of the fruit.

[0054] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described method for real-time estimation of fruit pose based on multi-frame information fusion when executing the computer program.

[0055] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for real-time estimation of fruit pose based on multi-frame information fusion.

[0056] It should be noted that the fruit pose real-time estimation device, electronic device, and computer-readable storage medium based on multi-frame information fusion provided in the above embodiments all belong to the same inventive concept as the fruit pose real-time estimation method based on multi-frame information fusion. For details of their specific implementation process, please refer to the embodiments of the fruit pose real-time estimation method based on multi-frame information fusion, which will not be repeated here.

[0057] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time fruit pose estimation method based on multi-frame information fusion, characterized in that, Includes the following steps: Simultaneously acquire image data observed by the sensor and the corresponding sensor pose; The image data is segmented into fruit semantic instances to obtain the fruit instances and depth point clouds of the current frame. Then, the fruit semantic instance segmentation results are correlated with the fruit map representation constructed based on the truncated symbolic range field, combined with the sensor pose, to determine the data association relationship between the current frame and historical frames. Based on the data association relationship and the fruit instances and depth point clouds of the current frame, the truncated symbolic range field map representation of the fruit is updated to achieve the fusion and update of observation information from multiple frames. The effective point cloud of the surface of each fruit instance is extracted from the updated map representation, and the pose information of the fruit is determined by filtering and principal component analysis. The size information of the fruit, including height, left and right width, and front and back width, is estimated. The real-time pose of the fruit is obtained by combining the pose information and the size information.

2. The real-time fruit pose estimation method based on multi-frame information fusion according to claim 1, characterized in that, The synchronous acquisition of image data observed by the sensor and the corresponding sensor pose includes: acquiring color and depth images synchronously collected by a depth camera mounted on a robotic arm, and calculating the camera sensor pose based on the joint angles of the robotic arm using a forward kinematics model; or, acquiring color and depth images synchronously collected by a depth camera mounted on a mobile platform, and running the RGBD-SLAM algorithm to calculate the camera sensor pose in real time.

3. The real-time fruit pose estimation method based on multi-frame information fusion according to claim 1 or 2, characterized in that, The step of performing fruit semantic instance segmentation on image data to obtain the fruit instance and depth point cloud of the current frame includes: performing fruit semantic instance segmentation on a single frame color image using a semantic segmentation model to obtain the pixel-level mask region of each fruit instance; and combining the depth image to convert the pixel-level mask region of each segmented fruit instance into the corresponding three-dimensional depth point cloud of the current single frame fruit instance.

4. The real-time fruit pose estimation method based on multi-frame information fusion according to claim 1, characterized in that, The process of combining sensor pose to segment fruit semantic instances and data association with the fruit map representation constructed based on the truncated symbol distance field to determine the data association relationship between the current frame and historical frames includes: transforming the depth point cloud of each fruit instance in the current frame to the world coordinate system according to the sensor pose; calculating the distance from the transformed point cloud to the truncated symbol distance field surface of each historical fruit instance in the fruit map representation; if the overlapping area of ​​the current fruit instance point cloud and a certain historical fruit instance satisfies the association condition, then it is determined that the two are the same fruit and a data association relationship is established; otherwise, the current fruit instance point cloud is initialized as a new fruit instance and a corresponding fruit map representation is established.

5. The real-time fruit pose estimation method based on multi-frame information fusion according to claim 1 or 4, characterized in that, The step of updating the truncated symbolic distance field map representation of the fruit based on data association and the fruit instance and depth point cloud of the current frame to achieve the fusion and update of observation information from multiple frames includes: for fruit instances with established data association, fusing the current fruit point cloud with the historical truncated symbolic distance field map, updating the symbolic distance values ​​and weights of voxels in the truncated symbolic distance field by weighted averaging, achieving the cumulative fusion of information from multiple frames, and improving the structural representation of the fruit.

6. The real-time fruit pose estimation method based on multi-frame information fusion according to claim 1, characterized in that, Principal component analysis is achieved by calculating the covariance matrix of the filtered point cloud and performing eigenvalue decomposition. The direction of the eigenvector corresponding to the largest eigenvalue is taken as the height axis of the fruit, and the directions of the other two mutually orthogonal eigenvectors are taken as the left and right width and front and back width axes of the fruit, respectively. A reference coordinate system is established by combining the centroid of the point cloud to determine the posture information of the fruit.

7. The real-time fruit pose estimation method based on multi-frame information fusion according to claim 1 or 6, characterized in that, The estimated dimensions of the fruit, including height, left and right width, and front and back width, are obtained by projecting the effective point cloud onto the fruit reference coordinate system determined by principal component analysis, and then using the coordinates of the point cloud in three axes as the initial height, initial left and right width, and initial front and back width of the fruit, respectively. If the reliability of the initial front and back width is lower than the preset requirement due to the limitation of the observation angle, the value of the initial left and right width is assigned to the initial front and back width as the final front and back width dimension.

8. A real-time fruit pose estimation device based on multi-frame information fusion, implemented using the real-time fruit pose estimation method based on multi-frame information fusion as described in any one of claims 1 to 7, characterized in that, include: Information input module, data association module, multi-frame fusion module, and pose generation module; The information input module is used to synchronously acquire image data observed by the sensor and the corresponding sensor pose; the data association module is used to perform fruit semantic instance segmentation on the image data to obtain the fruit instance and depth point cloud of the current frame, and then combine the sensor pose to perform data association between the fruit semantic instance segmentation result and the fruit map representation constructed based on the truncated symbolic distance field to determine the data association relationship between the current frame and the historical frames; the multi-frame fusion module is used to update the truncated symbolic distance field map representation of the fruit according to the data association relationship and the fruit instance and depth point cloud of the current frame, so as to realize the fusion and update of multi-frame observation information; the pose generation module is used to extract the effective point cloud of the surface of each fruit instance from the updated map representation, determine the pose information of the fruit through filtering and principal component analysis, and estimate the size information of the fruit including height, left and right width and front and back width, and obtain the real-time pose of the fruit by combining the pose information and size information.

9. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the real-time fruit pose estimation method based on multi-frame information fusion as described in any one of claims 1 to 7 when executing a computer program.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the real-time fruit pose estimation method based on multi-frame information fusion as described in any one of claims 1 to 7.

Images