Multi-mode collaborative apple detection and intelligent picking method

Abstract

The invention discloses a multi-mode cooperative apple detection and intelligent picking method. According to the method, a layered multi-sensor fusion composition and positioning system is constructed, and line planting interference is eliminated and positioning continuity under the satellite shielding condition is improved through dynamic switching of an orchard three-dimensional map construction and robot pose estimation system with differential GNSS-RTK and laser radar-inertial navigation IMU tight coupling in cooperation with point cloud remapping; constructing an apple detection visual system composed of a binocular camera and a depth camera; a YOLOv11 target detection model is improved, a mixed local attention mechanism MLCA is embedded in a backbone network, a neck adopts a grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure, and a head designs a group normalization sharing-scaling module, so that high-precision lightweight detection and occlusion grading are realized; the rigid and flexible mechanical arms perform cooperative work division picking according to shielding grading, the rigid arms pick low-shielding targets on the peripheries of fruit trees, and the flexible arms process high-shielding targets on the inner sides of the fruit trees. According to the method, the fruit identification precision, the positioning robustness and the picking efficiency in a complex environment are improved.

Description

Technical Field

[0001] This invention belongs to the field of agricultural robot technology, specifically a multimodal collaborative method for apple detection and intelligent harvesting. Background Technology

[0002] As the world's largest apple producer, China accounts for more than 50% of the world's total apple production. With the intelligent transformation of agriculture, harvesting robots are gradually replacing traditional manual harvesting. However, in complex open orchard environments, existing apple harvesting robots still face many technical challenges. Specifically, these challenges include: (1) In terms of visual perception, although the YOLO series models are widely used in the field of target detection, traditional YOLO algorithms are prone to missing detections in complex orchard scenarios such as small targets and overlapping occlusions, and their target recognition accuracy is insufficient under conditions of backlighting and occlusion; (2) Rigid robotic arms rely on high-precision positioning when avoiding obstacles, and the vibration generated during movement can cause the end effector to drift, and rigid arms are difficult to handle targets occluded on the inside of fruit trees; (3) GPS signals have low availability in densely planted orchards, and single-sensor positioning is easily affected by signal interference in complex orchard terrain, while the positioning error of relying solely on IMU is relatively high. Therefore, it is urgent to develop a lightweight and high-precision apple detection and intelligent harvesting method to improve the performance of harvesting robots in complex environments. Summary of the Invention

[0003] The purpose of this invention is to address the problems existing in the prior art by providing a multimodal collaborative apple detection and intelligent harvesting method. Through multi-sensor data fusion, an improved target detection model, and a harvesting strategy based on the degree of apple occlusion, this method achieves high-precision detection and non-destructive harvesting of apples in complex orchard environments, thereby improving harvesting efficiency and fruit integrity.

[0004] The technical solution to achieve the purpose of this invention is: a multimodal collaborative apple detection and intelligent harvesting method, the method comprising the following steps:

[0005] Step 1, Environmental Perception and Data Acquisition: Collect orchard environmental data based on the multi-sensor system carried by the picking robot;

[0006] Step 2, Multi-sensor data fusion mapping and localization: The data collected in Step 1 is fused and processed by a hierarchical multi-sensor data fusion mapping and localization system to realize the three-dimensional mapping of the orchard environment and the pose estimation of the picking robot.

[0007] Step 3, Apple Detection and Occlusion Assessment: Detect target apples using an improved target detection model and classify occlusion levels;

[0008] Step 4: Generate dynamic picking categories containing robotic arm type assignment instructions based on the occlusion level, so as to pick target apples with different occlusion levels by different types of robotic arms;

[0009] Step 5: Based on the results of steps 2 to 4, the picking robot uses a multimodal robotic arm group to collaboratively execute obstacle avoidance path planning and picking operations.

[0010] Furthermore, the multi-sensor system in step 1 includes a binocular camera, a lidar, an inertial navigation module (IMU), a GPS satellite system, a GNSS-RTK module, a data processing module, and a depth camera located at the end of the robotic arm of the harvesting robot, all mounted on the harvesting robot.

[0011] The binocular camera is used to acquire two-dimensional visual images of the orchard environment and transmit them to the data processing module to provide global information;

[0012] The lidar is used to collect three-dimensional point cloud data of the orchard environment and to perform data fusion and complementation with the binocular camera.

[0013] The depth camera is used to capture images of apples that are close to the end of the robotic arm and within the field of view of the depth camera;

[0014] The inertial navigation module (IMU) is used to measure the attitude and motion parameters of the harvesting robot in real time to ensure the attitude stability and trajectory control of the harvesting robot during the movement process.

[0015] The GPS satellite system and GNSS-RTK module are used to provide global positioning information for the picking robot, achieving centimeter-level positioning accuracy to ensure the precise navigation and positioning of the picking robot in the orchard;

[0016] The data processing module is used to identify all target apples based on the two-dimensional visual image; to extract target apples from the apple image; and to construct a three-dimensional point cloud map of the orchard environment based on the three-dimensional point cloud data.

[0017] Furthermore, in the hierarchical multi-sensor fusion mapping and localization system described in step 2, the lidar serves as the main sensor to realize the three-dimensional mapping of the orchard and the pose estimation of the picking robot, while the inertial navigation module (IMU) and GNSS-RTK module serve as auxiliary sensors to improve the mapping and localization accuracy and correct the lidar error; the hierarchical multi-sensor fusion mapping and localization system includes an input layer, a data preprocessing layer, a pose fusion layer, a reliability evaluation layer, and an output layer executed sequentially;

[0018] The input layer is used to simultaneously acquire 3D point cloud data from lidar, IMU data from inertial navigation, and GPS satellite positioning data.

[0019] The data preprocessing layer is used to implement:

[0020] For the 3D point cloud data of the input layer LiDAR, point cloud preprocessing operations are performed, including point cloud downsampling and outlier removal.

[0021] For the inertial navigation IMU data of the input layer, IMU pose estimation is performed based on the historical data of the inertial navigation IMU. Then it is determined whether the satellite signal missing time is less than a preset threshold. If so, IMU pose optimization is performed and the optimized IMU pose is sent to the credibility evaluation layer. Otherwise, IMU pose estimation is continued based on the historical data of the inertial navigation IMU.

[0022] For the GPS satellite positioning data of the input layer, it is determined whether the number of visible GPS satellites is higher than a preset threshold. If so, differential RTK pose calculation is performed on the GPS satellite positioning data and the result is sent to the pose fusion layer. Otherwise, GPS satellite positioning data is collected until the number of visible GPS satellites meets the preset threshold.

[0023] The pose fusion layer is used to establish a tightly coupled model of LiDAR-Inertial Navigation-GPS data. This model is used to fuse the LiDAR 3D point cloud data, IMU data and GPS satellite positioning data output from the data preprocessing layer to obtain the fused pose of the harvesting robot.

[0024] The credibility assessment layer is used to assess the credibility of the fused pose, the optimized IMU pose, and the pose calculated by differential RTK, so as to obtain the optimal pose of the picking robot.

[0025] The output layer is used to output the optimal pose of the harvesting robot.

[0026] Furthermore, the tightly coupled lidar-inertial navigation-GPS data model is as follows:

[0027] min T ∑ρ(||z lidar -T·ρ'|| 2 )+λ||q IMU -q RTK ||

[0028] In the formula, ρ is the Huber loss function for suppressing outlier interference, T is the transformation matrix, ρ' is the voxelization feature extraction operation of the lidar point cloud, and q IMU q RTK These are inertial navigation IMU data and GPS satellite positioning data, respectively. lidar The data is LiDAR point cloud data, where λ is the weighting coefficient and z is the weighting coefficient. lidar-T·ρ represents the environmental constraint residual between the LiDAR data after pose transformation and the fusion model.

[0029] Furthermore, the process of evaluating the reliability of the fused pose, the optimized IMU pose, and the pose calculated by differential RTK to obtain the optimal pose of the picking robot includes:

[0030] Step 2-1: The fused pose, the optimized IMU pose, and the pose calculated by differential RTK are respectively associated with the three sensors: LiDAR, IMU, and RTK. The following descriptions will use the three sensors: LiDAR, IMU, and RTK.

[0031] Establish a similarity vector S = [s] for quantifying the differences in poses among LiDAR, IMU, and RTK. 12 ,s 13 ,s 23 [,s 12 s 13 s 23 These represent the pairwise similarity between the three poses;

[0032] Step 2-2 involves assigning optimal weights to the three poses from Step 2-1 using both subjective and objective weighting methods. Specifically, this includes:

[0033] Step 2-2-1: Preset the subjective weights of LiDAR, IMU, and RTK respectively.

[0034] Step 2-2-2: Calculate the objective weights of LiDAR, IMU, and RTK in real time.

[0035]

[0036] in,

[0037] or

[0038] In the formula, i,j=1,2,3 correspond to the three sensors: lidar, IMU, and RTK, respectively. Let σ be the mean similarity between sensor i, sensor j and the other two sensors, respectively. i σ j Let be the observation variances of sensor i and sensor j, respectively; σ i ′、σ j These are the observation variance correction terms for sensor i and sensor j, respectively; is the objective weight of sensor i; k is the adjustment coefficient, which is adaptively adjusted according to the environment;

[0039] Step 2-2-3: Solve for the optimal weights using the divergence minimization algorithm;

[0040] By the following formula:

[0041]

[0042] Find the optimal weight And satisfy

[0043] Steps 2-3: Calculate the optimal pose P based on the optimal weights. final :

[0044]

[0045] In the formula, P Lidar P IMU P RTK These are the fused pose, the optimized IMU pose, and the pose calculated by differential RTK, respectively.

[0046] Steps 2-3: Determine whether the target difference ΔW satisfies the condition. If so, then P final The optimal pose of the picking robot is determined and transmitted to the output layer, and the process ends; otherwise, output correction is performed, and steps 2-4 are executed.

[0047] Steps 2-4: Determine whether RTK is invalid based on the DS evidence theory; if invalid, enforce execution. And recalculate the optimal pose P final Then return to steps 2-3.

[0048] Furthermore, the improved target detection model in step 3 is an improved YOLOv11 model, which specifically includes: based on the YOLOv11 network architecture, embedding a hybrid local attention mechanism, namely MLCA, into the backbone network; using a grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure to perform lightweight reconstruction of the neck; designing a group normalized shared-scale detection head, and achieving multi-scale occlusion feature fusion through a shared normalization layer.

[0049] Furthermore, the embedding of the hybrid local attention mechanism (MLCA) in the backbone network specifically involves introducing the MLCA mechanism into the backbone network and embedding it between the original C3k2 layers to form a stacked feature enhancement structure.

[0050] The lightweight reconstruction of the neck layer using grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure is specifically achieved by replacing the standard convolutional layer with the GSConv module and constructing the VoV-GSCSP cross-stage pathway to replace the traditional CSP module.

[0051] Furthermore, the design of the group-normalized shared-scale detection head includes three stages:

[0052] Group normalization layers (GN) are used on standard convolutions and depthwise separable convolutions in the detection branches;

[0053] The design incorporates a shared decoupling module. Feature maps of different scales are passed through a 1×1 convolutional module with group normalization layers to unify the feature map dimensions. Subsequently, the independent task branches in the original detection head of the YOLOv11 network architecture are merged into two parallel shared convolutional modules. One module is a concatenation of two 3×3 group normalized standard convolutions, and the other is a concatenation of two 3×3 group normalized depthwise separable convolutions and a 1×1 group normalized standard convolution. The output is then decoupled through two 1×1 convolutional layers, restoring the decoupling between the classification and localization tasks.

[0054] Multiple scale layers are integrated into the localization prediction branch to achieve scaling of features at different scales.

[0055] Furthermore, step 3 involves detecting the target apple and classifying its occlusion. The specific process includes:

[0056] Step 3-1: Collect several apple images from orchards and construct an apple image set;

[0057] Step 3-2: Expand the apple image set using data augmentation methods; the data augmentation methods include simulating different weather conditions, lighting changes, and Gaussian noise.

[0058] Step 3-3: Label the bounding box of the target apple and classify the occlusion level according to the occlusion degree threshold of the branches and leaves, and construct an apple training dataset containing occlusion level labels; wherein, the apples whose occlusion degree of branches and leaves is less than the occlusion degree threshold are labeled as the first category, and the apples whose occlusion degree of branches and leaves is greater than or equal to the occlusion degree threshold are labeled as the second category.

[0059] Steps 3-4: Input the apple training dataset containing occlusion level labels into the improved object detection model for end-to-end training to obtain the trained object detection model, which serves as the object detection and occlusion classification model.

[0060] Steps 3-5: Use the target detection and occlusion classification model to detect the target apple and classify its occlusion.

[0061] Further, step 4 involves generating dynamic harvesting categories based on occlusion levels, including robotic arm type assignment instructions, to enable the harvesting of target apples with different occlusion levels using different types of robotic arms. Specifically, this includes:

[0062] When the occlusion classification output in step 3 is the first category, that is, when the degree of occlusion of the apple by the branches and leaves is less than the occlusion threshold, the dynamic harvesting category is harvesting with a rigid robotic arm; when the occlusion classification output in step 3 is the second category, that is, when the degree of occlusion of the apple by the branches and leaves is greater than or equal to the occlusion threshold, the dynamic harvesting category is harvesting with a flexible robotic arm.

[0063] Compared with the prior art, the significant advantages of this invention are:

[0064] (1) Improved Model Detection Performance: By embedding a hybrid local attention mechanism (MLCA) into the YOLOv11 backbone network, and using a fusion of 7×7 block local average pooling and global average pooling, combined with one-dimensional convolutional compression and unpooling operations, the ability to extract key features of fruits in complex backgrounds is significantly enhanced, improving the mAP of the improved model. 50-95 The model achieved an accuracy of 86.46%, a precision of 91.2%, and a recall of 85.5%, representing improvements of 4.81, 2.6, and 1.7 percentage points respectively compared to the original model. By employing grouped sparse convolution (GSConv) in the neck region, which integrates depthwise separable convolution with a feature shuffling mechanism, combined with the dual-branch structure of the VoV-GSCSP cross-stage local network and multi-scale feature reuse of the group-normalized shared-scaled detection head, the model complexity and number of parameters were significantly reduced. The number of floating-point operations was reduced to 5.8 G FLOPs, and the number of parameters was reduced to 2.157452 M, representing reductions of 15.9% and 19.81% respectively. This effectively improved computational efficiency and lowered the hardware deployment threshold.

[0065] (2) High-precision and robust positioning in complex environments: A hierarchical multi-sensor fusion architecture consisting of an input layer, a preprocessing layer, a pose fusion layer, a credibility assessment layer, and an output layer is constructed. By utilizing the judgment of the number of visible GPS satellites in the preprocessing layer, the connection between differential GNSS-RTK centimeter-level positioning and the tightly coupled mode of laser-inertial navigation IMU is realized, ensuring the continuity and accuracy of positioning in scenarios where satellite signals are partially or completely blocked. Through the global remapping of laser point clouds in the pose fusion layer combined with the tight coupling of laser-inertial navigation IMU-GPS data, and the redistribution of subjective and objective weight deviations in the credibility assessment layer, the interference of orchard row planting structure on point cloud maps can be effectively reduced, thereby improving the credibility of fused data and the reliability of positioning results.

[0066] (3) Collaborative harvesting with occlusion grading and precise task division: By improving the YOLOv11 model to detect apples and assess the degree of occlusion by branches and leaves, the degree of occlusion is graded according to the grading threshold, and the rigid and flexible robotic arms are scheduled according to the occlusion grading results: the rigid arm picks the outer apples with low occlusion rate (<30%), and the flexible arm picks the inner apples or apples with high occlusion rate (≥30%) or severe occlusion / overlap, which significantly reduces the fruit miss rate in small targets and occlusion scenarios and improves the overall harvesting efficiency and success rate; in addition, the local three-dimensional shape information of the target is obtained by the depth camera at the end of the robotic arm to assist the robotic arm in finally approaching and grasping, and combined with the force sensor for real-time monitoring, to ensure precise grasping and non-destructive operation in the harvesting process and improve the fruit integrity rate.

[0067] (4) This invention utilizes a multi-sensor fusion architecture, comprising a vision system and a mapping and localization system consisting of a binocular camera, a depth camera, a lidar, an inertial navigation IMU, and a GNSS-RTK module, to achieve high-precision and robust perception and localization of target apples and their environment. By deploying an improved YOLOv11 detection model, it solves the problems of insufficient apple detection accuracy and computational redundancy in complex orchard environments. Through a hierarchical strategy based on occlusion degree and a rigid-flexible robotic arm collaborative control technology, it achieves adaptive and efficient collaborative harvesting of target apples on the inside and outside of fruit trees with varying occlusion conditions. The synergistic optimization of these three core components forms a complete apple harvesting solution with high-precision detection, robust localization, and efficient non-destructive execution capabilities in complex real orchard environments, significantly improving the accuracy, efficiency, and reliability of automated harvesting.

[0068] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating the harvesting process of the apple detection and harvesting method of the present invention.

[0070] Figure 2 This is a flowchart of the algorithm for hierarchical multi-sensor fusion mapping and localization of the present invention.

[0071] Figure 3 This is a block diagram of the improved YOLOv11 network structure of the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0073] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0074] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0075] In one embodiment, combined Figure 1 This paper provides a multimodal collaborative method for apple detection and intelligent harvesting, the method comprising the following steps:

[0076] Step 1, Environmental Perception and Data Acquisition: Collect orchard environmental data based on the multi-sensor system carried by the picking robot;

[0077] Step 2, Multi-sensor data fusion mapping and localization: The data collected in Step 1 is fused and processed by a hierarchical multi-sensor fusion mapping and localization system to realize the three-dimensional mapping of the orchard environment and the pose estimation of the picking robot.

[0078] Step 3, Apple Detection and Occlusion Assessment: Detect target apples using an improved target detection model and classify occlusion levels;

[0079] Step 4: Generate dynamic picking categories containing robotic arm type assignment instructions based on the occlusion level, so as to pick target apples with different occlusion levels by different types of robotic arms;

[0080] Step 5: Based on the results of steps 2 to 4, the picking robot uses a multimodal robotic arm group to collaboratively execute obstacle avoidance path planning and picking operations.

[0081] Furthermore, in one embodiment, the multi-sensor system in step 1 includes a binocular camera, a lidar, an inertial navigation module (IMU), a GPS satellite system, a GNSS-RTK module, a data processing module, and a depth camera located at the end of the robotic arm of the harvesting robot, all mounted on the harvesting robot.

[0082] The binocular camera is used to acquire two-dimensional visual images of the orchard environment and transmit them to the data processing module to provide global information;

[0083] The lidar is used to collect three-dimensional point cloud data of the orchard environment and to perform data fusion and complementation with the binocular camera.

[0084] The depth camera is used to capture images of apples that are close to the end of the robotic arm and within the field of view of the depth camera;

[0085] The inertial navigation module (IMU) is used to measure the attitude and motion parameters of the harvesting robot in real time to ensure the attitude stability and trajectory control of the harvesting robot during the movement process.

[0086] The GPS satellite system and GNSS-RTK module are used to provide global positioning information for the picking robot, achieving centimeter-level positioning accuracy to ensure the precise navigation and positioning of the picking robot in the orchard;

[0087] The data processing module is used to identify all target apples based on the two-dimensional visual image; to extract target apples from the apple image; and to construct a three-dimensional point cloud map of the orchard environment based on the three-dimensional point cloud data.

[0088] Furthermore, the robotic arm's end effector integrates a six-dimensional force sensor module to monitor the grasping force in real time. It issues an early warning when the normal force exceeds a threshold of 8 Newtons and immediately terminates the operation when the shearing force exceeds 3 Newtons.

[0089] Furthermore, in one embodiment, in the hierarchical multi-sensor fusion mapping and localization system described in step 2, the lidar serves as the main sensor to realize the 3D mapping of the orchard and the pose estimation of the picking robot, while the inertial navigation module (IMU) and GNSS-RTK module serve as auxiliary sensors to improve the mapping and localization accuracy and correct the lidar error; combined with Figure 2 The hierarchical multi-sensor fusion mapping and localization system includes an input layer, a data preprocessing layer, a pose fusion layer, a confidence evaluation layer, and an output layer, which are executed sequentially.

[0090] The input layer is used to simultaneously acquire 3D point cloud data from lidar, IMU data from inertial navigation, and GPS satellite positioning data.

[0091] The data preprocessing layer is used to implement:

[0092] For the 3D point cloud data of the input layer LiDAR, point cloud preprocessing operations are performed, including point cloud downsampling and outlier removal.

[0093] For the inertial navigation IMU data of the input layer, IMU pose estimation is performed based on the historical data of the inertial navigation IMU. Then it is determined whether the satellite signal missing time is less than a preset threshold. If so, IMU pose optimization is performed and the optimized IMU pose is sent to the credibility evaluation layer. Otherwise, IMU pose estimation is continued based on the historical data of the inertial navigation IMU.

[0094] For the GPS satellite positioning data of the input layer, it is determined whether the number of visible GPS satellites is higher than a preset threshold. If so, differential RTK pose calculation is performed on the GPS satellite positioning data and the result is sent to the pose fusion layer. Otherwise, GPS satellite positioning data is collected until the number of visible GPS satellites meets the preset threshold.

[0095] The pose fusion layer is used to establish a tightly coupled model of LiDAR-Inertial Navigation-GPS data. This model is used to fuse the LiDAR 3D point cloud data, IMU data and GPS satellite positioning data output from the data preprocessing layer and perform smoothing to obtain the fused pose of the harvesting robot.

[0096] The credibility assessment layer is used to assess the credibility of the fused pose, the optimized IMU pose, and the pose calculated by differential RTK, so as to obtain the optimal pose of the picking robot.

[0097] The output layer is used to output the optimal pose of the harvesting robot.

[0098] Here, after system initialization and self-test, the multi-sensor system's binocular camera begins acquiring global RGB images at a resolution of 1280×720; simultaneously, a 32-line lidar with a scanning frequency of 10Hz constructs a 3D point cloud map with a map density ≥160 points / m². 2 The GNSS-RTK module and GPS satellites provide the global pose of the harvesting robot. The GNSS-RTK module enables a positioning accuracy of ±1cm. When the number of visible GPS satellites is ≥6 (preferably), differential GNSS-RTK pose calculation is performed and the results are sent to the pose fusion layer. It is tightly coupled with the data from the lidar and IMU. The IMU's gyroscope has a zero-bias stability of ≤0.5° / h, which compensates for the robot's motion error in real time. When the distance between the robotic arm's end effector and the target apple is ≤30cm, the RealSense D435i depth camera is activated and acquires a local 3D point cloud. Subsequently, the data acquired by each sensor is timestamped and transmitted to the central processing unit.

[0099] Preferably, in some embodiments, the tightly coupled lidar-inertial navigation-GPS data model is as follows:

[0100] min T ∑ρ(||zlidar -T·ρ'|| 2 )+λ||q IMU -q RTK ||

[0101] In the formula, ρ is the Huber loss function for suppressing outlier interference, T is the transformation matrix, ρ' is the voxelization feature extraction operation of the lidar point cloud, and q IMU q RTK These are inertial navigation IMU data and GPS satellite positioning data, respectively. lidar The data is LiDAR point cloud data, where λ is the weighting coefficient and z is the weighting coefficient. lidar -T·ρ represents the environmental constraint residual between the LiDAR data after pose transformation and the fusion model. Preferably, the weighting coefficient λ is set to 0.75.

[0102] Preferably, in some embodiments, the process of evaluating the reliability of the fused pose, the optimized IMU pose, and the pose calculated by differential RTK to obtain the optimal pose of the picking robot specifically includes:

[0103] Step 2-1: The fused pose, the optimized IMU pose, and the pose calculated by differential RTK are respectively associated with the three sensors: LiDAR, IMU, and RTK. The following descriptions will use the three sensors: LiDAR, IMU, and RTK.

[0104] Establish a similarity vector S = [s] for quantifying the differences in poses among LiDAR, IMU, and RTK. 12 ,s 13 ,s 23 ], s 12 s 13 s 23 These represent the pairwise similarity between the three poses;

[0105] Step 2-2 involves assigning optimal weights to the three poses from Step 2-1 using both subjective and objective weighting methods. Specifically, this includes:

[0106] Step 2-2-1: Preset the subjective weights of LiDAR, IMU, and RTK respectively.

[0107] Step 2-2-2: Calculate the objective weights of LiDAR, IMU, and RTK in real time.

[0108]

[0109] in,

[0110] or

[0111] In the formula, i,j=1,2,3 correspond to the three sensors: lidar, IMU, and RTK, respectively. Let σ be the mean similarity between sensor i, sensor j and the other two sensors, respectively. i σ j Let σ′ be the observation variances of sensor i and sensor j, respectively; i σ′ j These are the observation variance correction terms for sensor i and sensor j, respectively; is the objective weight of sensor i; k is the adjustment coefficient, which is adaptively adjusted according to the environment;

[0112] Step 2-2-3: Solve for the optimal weights using the divergence minimization algorithm;

[0113] By the following formula:

[0114]

[0115] Find the optimal weight And satisfy

[0116] Steps 2-3: Calculate the optimal pose P based on the optimal weights. final :

[0117]

[0118] In the formula, P Lidar P IMU P RTK These are the fused pose, the optimized IMU pose, and the pose calculated by differential RTK, respectively.

[0119] Steps 2-3: Determine whether the target difference ΔW satisfies the condition. If so, then P final The optimal pose of the picking robot is determined and transmitted to the output layer, and the process ends; otherwise, output correction is performed, and steps 2-4 are executed.

[0120] Steps 2-4: Determine whether RTK is invalid based on the DS evidence theory; if invalid, enforce execution. And recalculate the optimal pose P final Then return to steps 2-3.

[0121] Furthermore, in one embodiment, the improved target detection model in step 3 is an improved YOLOv11 model, specifically including: based on the YOLOv11 network architecture, embedding a hybrid local attention mechanism, i.e., MLCA, into the backbone network; using a grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure to perform lightweight reconstruction of the neck; designing a group normalized shared-scale detection head, and achieving multi-scale occlusion feature fusion through a shared normalization layer.

[0122] Combination Figure 3 The embedding of a Hybrid Local Attention (MLCA) mechanism into the backbone network specifically involves introducing MLCA into the backbone network and embedding it between the original C3k2 layers to form a stacked feature enhancement structure. Specifically, a parallel dual-branch architecture is used to simultaneously enhance channel and spatial attention: the channel branch uses dynamic compression-activation operations to generate channel weights, focusing on key fruit feature responses; the spatial branch learns local texture features through deformable convolutional kernels, accurately capturing detailed information in areas obscured by branches and leaves. The attention weights output from the two branches are fused element-wise and multiplied with the original features to form an enhanced feature map with occlusion perception capabilities, significantly improving target discrimination capabilities in complex environments. Specifically, the input feature map is subjected to 7×7 block local average pooling (LAP) and global average pooling (GAP), then channel dimension feature transformation is performed using a one-dimensional convolution with a kernel size of 1×1×C. Then, feature weighted fusion is performed according to the following formula: the locally pooled features are fused with the original input features, and the globally pooled features are fused with the locally pooled features. Finally, depooling restores the original spatial dimension.

[0123]

[0124] Where F lap and F gap These are local pooling features and global pooling features, respectively. input and F out These represent the input and output features, respectively, and [;] indicates channel concatenation. This is for element-wise multiplication.

[0125] The output feature dimension of the 7×7 block local average pooling layer is H / 7×W / 7×49. After being compressed to H / 7×W / 7×C / 4 by 1×1×C convolution, it is concatenated with the 1×1×C feature output by global pooling and then restored to H×W×C by unpooling.

[0126] The method employs a grouped sparse convolution GSConv and a VoV-GSCSP cross-stage structure for lightweight neck reconstruction. Specifically, the GSConv module replaces the standard convolutional layer, achieving feature channel pruning and compression through a hybrid design of channel grouping sparsity and depthwise separable convolution, effectively reducing the number of parameters. A VoV-GSCSP cross-stage pathway replaces the traditional CSP module, reducing floating-point computation through dynamically adjusted cross-layer connections and bottleneck residual structure compression of the computation graph.

[0127] The design of the group-normalized shared-scaled detection head includes three stages, which improve the robustness of occluded target detection:

[0128] (1) A group normalization layer GN is used on the standard convolution and depthwise separable convolution of the detection branch; the input feature map is divided into several groups in the channel dimension according to the following formula, and the normalization operation is performed in each group to speed up the convergence speed of the model and avoid performance problems caused by batch size.

[0129]

[0130] Where GN is the number of channel groups, μ G and σ G Here are the mean and standard deviation for each group.

[0131] (2) Design a shared decoupling module. Feature maps of different scales are passed through a 1×1 convolutional module with group normalization layer to unify the feature map dimension. Then, the independent task branches in the original detection head of the YOLOv11 network architecture are merged into two parallel shared convolutional modules. One is two 3×3 group normalized standard convolutions connected in series, and the other is two 3×3 group normalized depth separable convolutions connected in series with a 1×1 group normalized standard convolution. Then, the output is decoupled through two 1×1 convolutional layers to restore the decoupling between the classification task and the localization task.

[0132] (3) Multiple scale scaling layers are integrated on the localization prediction branch to achieve scaling of features at different scales.

[0133] Furthermore, in one embodiment, step 3 involves detecting the target apple and classifying the occlusion level. The specific process includes:

[0134] Step 3-1: Collect several apple images from orchards and construct an apple image set;

[0135] Step 3-2: Expand the apple image set using data augmentation methods; the data augmentation methods include simulating different weather conditions, lighting changes, and Gaussian noise.

[0136] Step 3-3: Label the bounding box of the target apple and classify the occlusion level according to the occlusion degree threshold of the branches and leaves, and construct an apple training dataset containing occlusion level labels; wherein, the apples whose occlusion degree of branches and leaves is less than the occlusion degree threshold are labeled as the first category, and the apples whose occlusion degree of branches and leaves is greater than or equal to the occlusion degree threshold are labeled as the second category.

[0137] Preferably, the occlusion threshold is set to 30%.

[0138] Steps 3-4: Input the apple training dataset containing occlusion level labels into the improved object detection model for end-to-end training to obtain the trained object detection model, which serves as the object detection and occlusion classification model.

[0139] Steps 3-5: Use the target detection and occlusion classification model to detect the target apple and classify its occlusion.

[0140] Furthermore, in one embodiment, step 4, generating a dynamic picking category containing robotic arm type assignment instructions based on the occlusion level, to enable picking target apples with different occlusion levels using different types of robotic arms, specifically includes:

[0141] When the occlusion classification output in step 3 is the first category, that is, when the degree of occlusion of the apple by the branches and leaves is less than the occlusion threshold, the dynamic harvesting category is harvesting with a rigid robotic arm; when the occlusion classification output in step 3 is the second category, that is, when the degree of occlusion of the apple by the branches and leaves is greater than or equal to the occlusion threshold, the dynamic harvesting category is harvesting with a flexible robotic arm.

[0142] In one embodiment, a multimodal collaborative apple detection and intelligent harvesting device is provided, the device comprising sequentially executing:

[0143] The first module is used to realize environmental perception and data acquisition: the orchard environmental data is collected based on the multi-sensor system carried by the picking robot;

[0144] The second module is used to realize multi-sensor data fusion mapping and localization: the data collected by the first module is fused and processed through a hierarchical multi-sensor fusion mapping and localization system to realize three-dimensional mapping of the orchard environment and pose estimation of the picking robot.

[0145] The third module is used to implement apple detection and occlusion assessment: it uses an improved target detection model to detect target apples and classify occlusion.

[0146] The fourth module is used to generate dynamic picking categories containing robotic arm type assignment instructions based on the occlusion level, so as to pick target apples with different occlusion levels by different types of robotic arms;

[0147] The fifth module is used to: based on the results of the second to fourth modules, control the picking robot to perform obstacle avoidance path planning and picking operations in a coordinated manner through a multimodal robotic arm group.

[0148] Specific limitations regarding the multimodal collaborative apple detection and intelligent harvesting system can be found in the limitations of the multimodal collaborative apple detection and intelligent harvesting method described above, and will not be repeated here. Each module in the aforementioned multimodal collaborative apple detection and intelligent harvesting system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0149] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:

[0150] Step 1, Environmental Perception and Data Acquisition: Collect orchard environmental data based on the multi-sensor system carried by the picking robot;

[0151] Step 2, Multi-sensor data fusion mapping and localization: The data collected in Step 1 is fused and processed by a hierarchical multi-sensor fusion mapping and localization system to realize the three-dimensional mapping of the orchard environment and the pose estimation of the picking robot.

[0152] Step 3, Apple Detection and Occlusion Assessment: Detect target apples using an improved target detection model and classify occlusion levels;

[0153] Step 4: Generate dynamic picking categories containing robotic arm type assignment instructions based on the occlusion level, so as to pick target apples with different occlusion levels by different types of robotic arms;

[0154] Step 5: Based on the results of steps 2 to 4, the picking robot uses a multimodal robotic arm group to collaboratively execute obstacle avoidance path planning and picking operations.

[0155] For specific limitations on each step, please refer to the limitations on the multimodal collaborative apple detection and intelligent harvesting method mentioned above, which will not be repeated here.

[0156] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0157] Step 1, Environmental Perception and Data Acquisition: Collect orchard environmental data based on the multi-sensor system carried by the picking robot;

[0158] Step 2, Multi-sensor data fusion mapping and localization: The data collected in Step 1 is fused and processed by a hierarchical multi-sensor fusion mapping and localization system to realize the three-dimensional mapping of the orchard environment and the pose estimation of the picking robot.

[0159] Step 3, Apple Detection and Occlusion Assessment: Detect target apples using an improved target detection model and classify occlusion levels;

[0160] Step 4: Generate dynamic picking categories containing robotic arm type assignment instructions based on the occlusion level, so as to pick target apples with different occlusion levels by different types of robotic arms;

[0161] Step 5: Based on the results of steps 2 to 4, the picking robot uses a multimodal robotic arm group to collaboratively execute obstacle avoidance path planning and picking operations.

[0162] For specific limitations on each step, please refer to the limitations on the multimodal collaborative apple detection and intelligent harvesting method mentioned above, which will not be repeated here.

[0163] As a specific example, the invention will be further verified and illustrated in one embodiment.

[0164] A field test was conducted at an apple orchard with a standard row spacing of 4 meters and a tree spacing of 2 meters. The test platform used an AGV chassis with a maximum travel speed of 1.2 m / s, equipped with an NVIDIA Jetson AGX Orin computing unit. One hundred fruit trees were randomly selected as a sample. Of the 862 marked apples, 63% had an occlusion rate of less than 30%, while 37% had an occlusion rate of 30% or higher.

[0165] The mobile platform (harvesting robot) first initializes and performs a self-check. It then moves forward according to the 3D point cloud map of the orchard obtained by the hierarchical multi-sensor fusion mapping and positioning system. During the movement, the binocular camera, LiDAR, IMU inertial navigation module, GPS satellite system, GNSS-RTK module, and depth camera at the end of the robotic arm simultaneously collect orchard environmental data and feed it back to the computing unit to construct a 3D map of the orchard environment and estimate the robot's position and pose in 3D space. At the same time, it locates the harvesting area containing apples. The deployed YOLOv11 improved model uniformly scales the input image data to a resolution of 640×640 and outputs the apple position coordinates and occlusion level in real time. The rigid and flexible robotic arms obtain the division of harvesting instructions based on the occlusion classification results. The rigid robotic arm harvests apples on the periphery of the fruit trees with a branch and leaf occlusion rate of less than 30%, while the flexible robotic arm harvests apples on the inner side of the fruit trees with an occlusion rate of not less than 30%. This operation is repeated until all targets within the field of view are harvested, thus completing the harvesting of the entire orchard.

[0166] Test results show that the target detection model mAP50-95 The accuracy rate reached 86.46%, an improvement of 4.81 percentage points compared to the baseline model; the number of model parameters was successfully compressed to 2.157452M, achieving a parameter reduction of 19.81%; the continuity of the positioning system under satellite signal obstruction scenarios reached 99.3%, an improvement of 22.7 percentage points compared to traditional methods; and the overall harvesting success rate increased to 94.1%. Three repeated tests verified the comprehensive effectiveness of this solution in improving detection accuracy, enhancing positioning robustness, and optimizing harvesting efficiency.

[0167] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A multimodal collaborative method for apple detection and intelligent harvesting, characterized in that, The method includes the following steps: Step 1, Environmental Perception and Data Acquisition: Collect orchard environmental data based on the multi-sensor system carried by the picking robot; Step 2, Multi-sensor data fusion mapping and localization: The data collected in Step 1 is fused and processed by a hierarchical multi-sensor data fusion mapping and localization system to realize the three-dimensional mapping of the orchard environment and the pose estimation of the picking robot. Step 3, Apple Detection and Occlusion Assessment: Detect target apples using an improved target detection model and classify occlusion levels; Step 4: Generate dynamic picking categories containing robotic arm type assignment instructions based on the occlusion level, so as to pick target apples with different occlusion levels by different types of robotic arms; Step 5: Based on the results of steps 2 to 4, the picking robot uses a multimodal robotic arm group to collaboratively execute obstacle avoidance path planning and picking operations.

2. The multimodal collaborative apple detection and intelligent harvesting method according to claim 1, characterized in that, The multi-sensor system in step 1 includes a binocular camera, lidar, inertial navigation module (IMU), GPS satellite system, GNSS-RTK module, data processing module, and a depth camera located at the end of the robotic arm of the harvesting robot, all mounted on the harvesting robot. The binocular camera is used to acquire two-dimensional visual images of the orchard environment and transmit them to the data processing module to provide global information; The lidar is used to collect three-dimensional point cloud data of the orchard environment and to perform data fusion and complementation with the binocular camera. The depth camera is used to capture images of apples that are close to the end of the robotic arm and within the field of view of the depth camera; The inertial navigation module (IMU) is used to measure the attitude and motion parameters of the harvesting robot in real time to ensure the attitude stability and trajectory control of the harvesting robot during the movement process. The GPS satellite system and GNSS-RTK module are used to provide global positioning information for the picking robot, achieving centimeter-level positioning accuracy to ensure the precise navigation and positioning of the picking robot in the orchard; The data processing module is used to identify all target apples based on the two-dimensional visual image; to extract target apples from the apple image; and to construct a three-dimensional point cloud map of the orchard environment based on the three-dimensional point cloud data.

3. The multimodal collaborative apple detection and intelligent harvesting method according to claim 2, characterized in that, In the hierarchical multi-sensor fusion mapping and localization system described in step 2, the lidar serves as the main sensor to realize the 3D mapping of the orchard and the pose estimation of the picking robot. The inertial navigation module (IMU) and the GNSS-RTK module serve as auxiliary sensors to improve the mapping and localization accuracy and correct the lidar error. The hierarchical multi-sensor fusion mapping and localization system includes an input layer, a data preprocessing layer, a pose fusion layer, a confidence evaluation layer, and an output layer executed sequentially. The input layer is used to simultaneously acquire 3D point cloud data from lidar, IMU data from inertial navigation, and GPS satellite positioning data. The data preprocessing layer is used to implement: For the 3D point cloud data of the input layer LiDAR, point cloud preprocessing operations are performed, including point cloud downsampling and outlier removal. For the inertial navigation IMU data of the input layer, IMU pose estimation is performed based on the historical data of the inertial navigation IMU. Then it is determined whether the satellite signal missing time is less than a preset threshold. If so, IMU pose optimization is performed and the optimized IMU pose is sent to the credibility evaluation layer. Otherwise, IMU pose estimation is continued based on the historical data of the inertial navigation IMU. For the GPS satellite positioning data of the input layer, it is determined whether the number of visible GPS satellites is higher than a preset threshold. If so, differential RTK pose calculation is performed on the GPS satellite positioning data and the result is sent to the pose fusion layer. Otherwise, GPS satellite positioning data is collected until the number of visible GPS satellites meets the preset threshold. The pose fusion layer is used to establish a tightly coupled model of LiDAR-Inertial Navigation-GPS data. This model is used to fuse the LiDAR 3D point cloud data, IMU data and GPS satellite positioning data output from the data preprocessing layer to obtain the fused pose of the harvesting robot. The credibility assessment layer is used to assess the credibility of the fused pose, the optimized IMU pose, and the pose calculated by differential RTK, so as to obtain the optimal pose of the picking robot. The output layer is used to output the optimal pose of the harvesting robot.

4. The multimodal collaborative apple detection and intelligent harvesting method according to claim 3, characterized in that, The tightly coupled data model of lidar-inertial navigation-GPS is as follows: minutes T ∑ρ(||z lidar -T·p'|| 2 )+λ||q IMU -q RTK || In the formula, ρ is the Huber loss function for suppressing outlier interference, T is the transformation matrix, ρ' is the voxelization feature extraction operation of the lidar point cloud, and q IMU q RTK These are inertial navigation IMU data and GPS satellite positioning data, respectively. lidar The data is LiDAR point cloud data, where λ is the weighting coefficient and z is the weighting coefficient. lidar -T·ρ represents the environmental constraint residual between the LiDAR data after pose transformation and the fusion model.

5. The multimodal collaborative apple detection and intelligent harvesting method according to claim 1, characterized in that, The process of evaluating the reliability of the fused pose, the optimized IMU pose, and the pose calculated by differential RTK to obtain the optimal pose for the picking robot includes: Step 2-1: The fused pose, the optimized IMU pose, and the pose calculated by differential RTK are respectively associated with the three sensors: LiDAR, IMU, and RTK. The following descriptions will use the three sensors: LiDAR, IMU, and RTK. Establish a similarity vector S = [s] for quantifying the differences in poses among LiDAR, IMU, and RTK. 12 ,s 13 ,s 23 ], s 12 s 13 s 23 These represent the pairwise similarity between the three poses; Step 2-2 involves assigning optimal weights to the three poses from Step 2-1 using both subjective and objective weighting methods. Specifically, this includes: Step 2-2-1: Preset the subjective weights of LiDAR, IMU, and RTK respectively. Step 2-2-2: Calculate the objective weights of LiDAR, IMU, and RTK in real time. in, or In the formula, i,j=1,2,3 correspond to the three sensors: lidar, IMU, and RTK, respectively. Let σ be the mean similarity between sensor i, sensor j and the other two sensors, respectively. i σ j Let σ′ be the observation variances of sensor i and sensor j, respectively; i σ′ j These are the observation variance correction terms for sensor i and sensor j, respectively; is the objective weight of sensor i; k is the adjustment coefficient, which is adaptively adjusted according to the environment; Step 2-2-3: Solve for the optimal weights using the divergence minimization algorithm; By the following formula: Find the optimal weight And satisfy Steps 2-3: Calculate the optimal pose P based on the optimal weights. final : In the formula, P Lidar P IMU P RTK These are the fused pose, the optimized IMU pose, and the pose calculated by differential RTK, respectively. Steps 2-3: Determine if the target difference ΔW satisfies the condition. If so, then P final The optimal pose of the picking robot is determined and transmitted to the output layer, and the process ends; otherwise, output correction is performed, and steps 2-4 are executed. Steps 2-4: Determine whether RTK is invalid based on the DS evidence theory; if invalid, enforce execution. And recalculate the optimal pose P final Then return to steps 2-3.

6. The multimodal collaborative apple detection and intelligent harvesting method according to claim 1, characterized in that, The improved target detection model in step 3 is an improved YOLOv11 model, which specifically includes: based on the YOLOv11 network architecture, embedding a hybrid local attention mechanism (MLCA) in the backbone network; using a grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure to perform lightweight reconstruction of the neck; and designing a group normalized shared-scale detection head to achieve multi-scale occlusion feature fusion through a shared normalization layer.

7. The multimodal collaborative apple detection and intelligent harvesting method according to claim 6, characterized in that, The embedding of the hybrid local attention mechanism (MLCA) into the backbone network specifically involves introducing the MLCA mechanism into the backbone network and embedding it between the original C3k2 layers to form a stacked feature enhancement structure. The lightweight reconstruction of the neck layer using grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure is specifically achieved by replacing the standard convolutional layer with the GSConv module and constructing the VoV-GSCSP cross-stage pathway to replace the traditional CSP module.

8. The multimodal collaborative apple detection and intelligent harvesting method according to claim 6, characterized in that, The design of the group-normalized shared-scaled detection head includes three stages: Group normalization layers (GN) are used on standard convolutions and depthwise separable convolutions in the detection branches; The design incorporates a shared decoupling module. Feature maps of different scales are passed through a 1×1 convolutional module with group normalization layers to unify the feature map dimensions. Subsequently, the independent task branches in the original detection head of the YOLOv11 network architecture are merged into two parallel shared convolutional modules. One module is a concatenation of two 3×3 group normalized standard convolutions, and the other is a concatenation of two 3×3 group normalized depthwise separable convolutions and a 1×1 group normalized standard convolution. The output is then decoupled through two 1×1 convolutional layers, restoring the decoupling between the classification and localization tasks. Multiple scale layers are integrated into the localization prediction branch to achieve scaling of features at different scales.

9. The multimodal collaborative apple detection and intelligent harvesting method according to claim 1, characterized in that, Step 3 involves detecting the target apple and classifying its occlusion. The specific process includes: Step 3-1: Collect several apple images from orchards and construct an apple image set; Step 3-2: Expand the apple image set using data augmentation methods; the data augmentation methods include simulating different weather conditions, lighting changes, and Gaussian noise. Step 3-3: Label the bounding box of the target apple and classify the occlusion level according to the occlusion degree threshold of the branches and leaves, and construct an apple training dataset containing occlusion level labels; wherein, the apples whose occlusion degree of branches and leaves is less than the occlusion degree threshold are labeled as the first category, and the apples whose occlusion degree of branches and leaves is greater than or equal to the occlusion degree threshold are labeled as the second category. Steps 3-4: Input the apple training dataset containing occlusion level labels into the improved object detection model for end-to-end training to obtain the trained object detection model, which serves as the object detection and occlusion classification model. Steps 3-5: Use the target detection and occlusion classification model to detect the target apple and classify its occlusion.

10. The multimodal collaborative apple detection and intelligent harvesting method according to claim 9, characterized in that, Step 4: Generate dynamic harvesting categories based on occlusion levels, including robotic arm type assignment instructions, to enable the harvesting of target apples with different occlusion levels using different types of robotic arms. Specifically, this includes: When the occlusion classification output in step 3 is the first category, that is, when the degree of occlusion of the apple by the branches and leaves is less than the occlusion threshold, the dynamic harvesting category is harvesting with a rigid robotic arm; when the occlusion classification output in step 3 is the second category, that is, when the degree of occlusion of the apple by the branches and leaves is greater than or equal to the occlusion threshold, the dynamic harvesting category is harvesting with a flexible robotic arm.

Images