Tea bud picking obstacle avoidance path planning method and system based on multi-modal information fusion diffusion model

Abstract

The invention discloses a tea bud picking obstacle avoidance path planning method and system based on a multi-modal information fusion diffusion model. A tea tree canopy image is acquired through a light field camera, a dense three-dimensional point cloud is constructed, tea bud target detection and positioning are realized in combination with an improved YOLOV8 and a PointNet + + network, a path generation method based on a U-net architecture condition denoising diffusion model is provided, path planning is modeled as a condition track generation problem, and the problem is solved. According to the method, multi-modal condition information input and a target guiding steering function are combined, multi-step denoising iteration is executed, a continuous, smooth, collision-free, dynamic and feasible mechanical arm picking path is directly generated, the sawtooth-shaped defect and post-optimization dependence of a path generated through traditional sampling algorithms such as RRT are avoided, and the path quality and execution stability are improved. The system comprises a light field acquisition module, a three-dimensional reconstruction module, a visual positioning module, a path planning module, an execution control module and the like, realizes accurate obstacle-avoiding picking of tea buds, is suitable for path planning of picking operation of tea leaves and other fruits and vegetables, and provides technical support for agricultural picking and harvesting.

Description

Technical Field

[0001] This invention relates to the field of path planning technology for tea-picking robotic arms, specifically to a method and system for obstacle avoidance path planning in tea bud picking using a multimodal information fusion and diffusion model. Background Technology

[0002] Premium teas have extremely high standards for picking tender buds, and their quality and market price directly depend on the integrity and precision of the picked buds. However, tea picking is highly seasonal and requires a large amount of labor. To address the cost and efficiency issues of manual picking, various tea bud picking robot systems have emerged in recent years. However, the dense and irregularly distributed branches of tea trees create a complex three-dimensional obstacle space, increasing the difficulty of obstacle avoidance and path planning for the robotic arm. Among these, the path planning algorithm determines the quality of precise picking, and key aspects include: high-precision target recognition and positioning, strong spatial obstacle avoidance capabilities, and smooth and continuous paths to prevent damage to tender buds.

[0003] Currently, traditional methods such as A*, RRT, RRT-connect, and CTB-RRT*, while widely used in agricultural robot path planning, suffer from several drawbacks when faced with complex obstacles in tea tree canopies, dense foliage, limited space, and stringent accuracy requirements: low computational efficiency, poor path smoothness requiring subsequent smoothing, and increased system latency. These algorithms struggle to effectively meet the demands of real-time path planning. Optimization algorithms, such as gradient descent, ant colony optimization, and genetic algorithms, are slow to converge, prone to getting trapped in local optima, and lack global exploration capabilities. Existing technologies still rely on traditional planning or optimization algorithms, lacking end-to-end path planning solutions based on generative models.

[0004] Furthermore, the current tea bud picking path planning separates the perception and planning parts, and uses only a single modality of 2D images or 3D point clouds, failing to integrate the information of both, which affects the accuracy of target recognition and spatial location estimation; resulting in path generation based only on single input reasoning, lacking multimodal collaborative constraints, and making it difficult to generate executable paths that meet multiple constraints such as obstacle avoidance and path smoothing.

[0005] Therefore, there is an urgent need to address the problems of poor real-time performance, poor smoothness, and weak executability of path planning in the complex environment of tea tree canopy, and to improve the intelligence and engineering usability of tea bud picking robots. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a path planning method and system for obstacle avoidance in a tea bud picking robotic arm based on a diffusion model using multimodal information fusion. This aims to solve the problems of difficult path planning, poor smoothness, and insufficient real-time performance caused by the dense distribution of tea buds and the complex branch and leaf structure in the tea canopy, enabling the picking robot to accurately, collision-free, and efficiently pick tender buds in a natural tea garden environment. This invention is achieved through the following technical means, including:

[0007] (1) Data acquisition and light field three-dimensional point cloud reconstruction: use a light field camera to collect multi-view light field images of the tea tree canopy to obtain the original multi-dimensional image data of the canopy; generate a dense three-dimensional point cloud model of the tea tree canopy through the light field three-dimensional reconstruction algorithm to obtain the distribution information of tea buds and branches in space, and provide three-dimensional environmental perception data for subsequent target detection and path planning.

[0008] (2) Two-dimensional tea bud target detection: Improve the YOLOV8 deep learning target detection network, use the embedded multi-scale attention module EMA to perform tea bud target detection on the acquired image, extract the pixel coordinates of the tea bud target in the two-dimensional image, and combine the light field three-dimensional reconstruction results to realize the projection association of the target position in three-dimensional space.

[0009] (3) Three-dimensional tea bud segmentation and localization: The improved PointNet++ network is adopted, combined with the density-aware farthest point sampling method DA-FPS and the spatial attention module SAM to further extract the three-dimensional coordinate information of the tea bud target, so as to achieve high-precision spatial localization of the tea bud target. By improving the PointNet++ three-dimensional segmentation network in the three-dimensional point cloud of the tea tree canopy, the tea bud target point cloud segmentation and localization are realized.

[0010] (4) Obstacle environment construction: Segment and cluster non-target point clouds such as branches and leaves in the tea tree canopy to generate an environment map, which constitutes the multimodal condition input required for path planning.

[0011] (5) Conditional encoding generation: Convolutional neural network (CNN) is used to jointly encode the target position of tea buds and the obstacle environment map to generate scene condition vectors. By embedding time step encoding between convolutional layers, the joint input of conditional information and diffusion model time step information is realized, thereby improving the stability and controllability of path generation.

[0012] (6) Conditional denoising diffusion probability model path generation: Based on the U-net structure, the denoising diffusion probability model DDPM is used to input the scene condition encoding into the denoising diffusion probability model constructed based on the U-net structure. The forward diffusion and reverse denoising process of the model is used to introduce target condition constraints to modify the quality of the generated path. The initial noise path is iteratively optimized step by step to generate a collision-free, smooth, and executable robotic arm picking path from the initial position of the robotic arm to the tea bud target.

[0013] (7) The picking robot arm control, the output of the sensing module is directly input to the planning module, the planning module sends the generated path to the robot arm control platform in real time, and converts the generated path into the robot arm Cartesian trajectory; through inverse kinematics solution and stepper motor control, the parallel robot arm end effector completes the precise picking operation of the tea bud target.

[0014] This invention features innovative designs and methodological improvements for each module. Furthermore, the proposed three-dimensional light field reconstruction method addresses the common problems in traditional three-dimensional reconstruction methods for tea tree canopy light field, such as low reconstruction density, blurred boundaries, and even partial area loss due to repetitive leaf and branch textures, small target size of tea buds, and limited parallax baseline. This invention proposes a three-dimensional reconstruction method for tea tree canopy based on light field ray consistency constraints, specifically including:

[0015] (1) Light field camera data acquisition scheme

[0016] A light field camera is mounted on a movable track platform to acquire multi-view light field images of the tea tree canopy through surround shooting. This improves the observability and accuracy of 3D reconstruction.

[0017] (2) Ray manifold constraint

[0018] By constructing ray consistency constraints for multiple sub-views in the light field image, and utilizing the principle of projection consistency of the same object point in different sub-aperture views in the light field data, a ray manifold model is formed, which provides physical constraints for matching points in the camera pose solving stage, thereby enhancing reconstruction stability.

[0019] (3) Edge-preserving stereo matching method

[0020] To address the issue of tea bud boundaries easily overlapping with the background, an edge-preserving stereo matching algorithm is adopted. Edge-guided filtering is introduced during the cost aggregation process to preserve gradient changes at the edges and avoid the loss of tea bud boundary information caused by smoothing.

[0021] (4) Light field binding adjustment

[0022] After completing the depth calculation for each frame's light field sub-view, a light field bundling adjustment method was used to jointly optimize the camera extrinsic parameters and 3D structural points across multiple frames. This optimized the spatial consistency and density of the overall point cloud, resulting in a high-quality, spatially continuous 3D point cloud model of the tea tree canopy.

[0023] Furthermore, the improved two-dimensional tea bud target detection network method addresses the issue that tea buds are small in size, have colors similar to the background, and vary in shape. Directly using the standard YOLOv8 network for detection easily leads to missed detections and false detections. This invention proposes the following improvement: To address the problems of small size, dense distribution, and complex backgrounds of tea buds in tea bud harvesting tasks, the improved two-dimensional tea bud target detection network method introduces an efficient multi-scale attention module (EMA) to enhance the model's ability to detect small targets and complex spatial relationships. This improves the saliency representation of tea bud targets in low-resolution feature maps and solves the problem of lost target information during pooling and stride operations in traditional convolution.

[0024] To fully leverage the advantages of the EMA module, this invention embeds it into the ELAN module within the YOLOv8 network, constructing an ELAN-EMA composite module to improve the accuracy and robustness of small target detection in tea buds.

[0025] Furthermore, the improved 3D tea bud segmentation network method addresses the characteristics of tea canopy point clouds, such as uneven density-sparse distribution, a small number of voxels in the tea bud point cloud, and significant density differences compared to the background leaf point cloud. The original PointNet++ structure suffers from sampling point loss and insufficient spatial feature discrimination when segmenting tea bud targets. Therefore, this invention proposes the following improved method:

[0026] (1) DA-FPS density sensing sampling

[0027] In the sampling module of PointNet++, the Density Aware-Farthest Point Sampling method is introduced. By calculating the sampling point cloud density, the sampling probability distribution of the FPS sampling strategy is adjusted, which increases the sampling probability of sparse regions (tea bud point cloud), ensures the integrity of feature points in the target region, and avoids the loss of target information after sampling.

[0028] (2) SAM Spatial Attention Module

[0029] In the feature aggregation stage, the Spatial Attention Module (SAM) is integrated. By calculating the spatial similarity weight between a point and its neighboring points, points related to the target features are highlighted, while irrelevant background points are suppressed. This improves the model's ability to distinguish between tea bud point clouds and leaf background point clouds, and enhances the target consistency of the segmentation results.

[0030] Furthermore, the obstacle environment condition encoder design method

[0031] This invention employs a CNN conditional encoder to jointly encode the target location and obstacle map to generate a scene conditional vector, which is used for conditional guidance in the DDPM path generation process, including:

[0032] (1) Multi-layer convolutional structure

[0033] A three-layer one-dimensional convolutional structure is adopted to extract features along the path length dimension. The convolutional kernel size is 3 and the stride is 1. Each convolutional layer is followed by a ResBlock module to enhance the feature representation capability and avoid gradient vanishing.

[0034] (2) Residual connectivity and nonlinear activation

[0035] After each convolutional module, a residual connection structure is used to add the input and output, and the Mish activation function is used to increase the model's nonlinear fitting ability, thereby improving the diversity and stability of the conditional vectors.

[0036] (3) Time step encoding embedding

[0037] By using a single fully connected linear layer, the time step encoding in the DDPM diffusion process is mapped to a time embedding vector with the same dimension as the conditional vector, and then added between convolutional layers to achieve joint conditional-temporal input, thereby improving the generation stability of the denoising process.

[0038] Furthermore, addressing the issues of dense tea canopy foliage, long computation time, and uneven paths in traditional path planning methods, this invention proposes a tea bud harvesting path generation method based on the Conditional Denoising Diffusion Probability Model (DDPM). The implementation process specifically includes:

[0039] (1) Path generation task modeling

[0040] The harvesting path planning problem is modeled as a conditional generation problem, namely, generating a collision-free, smooth, and executable path from the initial position of the robotic arm to the tea bud target, given the target location and obstacle environment. Compared with traditional sampling search or optimization methods, the diffusion model can directly learn the path distribution, achieving diverse and constrained path generation.

[0041] (2) Design of the structure of the conditional DDPM model

[0042] The DDPM path generation model used in this invention adopts a conditional denoising diffusion model architecture based on the Unet structure, mainly including: an input layer: the input is a Gaussian noise-initialized path sequence, and the target position and obstacle condition vectors extracted by the CNN conditional encoder; a downsampling module, which progressively extracts multi-scale features of the path sequence; an intermediate bottleneck module, which fuses global path information and condition features; an upsampling module, which progressively restores the path length; and an output layer: the output is a denoised path sequence tensor.

[0043] (3) Combined input of condition and time step

[0044] By fusing the diffusion time step embedding vector and the conditional encoding vector in each layer of the network, the model can jointly perceive the dynamic state of the diffusion process and the target-obstacle conditions, thereby improving the conditional controllability and stability of the generated path.

[0045] (4) Target-guided turning function

[0046] To ensure that the generated path has good target reachability, obstacle avoidance capability and smooth motion, this invention introduces a target guidance steering function in the denoising step, which guides the end point of the path to the target coordinates of the tea bud through the target attraction term.

[0047] Furthermore, the harvesting robotic arm control method proposed in this invention achieves seamless integration of path planning and execution control, specifically including:

[0048] The path point sequence generated online by the conditional DDPM model is directly mapped to the Cartesian space trajectory of the end effector of the parallel robotic arm. The path point coordinates are analyzed in real time by the robotic arm motion control module. Combined with the robotic arm dynamics and inverse kinematics model, the angle command of the robotic arm link joint corresponding to each path point is calculated, and the control command is dynamically sent to the motor drive unit during system operation.

[0049] Furthermore, this invention systematically integrates three-dimensional light field reconstruction, two-dimensional target detection, three-dimensional point cloud segmentation, conditional encoding generation, DDPM path planning, robotic arm motion control, and end-effector execution modules, constructing an end-to-end closed-loop architecture of perception-planning-execution. This invention forms a closed-loop intelligent system from visual perception to path planning and robotic arm execution, avoiding coordinate system errors and path infeasibility problems caused by the separation of perception and planning modules in traditional methods. Path generation requires no post-processing, and the output path can be directly sent to the robotic arm control system for execution, reducing system response time. It has high environmental adaptability and scalability, is suitable for harvesting tasks of different tea tree varieties and different canopy morphologies, and can also be extended to other fruit and vegetable harvesting applications.

[0050] The present invention provides a tea bud picking obstacle avoidance path planning system based on a multimodal information fusion and diffusion model, comprising:

[0051] The light field acquisition module is used to acquire multi-view images and light field data of the tea tree canopy through a light field camera, and output them to an industrial computer through the GigE Vision interface of the Gigabit Ethernet vision protocol.

[0052] The three-dimensional reconstruction module, set up in an industrial computer, is used to receive the light field data, execute a light field-based three-dimensional reconstruction algorithm, and obtain a dense three-dimensional point cloud model of the tea tree canopy.

[0053] The visual positioning module is configured to identify and locate tea bud target points in a 3D point cloud model through 2D target detection and 3D point cloud segmentation algorithms, and output the 3D coordinates of the target.

[0054] The conditional encoding module is configured to input the target location of the tea bud and the point cloud of the obstacle environment into the convolutional neural network conditional encoder to generate a multimodal conditional vector;

[0055] The path planning module is used to perform multi-step denoising iterations on the input conditional vector and random noise based on the conditional denoising diffusion model and combined with the U-shaped convolutional neural network Unet structure, to generate a collision-free smooth path from the initial position of the robotic arm to the target position of the tea bud.

[0056] The inverse kinematics analysis module, located at the actuator control end, is used to convert the path trajectory points generated by the path planning module into parallel Delta robot arm angles.

[0057] The aforementioned 3D reconstruction module, visual positioning module, conditional coding module, path planning module, and inverse kinematics analysis module are implemented by building an algorithm program.

[0058] The execution control module is configured to send angle commands to the robotic arm controller via an RS485 serial communication bus, drive the stepper motor to perform path movement, and control the end gripper to complete the tea bud picking task, forming an end-to-end closed-loop operation.

[0059] Furthermore, the system is applicable to dense canopy crop scenarios where premium tea buds are picked. Its adaptation method includes: replacing the training dataset in the target detection network; adjusting the light field reconstruction parameters to adapt to the target crop size; modifying the safety distance parameter in the target guidance function of the denoising diffusion model; and adapting the robotic arm motion model according to the crop type. In the tea bud picking task, the parallel Delta robotic arm inverse kinematics model is replaced with a serial multi-degree-of-freedom robotic arm solution model.

[0060] The beneficial effects of the present invention include at least one of the following:

[0061] (1) This invention utilizes light field three-dimensional reconstruction and multi-view fusion method, combined with improved YOLOV8 target detection and PointNet++ segmentation network, to achieve high-precision spatial positioning of tea bud targets in the tea tree canopy, ensuring the accuracy of the robotic arm picking task and improving the quality of tea picking.

[0062] (2) This invention proposes a path planning method based on Conditional Denoising Diffusion Model (DDPM), which avoids the cumbersome process of smoothing the path generated by traditional sampling methods through filtering, spline fitting, and other steps. By jointly encoding the target position, obstacle information, and time step conditions, and combining the target guidance steering function with the DDPM inverse denoising process, the target attraction and obstacle avoidance of the path point sequence can be realized to quickly generate the obstacle avoidance picking path for the robotic arm, meeting the requirements of target accessibility and robotic arm dynamics feasibility, and improving the efficiency of tea bud picking.

[0063] (3) The present invention constructs an end-to-end closed-loop system based on light field three-dimensional reconstruction, two-dimensional detection, three-dimensional segmentation, conditional coding and DDPM path generation, realizing the integration of tea tree canopy environment perception, target recognition, path planning and robotic arm execution control, and improving the overall stability and robustness of the system.

[0064] (4) The method of the present invention has strong adaptability and versatility. It is not only applicable to tea bud picking tasks, but can also be extended to the picking path planning and execution of other fruits, vegetables and horticultural crops. It has broad engineering application and promotion value. Attached Figure Description

[0065] Figure 1 This is a flowchart of a tea bud picking obstacle avoidance path planning method based on multimodal information fusion and diffusion planning, as described in this invention.

[0066] Figure 2 The improved YOLOV8 tea bud target detection network EMA module provided by this invention;

[0067] Figure 3 The overall structure diagram of the improved PointNet++ 3D segmentation network provided by this invention;

[0068] Figure 4 A schematic diagram comparing the expected effects of RRT planning and denoising diffusion model planning in existing technologies;

[0069] Figure 5 The overall algorithm framework diagram for generating conditional denoising diffusion trajectories provided by this invention;

[0070] Figure 6 The network structure diagram for path generation of the conditional denoising diffusion model based on U-net provided by this invention;

[0071] Figure 7 A schematic diagram of the denoising path generation result of the diffusion model provided by the present invention;

[0072] Figure 8 This is a schematic diagram of the end-to-end system architecture provided by the present invention; Detailed Implementation

[0073] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0074] like Figure 1 As shown in the flowchart, the tea bud picking obstacle avoidance path planning method based on multimodal information fusion diffusion planning provided by the present invention mainly includes the following steps:

[0075] Step 1: Light Field Data Acquisition

[0076] A light field camera is fixed on a circular sliding rail to capture multi-view panoramic images of the tea tree canopy, obtaining multiple frames of sub-aperture images to form a light field image set.

[0077] Step 2: 3D Reconstruction of the Light Field

[0078] A three-dimensional reconstruction of multi-view light field images was performed using ray consistency constraints, edge-preserving stereo matching, and light field binding adjustment algorithms to generate a dense point cloud model of the tea tree canopy.

[0079] (1) Ray consistency constraint

[0080] By combining the ray consistency relationship of the multiple aperture image in the light field, the ray consistency residual of the multiple aperture image is constructed: ;

[0081] in, For point In the Ray projection of individual aperture views, by minimizing This improves the global geometric consistency between camera pose and 3D structure.

[0082] (2) Maintaining three-dimensional matching at the edges

[0083] By utilizing an edge-preserving cost aggregation algorithm, image gradient weights are incorporated when calculating disparity. :

[0084] ;

[0085] in, Dynamically adjust based on image gradient distribution to preserve depth variations at the boundaries of tea buds and leaves, avoiding excessive smoothing that could lead to loss of target information.

[0086] (3) Light field binding adjustment

[0087] By using Light Field Bundle Adjustment, the weighted sum of reprojection error and ray consistency error is minimized:

[0088] ;

[0089] in, For the observation image point, Let be the projection point. These are the weighting coefficients.

[0090] By optimizing the camera pose and 3D point coordinates, a high-precision, dense, and clearly defined tea tree canopy point cloud is obtained, providing environmental support for subsequent target detection and path planning.

[0091] Step 3: Two-dimensional tea bud target detection

[0092] like Figure 2 As shown, this invention introduces an efficient multi-scale attention module (EMA) into the YOLOv8 detection network to enhance the network's ability to detect small targets (tea buds). Through a multi-scale parallel branch structure, it achieves joint modeling of local and global dependencies. Specifically, it includes:

[0093] First, a channel-dimensional grouping strategy is applied to the input feature map. The input feature tensor C×H×W is first divided into G subgroups according to the channel dimension, with each subgroup having a dimension of (C / G)×H×W. Each subgroup is processed through three parallel paths. The first and second branches use 1×1 convolutions, combined with horizontal and vertical global average pooling respectively, to capture cross-dimensional channel information. The calculation formula is as follows:

[0094] ;

[0095] ;

[0096] The third branch uses a 3×3 convolutional kernel to capture local inter-channel interaction features and enhance the feature representation space. The output is then normalized using Softmax and combined with a two-dimensional Gaussian mapping to establish pixel-level spatial dependencies. The calculation formula is as follows:

[0097] ;

[0098] in, For channel weights, For Gaussian kernel bandwidth, () represents the Sigmoid function.

[0099] Finally, the weighted fusion of branch outputs is achieved through matrix dot product, and the result is reconstructed into a feature map of input dimension C×H×W, preserving the differences in channel importance and spatial structure information, thus enhancing the model's ability to discriminate small targets. This invention embeds the EMA module into the ELAN module of the YOLOv8 network, forming an ELAN-EMA composite structure, enabling the network to possess stronger context awareness and cross-dimensional interaction capabilities during multi-scale feature fusion.

[0100] With the above improvements, the YOLOV8 detection network has significantly improved the detection accuracy and robustness of small targets such as tea buds, while maintaining a fast inference speed, providing high-quality target location information input for subsequent 3D point cloud segmentation and path planning modules.

[0101] Step 4: Three-dimensional tea bud segmentation and positioning

[0102] This invention is based on the PointNet++ architecture, which has a significantly lower parameter count than similar models, making it suitable for deployment in computationally limited agricultural robot systems. However, its cross-scale feature fusion capability is limited, and its performance in segmenting small targets is insufficient. Improvements include...

[0103] (1) Introducing a spatial attention module

[0104] To enhance the model's ability to distinguish tea bud targets, this invention introduces a spatial attention module (SAM) into the PointNet++ network to improve its ability to model spatial dependencies and express key region features.

[0105] The input feature map X is processed by first generating a query matrix Q, a key matrix K, and a value matrix V through 1×1 convolution:

[0106] ;

[0107] in, These are learnable parameters.

[0108] Next, the spatial similarity between the query and the key matrix is ​​calculated to form the attention weight matrix:

[0109] ;

[0110] in, The dimension of the key is used to avoid gradient vanishing. Finally, attention weights are applied to the value matrix to complete spatial feature enhancement. ;

[0111] (2) Improve network structure design

[0112] As attached Figure 3As shown, the improved PointNet++ network structure includes: Before the first Set Abstraction (SA) module, a SAM module is introduced to enhance key region features through adaptive spatial weight allocation. The MLP layer adopts a dimensionality-upgrading-downgrading structure, with an internal dimension four times that of the input and output, enhancing feature extraction capabilities. Subsequently, the PointNet++ Feature Propagation (FP) module is connected. This module combines inverse distance weighted interpolation with residual connection technology. First, spatial correlation-weighted interpolation is performed on the input features. Then, features lost during downsampling in intermediate layers are fused through cross-layer skip connections. Finally, a fully connected classification layer is connected to complete the end-to-end tea bud segmentation task, achieving millimeter-level precision 3D localization of tea bud targets, providing accurate target input for subsequent conditional coding and path planning.

[0113] Step 5: Obstacle Environment Coding

[0114] In this invention, after completing the three-dimensional segmentation and localization of tea buds, it is necessary to input the environmental branch and leaf obstacle information and the tea bud target position into the path planning model to form a multimodal input for generating a picking path under certain conditions.

[0115] First, the point clouds of tea tree branches and trunks identified in the PointNet++ segmentation results are clustered and filtered to remove noise points. A 3D obstacle environment map is then constructed using the KD-tree indexing method. This map preserves: the distribution of branch and leaf obstacles, spatial geometric structure information, and their relative positional relationship with the target tea bud picking points.

[0116] To enable the path planning model to simultaneously perceive obstacle environmental features and target location, the CNN conditional encoder in this invention includes: a three-layer one-dimensional convolutional structure that extracts local obstacle features along the path length or point sequence dimension, with each convolutional layer having a kernel size of 3 and a stride of 1, and incorporating a ResBlock module to enhance feature representation and avoid gradient vanishing. After each convolutional module, the input and output are added through a residual connection structure, and the Mish activation function is used to improve the model's nonlinear fitting ability and the diversity and stability of the conditional vector. The obstacle features extracted by the convolution are encoded into a fixed-dimensional global descriptor through a global max-pooling layer, and concatenated with the target tea bud position vector. This is then fused through a multilayer perceptron (MLP) to form the final scene conditional vector. In the DDPM diffusion process input, a single-layer linear fully connected layer is used to map the time step encoding to a temporal embedding vector consistent with the conditional vector, which is added between convolutional layers to achieve joint conditional-temporal input, further improving the stability and controllability of the denoising generation.

[0117] Step 6: Path planning for the denoising diffusion model

[0118] This invention models the tea bud picking path planning task as a conditional trajectory generation problem. The core objective is to learn the trajectory distribution from the starting point to the target point in order to generate a collision-free, smooth picking path. Figure 4 As shown, traditional sampling-based path planning methods (such as RRT) typically generate paths by connecting random tree nodes. Although the calculation is simple, the results are jagged, discontinuous, and require post-processing for smoothing. In contrast, the denoising diffusion model of this invention is expected to directly generate smooth and continuous obstacle avoidance paths, reducing post-optimization steps and improving the execution efficiency and stability of the robotic arm.

[0119] The denoising diffusion model consists of two stages: forward diffusion and backward denoising. Forward diffusion progressively adds noise to the data, while backward denoising recovers the data from the noise. This process can be described as a Markov chain. The backward process is learned from the noise by a trained neural network model. Forward diffusion is a Markov forward diffusion process that distributes the training path data from distribution x0... The process of converting q(x) into Gaussian noise and performing T diffusion steps to obtain the output distribution:

[0120] ;

[0121] For some constants This is called variance scheduling, where t=1,...,N are the diffusion time steps, and N is the number of diffusion steps. It is the noise scale of time step t.

[0122] set up and The cumulative distribution is represented as:

[0123] ;

[0124] Given a data point A Gaussian noise data sample The output after the first t-direction diffusion step can be calculated.

[0125] ;

[0126] Inverse denoising aims to recover the true path distribution. Noise is predicted by training a neural network. By gradually removing noise, the formula is updated as follows: ;

[0127] in For a noise prediction network, C is the condition vector. , , Preset noise scheduling parameters;

[0128] After the last iteration, using t=1 and Perform the final output step.

[0129] The model training objective is to minimize the mean squared error loss between the predicted noise and the actual noise.

[0130] ;

[0131] in This is the sampling time step, T=1000. , , , This is the output path of the final step, and the only model that needs to be trained is one that can predict noise from noisy data x and time step t. Model.

[0132] Furthermore, in the reverse denoising and recovery process, for the tea bud target, a target-guided steering function is designed, which is designed for the tea bud target coordinates and critical path quality indicators (such as smoothness and collision). Specifically, it consists of the following three parts:

[0133] ;

[0134] in The smoothness of the path is constrained using second-order differences:

[0135] ;

[0136] In all denoising steps, through gradient Suppress mutations and guide smooth pathways.

[0137] Attraction function centered on the target tea bud Guide the harvesting organizations to move closer to the target.

[0138] Drawing on the obstacle avoidance function of an artificial potential field, for the center of the obstacle region on the blade ,definition

[0139] ;

[0140] in , Control the intensity and range of obstacle avoidance to ensure a safe distance between the picking path and branches and leaves.

[0141] And its gradient is used as a conditional term in each backdiffusion step. In the middle, based on noise predictor Perform standard sampling while introducing target gradient guidance. , =0.1 is the guiding strength coefficient. A gradient step is performed to maximize the score by modifying the mean in the reverse process. The sampling process of the guiding path guides the model to gradually recover from a random trajectory to a picking path that meets the target constraints.

[0142] like Figure 5 As shown, the algorithm framework for generating conditional denoising diffusion trajectories in this invention includes two stages: training and inference. During training, the conditional encoder outputs a multimodal conditional vector, and a batch of paths and a random time step are sampled from the training dataset to add random noise to the paths. For t=T,...,1, the neural network predicts the noise. Update the sample mean; update according to the inverse denoising formula. Furthermore, the training data comes from the initial solution of feasible harvesting paths generated by RRT. The paths are smoothed using a B-spline curve fitting optimization method to obtain expert training data, which is then input into the model for conditional forward diffusion and reverse denoising training. The model training is implemented using PyTorch, and all models are trained using the Adam optimizer with a batch size of 128 batches, a sampling frequency of 1000, and a learning rate of 0.0002. The training optimization objective is to minimize the mean squared error loss between the predicted noise and the actual noise. An early stopping mechanism is triggered when the validation loss does not decrease for 5 consecutive rounds to prevent overfitting.

[0143] Inference phase, sampling The network is initialized with random Gaussian noise and then denoised iteratively using conditionalization. The diffusion prior mean is calculated using the gradient of the target-guided steering function and iteratively updated during the denoising steps from T to 1, ultimately outputting an executable, smooth, collision-free target-reachable path. The data is directly sent to the robotic arm control system. The pseudocode for the inference generation process is as follows:

[0144]

[0145] Furthermore, such as Figure 6 As shown, the noise predictor in the algorithm framework is encoded using U-net, which is a residual convolutional neural network consisting of downsampling blocks followed by bottleneck and upsampling blocks. It has the following characteristics: the model consists of a downsampling module, an intermediate bottleneck module, an upsampling module, and an output module, and fuses multi-scale features through skip connections. The specific structure includes:

[0146] (1) The downsampling module consists of two ResNetBlockD modules and one convolutional layer with a stride of 2. Each ResNetBlockD module includes four one-dimensional convolutional layers. The first layer has C input channels and 2C output channels. The second layer maintains the channel dimension, the third layer expands the channels to 4C, and the fourth layer reduces the dimension back to 2C. Each convolutional layer is followed by group normalization and the Mish activation function. Finally, it is added to the residual connection through Dropout (p=0.1). The time step t embedding is mapped to a tensor with the same dimension as the current convolutional output through a single fully connected layer and is weighted and injected between convolutional layers to ensure the effective transmission of temporal condition information. After feature extraction, the convolutional layer with a stride of 2 halves the path length dimension and gradually expands the channel dimension after downsampling.

[0147] (2) The intermediate bottleneck module contains three layers of ResnetBlockD modules, keeping the tensor dimension unchanged, and introducing the time step t embedding vector obtained by linear transformation in each layer, which is added to the convolution output to ensure the injection of time condition information;

[0148] (3) The upsampling module consists of two ResNetBlockD modules and one transposed convolutional layer with a stride of 2. The ResNetBlockD module has the same structure as the downsampling module, with the channel dimension gradually decreasing and the transposed convolution doubling the path length dimension. At the same time, each upsampling module receives the skip connection output of the corresponding downsampling module, and multi-scale information fusion is achieved through dimension concatenation.

[0149] (4) The output module consists of two one-dimensional convolutional layers. The first convolutional layer has a kernel size of 3, a stride of 1, and C output channels. It combines the output features of the upsampling module. The second convolutional layer has a kernel size of 1×1, which is used to convert the tensor dimension into the number of channels for the final noise prediction. The path is represented as a C×n tensor. The model performs convolution along the path length dimension, which has the ability to adapt to variable length.

[0150] Throughout the U-net network architecture, the Mish function is used uniformly for activation to ensure gradient flow stability. GroupNorm is used instead of BatchNorm for normalization, improving robustness to different batch sizes. The network design features: adaptability to variable path lengths, preservation of local details through skip connections, and enhanced feasibility of path generation under environmental constraints.

[0151] By combining the aforementioned conditional denoising diffusion model with the U-net architecture and the target guidance function to optimize path smoothness, target reachability, and obstacle safety distance, the efficiency of path planning for tea bud picking and its robustness in adapting to complex environments are significantly improved.

[0152] like Figure 7As shown, the reverse denoising process of DDPM path generation is illustrated. In the figure, the left side represents the random path in the early stage of diffusion, exhibiting high noise disturbance; the right side represents the final generated path in the late stage of diffusion, presenting a continuous, smooth, and obstacle-avoiding picking trajectory. This figure intuitively demonstrates that as the number of diffusion steps decreases, the path is gradually denoised and optimized, ultimately generating a high-quality picking path that meets the target constraints.

[0153] Step 7: Robotic arm path execution

[0154] like Figure 8 The diagram shown illustrates the end-to-end closed-loop architecture of the tea bud harvesting system provided by this invention. The overall system comprises:

[0155] (1) Perception system

[0156] A light field camera is configured to acquire multi-view light field images of the tea tree canopy through the GigE Vision interface. An industrial computer receives the light field data and executes a 3D reconstruction algorithm to achieve comprehensive 3D perception of the harvesting environment.

[0157] (2) Decision-making system

[0158] Based on an industrial computer control platform, the system integrates a 3D reconstruction module, a target recognition and 3D segmentation module, a conditional coding module, and a path planning module. The internal modules of the industrial computer achieve high-speed data interaction through a bus, and the calculation results are output within milliseconds, meeting the real-time requirements of tea bud picking.

[0159] (3) Execution system

[0160] Based on the path trajectory sequence generated by the decision-making system, the industrial computer sends path instructions to the robotic arm controller via serial communication. The parallel Delta robotic arm controller drives the stepper motor to run at the inverse angle and controls the end gripper module to complete the opening and closing action, thus realizing the picking of tea buds.

[0161] This system forms a closed-loop architecture encompassing multimodal perception, target recognition and path planning, and execution control, constructing an intelligent operating system for tea bud harvesting and improving the level of operational intelligence. It can be extended to different parallel structures or multi-degree-of-freedom robotic arms, and migration can be achieved by updating the inverse kinematics solution module.

[0162] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A tea bud picking obstacle avoidance path planning method based on a multi-modal information fusion diffusion model, characterized in that, The method comprises the following steps: Light field data acquisition, using a light field camera to collect multi-view light field images of tea tree canopy, and obtaining original multi-dimensional image data of the canopy; Three-dimensional point cloud reconstruction, based on the ray consistency constraint, edge preserving stereo matching and light field binding adjustment algorithm, a dense three-dimensional point cloud model of the tea tree canopy is reconstructed; Two-dimensional tea bud target detection, based on an improved YOLOV8 network model, using an embedded multi-scale attention module EMA, two-dimensional pixel coordinates of the tea bud target are extracted; Three-dimensional tea bud segmentation and positioning, using an improved PointNet++ network, combining a density perception farthest point sampling method DA-FPS and a spatial attention module SAM, the two-dimensional detection result is projected to a three-dimensional point cloud, and three-dimensional coordinates of the tea bud target are obtained; Obstacle environment coding, the branch and leaf point cloud is segmented, a three-dimensional obstacle environment map is constructed, a convolutional neural network CNN conditional encoder is used to extract a condition vector, and a multi-modal input is formed with the target position; Denoising diffusion probability model DDPM path generation, based on a U-net architecture DDPM model, the condition vector is input, multi-step denoising iteration is performed, a target guiding steering function is introduced in the reverse denoising process, and the path is optimized through gradient iteration; The U-net is a U-shaped convolutional neural network structure commonly used in generation tasks; Mechanical arm path execution, the perception module output is directly input into the planning module, the generated path is real-time issued to the mechanical arm control platform by the planning module, the mechanical arm is driven to complete the tea bud picking task, and an end-to-end closed loop picking operation of perception-planning-execution is completed.

2. The method of claim 1, wherein, The specific implementation of the light field three-dimensional point cloud reconstruction includes: a light field camera is installed on a ring-shaped moving slide rail to perform multi-view surround shooting; a multi-sub-aperture view ray consistency residual is constructed: , and a camera pose and a three-dimensional structure are optimized, wherein represents a projection ray of a three-dimensional point X under the i-th view; an edge-preserving cost aggregation algorithm is adopted: , wherein is a stereo matching cost function, is a gradient weight, represents image intensity values of left and right views, and d represents a parallax; is a target tea bud coordinate point; a light field bundling adjustment is performed to minimize a weighted error: , wherein represents weight coefficients of a re-projection error and a ray consistency error, , represents observed image points and projected image points.

3. The method of claim 1, wherein, The improved YOLOV8 network embeds an efficient multi-scale attention module in an efficient layer aggregation network structure module ELAN, and the implementation manner comprises the following steps: input feature maps are grouped by channels, three branches are processed in parallel, the output is combined through attention weight Softmax normalization and two-dimensional Gaussian mapping, a pixel-level spatial dependency relationship is established, and a calculation formula is as follows: wherein is an attention weight, used for spatial dependency modeling, is a standard deviation of Gaussian distribution, is an attention weight coefficient, denotes the coordinates of a target pixel point; the EMA module embeds the efficient layer aggregation network module ELAN to form an ELAN-EMA composite structure, and the small target detection capability is enhanced.

4. The method of claim 1, wherein, The improved PointNet++ network comprises: calculating the point cloud density distribution through DA-FPS to improve the sampling probability of the tea bud sparse area; the SAM calculates the spatial similarity weight of the point and the neighborhood point in the feature aggregation stage; the multi-layer perception machine MLP adopts a structure of first increasing the dimension to 4 times the input dimension and then reducing the dimension; and the features are fused through inverse distance weight interpolation and cross-level jump connection.

5. The method of claim 1, wherein, The CNN conditional encoder comprises: three one-dimensional convolutions with a kernel size of 3 and a step size of 1, each layer being connected with a residual block ResBlock; residual connection and Mish activation function; time step embedding: the time step encoding is mapped to an embedding vector with the same dimension as the condition vector through a fully connected layer, and is stacked between the convolution layers.

6. The method of claim 1, wherein, The denoising diffusion model path generation method models the tea bud picking path planning task as a conditional trajectory generation problem. The core objective is to learn the trajectory distribution from the starting point to the target point to generate a collision-free, smooth picking path. Specifically, it includes: a forward diffusion process, adding noise to the input expert data using a Markov process, and diffusing the training path data from distribution x0... q(x) is converted into Gaussian noise, where x0 is the original distribution of the expert path data and q(x) is the probability distribution function of the path data; and T diffusion steps are performed to obtain the output distribution: ,in This is called variance scheduling, where t=1,...,N are the diffusion time steps, and N is the number of diffusion steps. Let be the identity matrix; and , To control noise intensity, The cumulative noise scheduling parameters are used to calculate the output after the first t-step forward diffusion. ,in Let t be the noisy path in the diffusion process at step t. Gaussian noise is added at step t, conforming to N(0,1); the reverse denoising process predicts the noise by training a neural network. By gradually removing noise, the formula is updated as follows: ,in For a noise prediction network, C is the condition vector. , , To preset noise scheduling parameters, Standard Gaussian noise is used for random sampling during the reverse process; the model training objective is to minimize the mean squared error loss between the predicted noise and the true noise. ,in Let be the loss function at step t. This is the sampling time step, T=1000 diffusion steps. , normal distribution, , This is the output path.

7. The method of claim 6, wherein, In the reverse denoising process, a target guiding steering function is further introduced to meet the target accessibility, path smoothness and obstacle avoidance requirements, which consists of the following three parts: where is the target guiding steering function, the smoothness term constraint of the path, adopts the second-order difference, is the target proximity constraint term, is the obstacle avoidance constraint term: In all denoising steps, the gradient suppresses mutations and guides smooth paths; the attractive function centered on the target tea bud point guides the target to approach, where represents the target tea bud position coordinates, and the obstacle avoidance function inspired by the artificial potential field is defined for the center of the leaf obstacle region where , controls the obstacle avoidance strength and range, and ensures the safe distance between the picking path and the branches and leaves.​ 8. The method of claim 6, wherein, The network of the U-net structure comprises: a down-sampling module containing two layers of ResnetBlockD modules, each module being composed of four one-dimensional convolution layers, each layer being followed by a Mish activation and a GroupNorm normalization, a Dropout rate being 0.1, and being fused through a residual connection, a down-sampling convolution step being 2, realizing a path length reduction by half, and a channel dimension being gradually expanded; an intermediate bottleneck module containing three layers of ResnetBlockD modules, keeping the tensor dimension unchanged, and introducing a time step t embedding linearly transformed by a full connection into each layer and added to the convolution output; an up-sampling module comprising two layers of ResnetBlockD modules and a transpose convolution layer with a step of 2, realizing a path length doubling, simultaneously receiving a jump connection input from a corresponding down-sampling module, fusing multi-scale features through splicing, and ensuring the global smoothness and local detail preservation of the path generation; and an output module composed of two convolution layers, the first layer being a standard convolution for integrating the up-sampling output features, and the second layer being a convolution kernel with a size of 1x1 for converting the output tensor dimension to a required channel number, realizing final noise prediction; the path is represented as a Cxn tensor, where C is the channel dimension and n is the path length, the model performs a one-dimensional convolution operation along the path length dimension, can process path inputs of different lengths, and has length-variable adaptability; and the multi-modal information joint input with the conditional coding module is supported, and the conditional path generation based on the target position and the obstacle environment is realized through network condition injection.

9. The method of claim 8, wherein, The training network parameter setting comprises: the training data is derived from RRT (Rapidly-exploring Random Tree), expert data is obtained by using a B-spline curve fitting optimization method to smooth the path after generating an initial solution of a feasible picking path, model training is realized by using a deep learning framework PyTorch, a batch size of 128 batches, a sampling frequency of 1000, and a learning rate of 0.0002 of an adaptive matrix estimation optimizer Adam optimizer are used for training, and an early stop mechanism is triggered when the validation loss does not decrease for 5 consecutive rounds, so as to prevent overfitting.

10. A tea bud picking obstacle avoidance path planning system based on a multi-modal information fusion diffusion model, characterized in that, The method comprises the following steps: The light field acquisition module is configured to acquire multi-view images and light field data of a tea tree canopy through a light field camera and output the data to an industrial computer through a GigE Vision interface; The three-dimensional reconstruction module is configured to receive the light field data, execute a light field three-dimensional reconstruction algorithm, and acquire a dense three-dimensional point cloud model of the tea tree canopy; The visual positioning module is configured to identify and locate a tea bud target point in the three-dimensional point cloud model through a two-dimensional target detection and three-dimensional point cloud segmentation algorithm and output a target three-dimensional coordinate; The conditional coding module is configured to input the tea bud target position and obstacle environment point cloud into a convolutional neural network conditional encoder to generate a multi-modal conditional vector. The method comprises the following steps: The light field acquisition module is configured to acquire multi-view images and light field data of a tea tree canopy through a light field camera and output the data to an industrial computer through a GigE Vision interface; The three-dimensional reconstruction module is configured to receive the light field data, execute a light field three-dimensional reconstruction algorithm, and acquire a dense three-dimensional point cloud model of the tea tree canopy; The visual positioning module is configured to identify and locate a tea bud target point in the three-dimensional point cloud model through a two-dimensional target detection and three-dimensional point cloud segmentation algorithm and output a target three-dimensional coordinate; The conditional coding module is configured to input the tea bud target position and obstacle environment point cloud into a convolutional neural network conditional encoder to generate a multi-modal conditional vector. The path planning module is configured to perform multi-step denoising iteration on the input condition vector and random noise based on a conditional denoising diffusion model combined with a U-shaped convolutional neural network Unet structure to generate a collision-free smooth path from an initial position of the mechanical arm to a target position of the tea bud. The inverse kinematics analysis module is arranged at a control end of an actuator and is configured to convert the path trajectory points generated by the path planning module into angles of the parallel Delta mechanical arm. The execution control module is configured to send angle instructions to the mechanical arm controller through an RS485 serial communication bus to drive the stepper motor to perform path movement and control the end gripper to complete the tea bud picking task, thereby forming an end-to-end closed loop operation.

Images