Strawberry picking method and system based on structured geometric guidance and diffusion strategy

Abstract

This invention provides a strawberry picking method and system based on a structured geometric guidance and diffusion strategy, relating to the field of mechanical control technology. The method includes: acquiring strawberry images and picking robot arm information; extracting visual feature vectors, strawberry geometric information vectors, and picking robot arm body feature vectors; performing feature fusion using the strawberry center point coordinates and the strawberry stem growth direction vector as strong priors to obtain an observation condition vector; adding preset noise to the ideal picking trajectory through a diffusion strategy to generate a noisy picking trajectory; training a noise prediction network; generating a noisy action sequence including the end-effector pose of the picking robot arm with a preset step length; inputting the observation condition vector, the noisy action sequence, and the current diffusion step number into the trained noise prediction network and outputting the current predicted noise; iteratively removing the current predicted noise from the noisy action sequence to obtain a target action sequence; and controlling the picking robot arm according to the target action sequence to pick strawberries.

Description

Technical Field

[0001] This invention relates to the field of mechanical control technology, and in particular to a strawberry picking method and system based on a structured geometric guidance and diffusion strategy. Background Technology

[0002] Strawberry fruits are small and easily damaged, with uneven ripening and distribution at different heights and locations. The fruits are often closely connected, and the leaves and branches severely obstruct the view, increasing the difficulty of visual identification and mechanical handling. In addition, the strawberry stems are fragile, and the fruits are easily crushed or pulled during harvesting. Manual harvesting is inefficient and labor-intensive, making it difficult to achieve efficient and safe harvesting.

[0003] Automated strawberry harvesting can significantly improve harvesting efficiency, reduce labor intensity, decrease fruit damage, and ensure fruit quality. Through intelligent visual recognition and precise control of robotic arms, selective harvesting of fruits at different stages of ripeness can be achieved, improving the level of precision management in agricultural production. At the same time, automated harvesting also helps alleviate seasonal labor shortages, enhancing the overall efficiency and sustainable development capabilities of the industry chain.

[0004] However, the existing automated strawberry picking process lacks accurate perception of the fruit position, stem direction, and shading environment. Visual information is easily interfered with by light and foliage, resulting in low picking efficiency and high fruit damage rate. Summary of the Invention

[0005] To address the technical problems of existing automated strawberry harvesting processes, such as the lack of precise perception of fruit position, stem direction, and shading environment, and the susceptibility of visual information to interference from light and foliage, leading to low harvesting efficiency and high fruit damage rate, this invention provides a strawberry harvesting method and system based on a structured geometric guidance and diffusion strategy.

[0006] The technical solutions provided by the embodiments of the present invention are as follows: First aspect This invention provides a strawberry picking method based on a structured geometry guidance and diffusion strategy, comprising: S1: Acquire strawberry images and information about the picking robotic arm; S2: Extract visual feature vectors and strawberry geometric information vectors from strawberry images; and extract the main feature vector of the picking robot arm from the picking robot arm information; wherein, the strawberry geometric information vector includes the coordinates of the strawberry center point and the strawberry stem growth direction vector; S3: Using the strawberry center point coordinates and the strawberry stem growth direction vector as strong priors, feature fusion is performed on the visual feature vector, strawberry geometric information vector and the picking robot arm body feature vector to obtain the observation condition vector. S4: Add preset noise to the ideal picking trajectory through a diffusion strategy to generate a noisy picking trajectory; S5: Train the noise prediction network based on the noisy picking trajectory, the diffusion steps of the diffusion strategy, and the observation condition vector corresponding to the ideal picking trajectory; S6: Generate a noisy motion sequence with a preset step length, including the end-effector pose of the picking robot arm; S7: Input the observation condition vector, noisy action sequence, and current diffusion step number into the trained noise prediction network, and output the current predicted noise; S8: Iteratively remove the current predicted noise from the noisy action sequence to obtain the target action sequence; S9: Control the picking robotic arm according to the target action sequence to pick strawberries.

[0007] Second aspect This invention provides a strawberry picking system based on a structured geometry guidance and diffusion strategy, comprising: processor; A memory storing computer-readable instructions, which, when executed by the processor, implement the strawberry picking method based on a structured geometry guidance and diffusion strategy as described in the first aspect.

[0008] Third aspect The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the strawberry picking method based on a structured geometry guidance and diffusion strategy as described in the first aspect.

[0009] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this embodiment of the invention, by deeply fusing explicit geometric information of the strawberry with visual features and the body features of the robotic arm, an observation condition vector containing the fruit position and stem direction is constructed. Using this vector as input, a noisy action sequence is generated from the ideal picking trajectory using a diffusion strategy. Then, a noise prediction network iteratively denoises the sequence to obtain a precise target action sequence, achieving accurate control of the picking robotic arm. This method effectively removes visual noise caused by changes in lighting and foliage occlusion, retaining only information highly relevant to the picking task. This allows the robotic arm to accurately identify the fruit position and stem direction even in complex environments, significantly improving picking efficiency and reducing fruit damage rate. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a strawberry picking method based on a structured geometry guidance and diffusion strategy, provided in an embodiment of the present invention; Figure 2 A schematic diagram of a noise addition and denoising process provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a noise prediction network provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the action mechanism of a strawberry-inspired bending and separating action according to an embodiment of the present invention. Figure 5 This is a schematic diagram of a strawberry picking system based on a structured geometry guidance and diffusion strategy, provided as an embodiment of the present invention. Detailed Implementation

[0012] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0013] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0014] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0015] Reference manual attached Figure 1 The diagram illustrates a flowchart of a strawberry harvesting method based on a structured geometry guidance and diffusion strategy provided by an embodiment of the present invention.

[0016] Reference manual attached Figure 2 The diagram illustrates a process flow diagram for adding and removing noise provided by an embodiment of the present invention.

[0017] Figure 2 N in the equation represents the total number of diffusion steps. A 0 To complete the target action sequence for inverse denoising. It is a noisy action sequence that includes N steps of diffused noise.

[0018] Reference manual attached Figure 3 The diagram shows a structural schematic of a noise prediction network provided in an embodiment of the present invention.

[0019] Figure 3 middle, Right now t Feature dimensions of strawberry images at any given time. Let the direction of fruit stalk growth be the vector. The coordinates of the center point of the fruit, The current joint angle of the robotic arm. This is the pose of the robotic arm's end effector.

[0020] Figure 3 The multi-head self-attention module in the decoder is the first multi-head self-attention module. Figure 3 The multi-head self-attention module in the encoder is the second multi-head self-attention module.

[0021] Reference manual attached Figure 4 The diagram illustrates the action mechanism of a strawberry biomimetic bending and separation according to an embodiment of the present invention.

[0022] Figure 4 ① refers to the rotational bending action, and ② refers to the separation action.

[0023] This invention provides a strawberry picking method based on a structured geometry guidance and diffusion strategy. This method can be implemented using a strawberry picking device based on the structured geometry guidance and diffusion strategy, which can be a terminal or a server. The processing flow of the strawberry picking method based on the structured geometry guidance and diffusion strategy may include the following steps: S1: Acquire strawberry images and information about the picking robotic arm; Specifically, strawberry images can be obtained using an RGB-D depth camera.

[0024] In one possible implementation, the picking robot arm information includes the picking robot arm joint angles and the picking robot arm end-effector pose.

[0025] It should be noted that by acquiring strawberry images using an RGB-D depth camera and simultaneously collecting the joint angles and end-effector pose of the robotic arm, a multimodal information acquisition mechanism combining visual perception and proprioception was constructed. The depth camera can simultaneously capture color texture and three-dimensional spatial structure, providing a data foundation for accurately locating the fruit and analyzing the orientation of the stem; while the proprioceptive state information of the robotic arm allows the system to have a complete understanding of its own posture. Unifying the spatial geometry of the external environment and the motion state of the internal actuator under the same spatiotemporal coordinates provides a data foundation for subsequent model learning and refined biomimetic control.

[0026] S2: Extract visual feature vectors and strawberry geometric information vectors from strawberry images; and extract the main feature vector of the picking robot arm from the picking robot arm information; wherein, the strawberry geometric information vector includes the coordinates of the strawberry center point and the strawberry stem growth direction vector; It should be noted that by constructing a multimodal, structured perception system, deep semantics, explicit geometry, and ontology state are decoupled and extracted, effectively solving the problem of unreliability of single visual features in complex agricultural environments. By independently encoding the coordinates of the strawberry center point and the growth direction vector of the fruit stalk, the model is provided with accurate and generalizable strong geometric priors, which can directly guide the robotic arm to perform biomimetic bending and separation along the fruit stalk axis, thereby significantly improving the accuracy of picking actions and the ability to protect the fruit.

[0027] In one possible implementation, S2 specifically includes: S201: Extract visual feature vectors through the image semantic coding module, which includes the ResNet-18 image semantic coding module and the ViT image semantic coding module; The specific formula for extracting visual feature vectors is as follows: ; in, Represents visual feature vectors. This represents the image semantic coding module. express t A strawberry image.

[0028] S202: Extract the coordinates of the strawberry center point and the strawberry stem growth direction vector using the YOLO v8 target geometry analysis module; The coordinates of the center point of the strawberry are three-dimensional coordinates.

[0029] S203: Normalize the coordinates of the strawberry center point and the strawberry stem growth direction vector respectively; S204: The normalized strawberry center point coordinates and strawberry stem growth direction vector are mapped into strawberry geometric information vectors through sine-cosine position encoding.

[0030] Understandably, in this case, when the fruit stalk features are not significant, visual features are mainly relied upon to implicitly contain geometric information; when the fruit stalk features are significant, explicit geometric features are fused to improve accuracy.

[0031] S205: The ontology state mapping module based on the multilayer perceptron (MLP) network maps the information of the harvesting robot arm into the ontology feature vector of the harvesting robot arm.

[0032] The mapping formula for the feature vector of the harvesting robotic arm is: ; in, This represents the feature vector of the harvesting robotic arm. This represents a multilayer perceptron (MLP) network. They represent t Constantly picking the joint angle of the robotic arm and t The position of the robotic arm's end effector is constantly being captured.

[0033] Specifically, this process constructs a decoupled and complementary multimodal perception system to extract deep visual semantics and explicit geometric structures in parallel from a single image, and independently encodes the ontological state of the robotic arm, fundamentally overcoming the shortcomings of single-modal representation in agricultural environments. The image semantic encoding module captures the global appearance and texture features of the fruit, while the target geometry analysis module accurately regresses the three-dimensional coordinates of the strawberry's center point and the growth direction vector of the fruit stalk. This crucial geometric information is then injected into the model through sine-cosine position encoding, and a multilayer perceptron maps the joint angles and end-effector pose to unified ontological features of the robotic arm. When the fruit stalk features are clear, the explicit geometric vector directly serves as a strong prior, guiding the action to strictly follow the stalk axis for biomimetic bending and separation. When branches and leaves obscure the fruit stalk features, making them insignificant, visual features implicitly supplement the missing geometric information, ensuring the system possesses robust geometric perception and action decision-making capabilities under any complex conditions, significantly improving harvesting accuracy and fruit protection.

[0034] S3: Using the strawberry center point coordinates and the strawberry stem growth direction vector as strong priors, feature fusion is performed on the visual feature vector, strawberry geometric information vector and the picking robot arm body feature vector to obtain the observation condition vector. It is understandable that the observation condition vector includes strawberry location information and strawberry stem orientation information.

[0035] It should be noted that in this process, the module uses low-dimensional explicit geometric features as prior constraints to perform cross-modal space alignment and redundant information filtering on high-dimensional implicit visual features, uniformly mapping them into low-dimensional, dense conditional representations. Finally, the fused mapping generates a low-dimensional, high-information-density observation conditional vector. This vector serves as the conditional input to the subsequent diffusion strategy network. This vector not only contains explicit geometric constraints such as "where the fruit is and which way the stem is pointing," but also preserves the necessary visual context, thus greatly enhancing task relevance before entering the Transformer denoising network and significantly reducing the computational complexity of the generative model.

[0036] Specifically, using the strawberry center point coordinates and strawberry stem growth direction vector extracted by YOLO as strong priors, irrelevant visual noise affected by lighting and leaf movement is extracted from the high-dimensional image, retaining only low-dimensional geometric elements strongly related to the picking task. Since implicit visual features, explicit geometric features, and ontological features reside in different feature spaces, this process first performs sinusoidal multi-frequency positional encoding on the strawberry geometric information vector to enhance its spatial geometric representation capability; subsequently, a cross-modal attention mechanism or multilayer perceptron is used to fuse the features of the three. During this process, using low-dimensional explicit geometric features as prior constraints, cross-modal spatial alignment and redundant information filtering are performed on the high-dimensional implicit visual features, uniformly mapping them into low-dimensional, dense conditional representations. Finally, the fused mapping generates a low-dimensional, high-information-density observation conditional vector.

[0037] By fusing multimodal features using the strawberry center point coordinates and the stem growth direction vector as strong priors, the model achieves complementary advantages between geometric structure and visual semantics, endowing it with stable geometric perception capabilities in complex scenes. This mechanism forces the model to learn to adjust its action strategy based on key geometric information such as stem orientation. Even under interference such as dense fruit or foliage obstruction, it can generate high-precision biomimetic bending trajectories that strictly follow the stem growth direction, improving harvesting efficiency.

[0038] In one possible implementation, S3 specifically refers to: Using the strawberry center point coordinates and the strawberry stem growth direction vector as strong priors, the visual feature vector, strawberry geometric information vector, and picking robot feature vector are fused through a cross-modal attention mechanism or a multilayer perceptron to obtain the observation condition vector.

[0039] The specific fusion formula is as follows: ; in, Represents the observation condition vector. These represent the visual feature vector, the strawberry geometric information vector, and the feature vector of the picking robot arm, respectively. This represents the mapping function for cross-modal attention mechanisms or multilayer perceptrons.

[0040] It's important to note that this process isn't simply about piecing together multimodal features. Instead, it uses low-dimensional, precise explicit geometric vectors as strong prior constraints. Through cross-modal attention mechanisms or multilayer perceptrons, it actively guides high-dimensional implicit visual features and robotic arm proprioceptive features to achieve spatial alignment and redundancy filtering. Ultimately, this compresses and maps heterogeneous information into a low-dimensional, high-information-density unified observation condition vector. This process significantly improves biomimetic harvesting capabilities in complex scenarios because it removes visual noise irrelevant to the harvesting task, such as changes in lighting and swaying branches, forcing the model to focus strictly on the two key geometric elements: fruit position and stem orientation. The resulting observation condition vector possesses both geometric precision and visual contextual information, enabling it to stably generate high-precision bending and separation trajectories that strictly follow the stem axis, even under conditions of dense fruit or occlusion interference. This achieves strong geometric alignment from perception to action.

[0041] The process first will and Normalization is performed, and the result is mapped to a strawberry geometric information vector using sine-cosine multi-frequency position coding. To enhance its spatial geometric representation capabilities. Due to implicit visual features Strawberry geometric information vector With ontological features Since they exist in different feature spaces, a cross-modal attention mechanism or a multilayer perceptron is then used to fuse the features of the three. This ensures that the location of the strawberry can be accurately obtained even when interference is received.

[0042] S4: Add preset noise to the ideal picking trajectory through a diffusion strategy to generate a noisy picking trajectory; The ideal picking trajectory can be obtained by human experts through remote operation, teaching dragging, and other methods, resulting in a robotic arm motion sequence that includes the entire process of "approach-bending-separation." Pre-set noise refers to random perturbation signals intentionally added to the ideal trajectory during the forward propagation process of the diffusion strategy. By adding pre-set noise to the expert-taught trajectory containing the entire process of "approach-bending-separation" using a diffusion strategy, the complex problem of generating bionic motion sequences is cleverly transformed into a series of gradual and simple noise reduction tasks. This method no longer requires the model to directly output a long, intricate sequence that is difficult to generate perfectly in one go. Instead, it allows the model to learn to repair the subtle structure and overall trend of the trajectory at different noise scales, thereby stably generating highly smooth, accurate, and highly diverse bionic picking actions. This effectively solves the problem of traditional teaching learning easily getting trapped in local optima or stiff movements.

[0043] In one possible implementation, S4 specifically includes: S401: Obtain the ideal picking trajectory including the entire picking process, which includes the approach process, bending process and separation process; S402: Add preset noise to the ideal picking trajectory through a diffusion strategy to obtain a noisy picking trajectory.

[0044] The formula for generating the noisy picking trajectory is as follows: ; ; in, Represents the number of diffusion steps i Related noisy harvesting trajectories, Indicates standard Gaussian noise. Represents the number of diffusion steps i The relevant cumulative ideal harvesting trajectory scaling factor, Represents the number of diffusion steps j The relevant preset ideal picking trajectory scaling factor; It should be noted that those skilled in the art can set the preset ideal picking trajectory scaling factor according to actual needs, and this invention does not limit this. The process employs a diffusion strategy, weighting and mixing the ideal picking trajectory, which includes the complete process of "approach-bending-separation" collected by human experts through remote operation, with standard Gaussian noise scaled according to specific rules. This generates a series of noisy trajectory samples ranging from slight perturbations to complete destruction. The complex problem of generating a complete, high-precision bionic action sequence at once is broken down into a series of progressive, simple denoising tasks. The model no longer needs to directly output a perfect final trajectory, but is trained to learn how to repair the local details and global trends of the trajectory step by step under different noise scales. This allows the system to capture the rich action distribution contained in the expert teaching data, rather than simply replicating the average value. Therefore, during inference, it can stably recover highly smooth, accurate, and diverse bionic picking actions from pure noise, significantly enhancing the system's adaptability and generalization performance in complex agricultural environments.

[0045] S5: Train the noise prediction network based on the noisy picking trajectory, the diffusion steps of the diffusion strategy, and the observation condition vector corresponding to the ideal picking trajectory; It should be noted that by training the network with the noisy picking trajectory, diffusion steps, and observation condition vectors incorporating strong geometric priors, the network learns to establish a precise causal mapping between "perceiving the scene" and "repairing actions." That is, the network not only learns to recover the general shape of the trajectory from noise, but is also forced to "understand" the current orientation and spatial position of the strawberry stem through the observation condition vectors. This allows it to specifically reconstruct biomimetic bending and separating actions that closely conform to actual geometric constraints during the denoising process.

[0046] In one possible implementation, S5 specifically includes: Using noisy picking trajectories, diffusion strategy diffusion steps, and observation condition vectors as samples, and using preset noise as labels, the noise prediction network is trained until the loss value between the preset noise and the predicted noise is less than the preset loss value.

[0047] The specific formula for calculating the loss value is as follows: ; in, Indicates the loss value. Expressing expectations, Indicates preset noise. This represents the observation condition vector corresponding to the ideal picking trajectory. Indicates the number of diffusion steps. Represents the number of diffusion steps i Related noisy harvesting trajectories, Indicates that the hyperparameters are A noise prediction network.

[0048] It's important to note that this process employs supervised learning, using pre-added noise as labels to train a noise prediction network to accurately reconstruct the added noise from the noisy picking trajectory, the current diffusion step count, and the observation condition vector at the corresponding moment. This process endows the model with scene-adaptive biomimetic action generation capabilities because the optimization objective of the loss function forces the network to learn not only a general denoising template but also to deeply analyze the strong geometric priors contained in the observation condition vector, such as the strawberry center point coordinates and the fruit stem growth direction, establishing a strict causal mapping between perception and reconstruction. While learning how to recover the trajectory structure from noise, the network is also forced to understand the current spatial posture of the fruit. This allows it to specifically reconstruct high-precision bending and separation actions that strictly follow the fruit stem axis and closely fit actual geometric constraints during the denoising process, ensuring that the generated trajectory is both smooth and universal, while possessing accurate scene response characteristics.

[0049] In one possible implementation, the noise prediction network is specifically a noise prediction network based on the Transformer architecture; The noise prediction network includes an encoder and a decoder, wherein the decoder includes a first multi-head self-attention module and a multi-head cross-attention module interconnected with each other; the encoder includes a second multi-head self-attention module connected to the multi-head cross-attention module; In this module, the query input of the multi-head cross-attention module is the output of the first multi-head self-attention module; the key input and value input of the multi-head cross-attention module are both the outputs of the second multi-head self-attention module. The input to the first multi-head self-attention module is the current noisy action sequence, and the input to the second multi-head self-attention module is the observation condition vector.

[0050] The specific processing formula for the multi-head cross-attention module is as follows: ; in, These represent the query matrix, key matrix, and value matrix, respectively. This indicates a multi-head cross-attention module. express Activation function Indicates the feature dimensions of the query matrix and key matrix, and the index. Indicates transpose; It is understandable that the query input is the Q input head, the key input is the K input head, and the value input is the V input head. In practical applications, the encoder and decoder can be stacked according to actual needs. Here, Q comes from the current noisy action sequence, and K and V come from the observation condition vector. Through cross attention, the model learns to "adjust the normal vector of the bending plane (Query) according to the orientation of the stem (Key)," thereby achieving geometric generalization of biomimetic actions.

[0051] It's important to note that a noise prediction network based on the Transformer architecture deeply interacts with the noisy picking trajectory and the observation condition vector fused with geometric priors, establishing a precise perception-action correlation using a multi-head cross-attention mechanism. This process uses the noisy trajectory as the query input and the stem direction and fruit position from the observation conditions as key-value inputs, enabling the network to dynamically adjust the action trajectory during denoising, ensuring the generated picking path strictly adheres to the geometric constraints of the stem. This architecture achieves high-precision biomimetic motion generation because the cross-attention mechanism forces the network to simultaneously consider the current trajectory state and scene geometric features when predicting noise, thus learning how to adaptively correct the action direction based on the fruit's spatial pose, ultimately producing a professional picking trajectory that conforms to both the kinematics of the robotic arm and the growth morphology of strawberries.

[0052] S6: Generate a noisy motion sequence with a preset step length, including the end-effector pose of the picking robot arm; It should be noted that those skilled in the art can set the preset step size according to actual needs, and this invention does not limit it.

[0053] In one possible implementation, the end-effector pose of the picking robot arm includes an end-effector displacement increment and an end-effector pose rotation increment. S6 specifically includes: S601: Determine the preset step size and the motion dimension corresponding to each step; S602: Generate an initial random noise sequence with the same dimension as the action sequence to be generated based on a standard Gaussian distribution, and use the initial random noise sequence as the initial state of the noisy action sequence.

[0054] It should be noted that in this process, the noisy action sequence is used as the initial input for the inverse denoising process of the diffusion model, so that the target action sequence can be gradually generated under the constraints of the observation condition vector and the current diffusion step number, forming a series of action sequence samples from completely random to near-ideal state. The complex trajectory planning problem is decomposed into multiple manageable sub-tasks, enabling the model to progressively correct errors in the action sequence. By controlling the iterative denoising process, the system can explore a richer action possibility space while maintaining action continuity, thereby generating diverse picking trajectories that conform to the robotic arm's motion constraints and adapt to different fruit growth postures. Ultimately, this achieves a high success rate and low damage biomimetic picking effect, significantly improving the generation quality of the picking action.

[0055] S7: Input the observation condition vector, noisy action sequence, and current diffusion step number into the trained noise prediction network, and output the current predicted noise; When step S7 is executed for the first time, the current diffusion step number is initialized to the preset maximum diffusion step number.

[0056] It should be noted that those skilled in the art can set the maximum number of diffusion steps according to actual needs, and this invention does not limit this.

[0057] S8: Iteratively remove the current predicted noise from the noisy action sequence to obtain the target action sequence; In one possible implementation, S8 specifically includes: S801: Based on the current predicted noise, the noisy action sequence corresponding to the current diffusion step number is denoised and updated to obtain a candidate noisy action sequence; The specific formula for generating candidate noisy action sequences is as follows: ; in, Represents the number of diffusion steps i The relevant preset ideal picking trajectory scaling factor, Represents the number of diffusion steps i -1 related candidate noisy action sequences Represents the number of diffusion steps i Related noisy action sequences, Represents the number of diffusion steps i The relevant backsampling noise standard deviation, This represents random noise that follows a standard normal distribution. S802: Decrease the current diffusion step count by 1, and use the candidate noisy action sequence as the new noisy action sequence. Determine if the current diffusion step count is greater than zero. If yes, return to step S7; otherwise, output the candidate noisy action sequence as the target action sequence.

[0058] It's important to note that the generated trajectory, i.e., the target action sequence, continuously comprises three stages in the time dimension: Approach and Envelopment: Smoothly approaching the fruit and pre-adjusting the end posture; Bionic Bending: Performing an arc motion around the stem, causing the fruit to rotate approximately 90° relative to the stem, and the generated trajectory automatically presents an arc shape centered on the stem; By learning from a large amount of teaching data, the Transformer model "remembers" that after contacting the fruit, it cannot pull directly but must first rotate around the stem. The Attention mechanism ensures that the rotation axis is perpendicular to the plane formed by the stem and gravity. Adaptive Separation: While maintaining the bending posture, separation is achieved by translating along a specific direction. The final segment of the generated trajectory automatically manifests as a translational motion along the growth direction of the stem. The Transformer uses Cross-Attention to focus on the input strawberry stem growth direction vector and directly generates a displacement trajectory parallel to this vector.

[0059] It should be noted that the noisy action sequence is progressively optimized into the target action sequence through an iterative denoising process, and a biomimetic trajectory conforming to geometric constraints is reconstructed using a noise prediction network. This process precisely controls the intensity and direction of denoising at each step, gradually correcting trajectory deviations while preserving action continuity. Specifically, firstly, staged denoising achieves coarse-to-fine action generation, ensuring a smooth and stable trajectory; secondly, a cross-attention mechanism ensures that the denoising process is always guided by geometric features such as the direction of the fruit stalk, and the generated arc bending and separation actions strictly conform to plant growth characteristics; finally, the introduction of random noise terms avoids local optima, making the actions both accurate and adaptable. This enables the system to automatically generate professional picking trajectories containing three stages: "approach-bending-separation," mimicking manual picking techniques while overcoming the problems of stiff movements and easy fruit damage caused by traditional methods.

[0060] S9: Control the picking robotic arm according to the target action sequence to pick strawberries.

[0061] In practical applications, this method achieves efficient and precise strawberry picking by constructing a biomimetic picking system that combines structured geometric guidance with a diffusion strategy. The principle is as follows: First, multimodal perception is used to simultaneously acquire strawberry images and the state of the robotic arm. Visual semantics, explicit geometry (fruit position and stem direction), and ontological features are extracted through decoupling to form a complementary and robust environmental representation. Next, using low-dimensional geometric vectors as strong priors, heterogeneous information is compressed into high-density observation conditions through cross-modal fusion, thereby removing environmental noise and focusing on key picking elements. Then, a diffusion strategy is used to add noise to the ideal trajectory taught by experts and train a noise prediction network, enabling the model to learn to repair trajectory details at different noise scales. Simultaneously, the geometric constraints in the observation conditions (if the stem direction) are deeply bound to the action generation depth through a Transformer cross-attention mechanism, ensuring that the denoising process strictly follows the fruit's spatial posture. Finally, through iterative denoising, a smooth trajectory containing the entire process of "approach-bending-separation" is gradually restored from random noise, and the robotic arm is controlled to execute the denoising. By using strong geometric priors to guide the perception system to remain stable in complex scenarios, and by utilizing the progressive generation characteristics of the diffusion model, the biomimetic motion is decomposed into a series of simple denoising tasks. This combines geometric accuracy with motion diversity, ultimately generating a professional picking trajectory that conforms to both plant growth characteristics and robotic arm motion constraints, thereby achieving high-precision and low-damage picking results.

[0062] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this embodiment of the invention, by deeply fusing explicit geometric information of the strawberry with visual features and the body features of the robotic arm, an observation condition vector containing the fruit position and stem direction is constructed. Using this vector as input, a noisy action sequence is generated from the ideal picking trajectory using a diffusion strategy. Then, a noise prediction network iteratively denoises the sequence to obtain a precise target action sequence, achieving accurate control of the picking robotic arm. This method effectively removes visual noise caused by changes in lighting and foliage occlusion, retaining only information highly relevant to the picking task. This allows the robotic arm to accurately identify the fruit position and stem direction even in complex environments, significantly improving picking efficiency and reducing fruit damage rate.

[0063] Reference manual attached Figure 5 The diagram shows a structural schematic of a strawberry picking system based on a structured geometry guidance and diffusion strategy provided by the present invention.

[0064] This invention also provides a strawberry picking system 20 based on a structured geometry guidance and diffusion strategy, applied to the aforementioned strawberry picking method based on a structured geometry guidance and diffusion strategy, comprising: Processor 201.

[0065] The memory 202 stores computer-readable instructions, which, when executed by the processor 201, implement the strawberry picking method based on a structured geometry guidance and diffusion strategy as described in the method embodiment.

[0066] The strawberry picking system 20 based on structured geometry guidance and diffusion strategy provided by the present invention can execute the strawberry picking method based on structured geometry guidance and diffusion strategy described above and achieve the same or similar technical effects. To avoid repetition, the present invention will not elaborate further.

[0067] This invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the strawberry picking method based on a structured geometry guidance and diffusion strategy as described in the method embodiment.

[0068] The present invention provides a computer-readable storage medium that can implement the steps and effects of the strawberry picking method based on the structured geometry guidance and diffusion strategy in the above-described method embodiments. To avoid repetition, the present invention will not elaborate further.

[0069] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0070] The following points need to be explained: (1) The accompanying drawings of the embodiments of the present invention only involve the structures involved in the embodiments of the present invention. Other structures can refer to the general design.

[0071] (2) For clarity, the thickness of layers or regions is enlarged or reduced in the drawings used to describe embodiments of the invention, i.e., these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element or there may be intermediate elements.

[0072] (3) Where there is no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.

[0073] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for strawberry picking based on structured geometric guidance and diffusion strategy, characterized in that, include: S1: Acquire strawberry images and information about the picking robotic arm; S2: Extract visual feature vectors and strawberry geometric information vectors from the strawberry image; And extract the feature vector of the picking robot arm body from the information of the picking robot arm; wherein, the strawberry geometric information vector includes the coordinates of the strawberry center point and the growth direction vector of the strawberry stem; S3: Using the strawberry center point coordinates and the strawberry stem growth direction vector as strong priors, perform feature fusion on the visual feature vector, the strawberry geometric information vector, and the picking robot arm body feature vector to obtain the observation condition vector. S4: Add preset noise to the ideal picking trajectory through a diffusion strategy to generate a noisy picking trajectory; S5: Train the noise prediction network based on the noisy picking trajectory, the diffusion steps of the diffusion strategy, and the observation condition vector corresponding to the ideal picking trajectory; S6: Generate a noisy motion sequence with a preset step length, including the end-effector pose of the picking robot arm; S7: Input the observation condition vector, the noisy action sequence, and the current diffusion step number into the trained noise prediction network, and output the current predicted noise; S8: Iteratively remove the current predicted noise from the noisy action sequence to obtain the target action sequence; S9: Control the picking robotic arm according to the target action sequence to pick strawberries.

2. The method for strawberry picking based on structured geometry guidance and diffusion strategy according to claim 1, characterized in that, The information of the harvesting robotic arm includes the joint angles of the harvesting robotic arm and the end-effector pose of the harvesting robotic arm.

3. The method for strawberry picking based on structured geometry guidance and diffusion strategy according to claim 1, characterized in that, S2 specifically includes: S201: Extract the visual feature vector through the image semantic coding module, wherein the image semantic coding module includes a ResNet-18 image semantic coding module and a ViT image semantic coding module; S202: Extract the coordinates of the strawberry center point and the strawberry stem growth direction vector using the YOLO v8 target geometry analysis module; S203: Normalize the coordinates of the strawberry center point and the strawberry stem growth direction vector respectively; S204: The normalized strawberry center point coordinates and strawberry stem growth direction vector are mapped to the strawberry geometric information vector by sine-cosine position encoding; S205: The information of the harvesting robot arm is mapped into the body feature vector of the harvesting robot arm through the ontology state mapping module based on the multilayer perceptron (MLP) network.

4. The strawberry picking method based on structured geometry guidance and diffusion strategy according to claim 1, characterized in that, Specifically, S3 is: Using the strawberry center point coordinates and the strawberry stem growth direction vector as strong priors, the visual feature vector, the strawberry geometric information vector, and the picking robot arm body feature vector are fused through a cross-modal attention mechanism or a multilayer perceptron to obtain the observation condition vector.

5. The method for picking strawberries based on structured geometric guidance and diffusion strategy according to claim 1, characterized in that, S4 specifically includes: S401: Obtain an ideal picking trajectory that includes the entire picking process, wherein the entire picking process includes an approach process, a bending process, and a separation process; S402: Add preset noise to the ideal picking trajectory through a diffusion strategy to obtain the noisy picking trajectory.

6. The method for picking strawberries based on structured geometric guidance and diffusion strategy according to claim 5, characterized in that, Specifically, S5 is: Using the noisy picking trajectory, the diffusion steps of the diffusion strategy, and the observation condition vector as samples, and the preset noise as labels, the noise prediction network is trained until the loss value between the preset noise and the predicted noise is less than the preset loss value.

7. The method for picking strawberries based on structured geometric guidance and diffusion strategy according to claim 1, characterized in that, The noise prediction network is specifically a noise prediction network based on the Transformer architecture; The noise prediction network includes an encoder and a decoder, wherein the decoder includes a first multi-head self-attention module and a multi-head cross-attention module interconnected with each other; the encoder includes a second multi-head self-attention module connected to the multi-head cross-attention module; Wherein, the query input of the multi-head cross-attention module is the output of the first multi-head self-attention module; the key input and value input of the multi-head cross-attention module are both the outputs of the second multi-head self-attention module; The input to the first multi-head self-attention module is the noisy action sequence, and the input to the second multi-head self-attention module is the observation condition vector.

8. The method for strawberry picking based on structured geometry guidance and diffusion strategy according to claim 1, characterized in that, The end-effector pose of the harvesting robotic arm includes the end-effector displacement increment and the end-effector pose rotation increment. S6 specifically includes: S601: Determine the preset step size and the motion dimension corresponding to each step; S602: Generate an initial random noise sequence with the same dimension as the action sequence to be generated based on a standard Gaussian distribution, and use the initial random noise sequence as the initial state of the noisy action sequence.

9. The method for strawberry picking based on structured geometric guidance and diffusion strategy according to claim 1, characterized in that, S8 specifically includes: S801: Based on the current predicted noise, the noisy action sequence corresponding to the current diffusion step number is denoised and updated to obtain a candidate noisy action sequence; The specific formula for generating candidate noisy action sequences is as follows: ； in, Represents the number of diffusion steps i The relevant preset ideal picking trajectory scaling factor, Represents the number of diffusion steps i -1 related candidate noisy action sequences Represents the number of diffusion steps i Related noisy action sequences, Represents the number of diffusion steps i The relevant backsampling noise standard deviation, This represents random noise that follows a standard normal distribution. S802: Decrease the current diffusion step count by 1, and use the candidate noisy action sequence as the new noisy action sequence. Determine whether the current diffusion step count is greater than zero. If yes, return to step S7; otherwise, output the candidate noisy action sequence as the target action sequence.

10. A strawberry picking system based on a structured geometry guidance and diffusion strategy, characterized in that, include: processor; A memory storing computer-readable instructions, which, when executed by the processor, implement the strawberry picking method based on a structured geometry guidance and diffusion strategy as described in any one of claims 1 to 9.

Images