A non-contact tomato picking control method based on a visual-linguistic-action model and a control system thereof

Abstract

The application provides a non-contact tomato picking control method and control system based on a vision-language-action model, comprising the following steps: collecting an RGB-D image and a natural language instruction, and extracting visual and semantic features through ViT-Large and T5-Large respectively; detecting fruit and calyx key points using CenterNet, and combining a depth map to restore a three-dimensional picking point and direction; taking language features as a query and visual features as a key value, and fusing the body perception information through cross-attention to construct a state vector; outputting an action instruction using a multi-layer perception machine strategy network; and designing a reward function containing a high-weight collision penalty, and combining two-stage training of behavior cloning and PPO. The application realizes understanding of the natural language instruction and collision-free accurate approach, significantly reduces the branch and leaf collision rate, and improves the picking success rate and sim2real migration efficiency.

Description

Technical Field

[0001] This invention relates to the field of agricultural robots, and in particular to a non-contact tomato harvesting control method and its control system based on a vision-language-action model. Background Technology

[0002] Tomatoes are one of the most widely planted vegetable crops in my country's facility agriculture. Harvesting is labor-intensive and highly seasonal, accounting for over 30% of the total labor demand throughout the production cycle. With the continued transfer of rural labor to non-agricultural industries, the labor shortage in tomato harvesting is becoming increasingly prominent, making the development of intelligent harvesting robots an urgent need for industry development.

[0003] Currently, there are numerous research findings on tomato-harvesting robots both domestically and internationally. In terms of visual perception, most methods employ instance segmentation or object detection networks based on RGB images to identify the fruit, and then obtain its 3D position through stereo matching or depth cameras. Regarding control strategies, common approaches include using pure visual servoing or generating motion trajectories based on traditional planning algorithms (such as RRT and PRM). However, existing technologies still face the following prominent problems in practical applications:

[0004] First, there is a lack of ability to parse high-level semantic instructions. Existing harvesting robots can typically only execute predefined, simple instructions and cannot understand natural language descriptions such as "pick the reddest tomatoes" or "pick the less obscured fruits on the right side first." This makes it difficult for the robots to adapt to the personalized needs of different users and to make intelligent decisions based on semantic information in complex scenarios. Although visual-language-action (VLA) models have made some progress in indoor grasping tasks in recent years, existing models are mainly designed for desktop scenarios and do not consider the special characteristics of agricultural environments, such as dense foliage and partial occlusion of the target. Direct application of these models leads to a serious disconnect between semantic understanding and physical constraints.

[0005] Second, the 3D positioning accuracy is insufficient and it is not integrated with end-to-end motion control. Most existing harvesting methods adopt a step-by-step architecture of detection first and planning later: first, the 3D coordinates are obtained by fruit detection, and then an independent motion planner is called to generate a path. This step-by-step processing has the problem of error accumulation and cannot adjust the strategy according to real-time feedback during the approach process. Especially in scenarios where branches and leaves cover the fruit, it is difficult to determine the optimal approach direction by simply relying on the center position of the fruit, which can easily cause the robotic arm to collide with the fruit stem or leaves.

[0006] Third, the high collision rate of branches and leaves during the approach process damages the plant. Most existing harvesting control methods take reaching the target location as the sole optimization objective, lacking explicit constraints on non-contact behaviors. In actual harvesting operations, the branches and leaves of tomato plants are relatively fragile, and frequent collisions during the robotic arm's approach can lead to fruit damage, plant infection, or even the death of the entire plant. A few studies have attempted to incorporate obstacle avoidance constraints into the planning process, but due to the high-dimensional state space and unstructured environmental characteristics, it is difficult to balance real-time performance and success rate.

[0007] Fourth, the transfer from simulation training to real-world deployment is difficult. The agricultural environment is highly uncertain, with drastic variations in factors such as light, background, and plant morphology. Existing simulation platforms have limited modeling accuracy for plant models and lack coverage of the diversity of physical parameters, leading to a significant performance drop when strategies trained in simulation environments are transferred to real robots. Effective transfer often requires extensive data collection and retraining in real-world environments, which is time-consuming and labor-intensive.

[0008] Therefore, there is an urgent need for a picking control method that can integrate vision, language and motion to achieve safe, non-contact approach while ensuring accurate positioning, and to have efficient transfer capabilities from simulation to reality. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a non-contact tomato harvesting control method and its control system based on a vision-language-action model. By integrating RGB-D visual perception with natural language command understanding, and combining three-dimensional harvesting point estimation and PPO end-to-end training, it achieves collaborative non-contact proximity control between a wheeled chassis and a robotic arm. The simulation proximity success rate reaches 92.3%, effectively solving the technical problems of high branch and leaf collision rate, insufficient positioning accuracy, and lack of semantic understanding ability in existing harvesting technologies.

[0010] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0011] A non-contact tomato harvesting control method based on a vision-language-action model includes the following steps:

[0012] RGB-D images are acquired by an RGB-D depth camera installed at the end of the robotic arm, natural language picking instructions are received through a human-machine interface, and robot body perception information is obtained through an encoder. The body perception information includes the joint angles, joint speeds, and chassis status of the robotic arm.

[0013] The RGB-D image is input into the visual encoder to extract the visual feature vector, and the natural language instruction is input into the language encoder to extract the semantic feature vector.

[0014] RGB-D images are input into a 3D picking point estimation network to detect 2D key points at the center of the tomato fruit and the center of the calyx. The 3D coordinates are recovered by combining the depth image, and the picking direction vector and picking point position are calculated.

[0015] Using semantic feature vectors as queries and visual feature vectors as keys and values, a state vector is constructed by fusing 3D position embedding and ontology-aware features through a cross-attention mechanism.

[0016] The state vector is input into the policy network, and the parameters of the action distribution are output. The action is sampled and obtained. The action includes the chassis speed command and the target joint angle of the robotic arm.

[0017] The robot moves its chassis and robotic arm according to motion commands, monitors its contact with branches and leaves in real time through collision detection, and determines that it is close to success when the end effector reaches the target picking point.

[0018] Furthermore, the visual encoder adopts the ViT-Large architecture, which divides the input RGB image into multiple image blocks of a preset size, and then inputs them into a multi-layer Transformer encoder after linear embedding and positional encoding; the depth information adopts an early fusion strategy, which concatenates the RGB image and the depth image in the channel dimension to form an RGB-D multi-channel input, which is then mapped to a three-channel feature map through a convolutional layer before being input into ViT-Large.

[0019] The language encoder adopts a T5-Large architecture, takes the mean pooling result of the last hidden state of the encoder as the semantic feature vector, and adopts an instruction randomization strategy during training to generate multiple natural language instructions with different expressions for the same operation target.

[0020] Furthermore, the 3D picking point estimation network is based on CenterNet architecture, using EfficientNet-B3 as the backbone network, and embedding a CBAM attention module after the backbone network to enhance attention to occluded targets; the detection head includes a center point heatmap branch, a size regression branch, and an offset regression branch, which output the two-dimensional coordinates (u) of the fruit center and the calyx center, respectively. f ,v f ) and (u c ,v c The three-dimensional coordinate recovery uses the following formula:

[0021] ,

[0022] In the formula, and , respectively, represent the three-dimensional coordinates of the fruit center and calyx center in the camera coordinate system; K is the camera intrinsic parameter matrix; d f d represents the depth value at the center pixel of the fruit.c This represents the depth value at the center pixel position of the calyx.

[0023] Through hand-eye transformation matrix Transform the center of the fruit and the center of the calyx into the robot's base coordinate system to obtain... and ;

[0024] Picking direction vector n pick Calculate using the following formula: ,

[0025] Picking point location p pick Calculate using the following formula: ,

[0026] In the formula: Represents the Euclidean norm; r f The radius of the fruit is determined by half the width of the bounding box output by the size regression branch.

[0027] Furthermore, the cross-attention mechanism calculates the fused features using the following formula:

[0028] ,

[0029] In the formula: z l z is a semantic feature vector; v W is a visual feature vector. Q To query the projection matrix; W K W is the key projection matrix; V The projection matrix is ​​d; k For attention head dimension;

[0030] The fused features are then processed by layer normalization and a feedforward network to obtain the multimodal vector z. vl The three-dimensional position embedding e pos The coordinates p of the picking point output by the 3D picking point estimation network pick The ontology-aware feature z is obtained through encoding via a fully connected layer. prop This includes the robotic arm joint angles, joint speeds, and chassis status;

[0031] The multimodal vector z vl 3D position embedding vector e pos and ontology-aware feature vector z prop By concatenating along the channel dimension, a complete state vector is formed: .

[0032] Furthermore, the policy network adopts a multilayer perceptron architecture, including multiple hidden layers, each with a preset number of neurons, and the activation function is ReLU; the policy network outputs a Gaussian distribution with a mean μ. θ(s) and log-standard deviation logσ θ (s), the action is obtained by sampling using the reparameterization technique:

[0033] ,

[0034] in Let I be the standard normally distributed noise, θ be the identity matrix, and θ be the trainable parameters of the policy network. Action a includes chassis speed commands and target joint angles of the robotic arm.

[0035] Furthermore, training is guided by a reward function derived from a non-contact approach strategy, which is expressed as:

[0036] ,

[0037] Among them, the distance reward r d The reward is a negative exponential function of the distance between the end effector and the target pickup point; the closer the distance, the greater the reward. Orientation reward r o The end effector orientation is a square function of the direction of picking; collision penalty r c A negative value is assigned when contact is detected, otherwise zero; a successful reward of r is given. s A positive reward is given when the end effector reaches the vicinity of the target picking point and the orientation deviation is less than a preset threshold; otherwise, the reward is zero. A time penalty r is applied. t The negative constant applied at each time step; the weight w of each reward item. d w o w c w s w t All values ​​are preset positive values, with the collision penalty weight set to the highest value among all values.

[0038] Furthermore, the policy network adopts a two-stage training method: the first stage is behavior cloning pre-training, which uses a collision-free motion planning algorithm to generate multiple expert trajectories, and supervises learning by minimizing the mean square error loss between the policy network output and the expert actions;

[0039] The second stage involves fine-tuning and optimizing the near-end strategy, with the objective function being:

[0040] ,

[0041] ,

[0042] in, This is the ratio of the probability densities of the new and old strategies; Indicates the policy network in state s t Output action a t The probability density; This indicates that the old policy network in the previous update is in state s. t-1 Output action a t-1 The probability density; For the expected experience at time steps; A t For generalized advantage estimation; This is the clipping function; This is the cutting factor.

[0043] Furthermore, it also includes transferring the policies trained in the simulation environment to the real robot platform. The transfer methods include zero-shot transfer or fine-tuning transfer. When it is fine-tuning transfer, a small number of trajectories are collected in the real environment for parameter fine-tuning. The parameters of the visual encoder and language encoder are kept fixed, and only the cross-attention projection matrix and policy network parameters are updated. The learning rate is set to the preset multiple during simulation training.

[0044] A control system for a non-contact tomato harvesting control method based on a vision-language-action model includes:

[0045] The data acquisition module is used to control the RGB-D depth camera installed at the end of the robotic arm to acquire RGB-D images, receive natural language picking instructions through the human-machine interface, and obtain robot body perception information through the encoder;

[0046] The 3D picking point estimation module is used to detect 2D key points at the center of the tomato fruit and the center of the calyx from RGB-D images, recover 3D coordinates by combining depth images, and calculate the picking direction vector and picking point position.

[0047] The visual encoding module is used to input RGB-D images into the visual encoder to extract visual feature vectors;

[0048] The language encoding module is used to input natural language instructions into the language encoder to extract semantic feature vectors;

[0049] The multimodal fusion module is used to construct a state vector by fusing 3D position embedding and ontology-aware features through a cross-attention mechanism, using semantic feature vectors as queries and visual feature vectors as keys and values.

[0050] The motion generation module is used to input the state vector into the policy network, output the parameters of the motion distribution, and sample the motion including the chassis speed command and the target joint angle of the robotic arm.

[0051] The non-contact proximity execution module is used to control the movement of the chassis and robotic arm according to the action command. It monitors the contact with branches and leaves in real time through collision detection and determines that the approach is successful when the end effector reaches the target picking point.

[0052] The beneficial effects of this invention are as follows:

[0053] 1. The non-contact tomato picking control method based on a vision-language-action model described in this invention combines a ViT-Large visual encoder with a T5-Large language encoder and employs a cross-attention mechanism to achieve deep fusion of visual and linguistic features. This enables the robot to understand complex natural language commands such as "pick the reddest tomato" and guide visual attention based on semantic information, achieving semantically adaptive picking in complex scenarios. Experiments show that compared to the pure vision-based PPO baseline method, the success rate of this invention in complex scenarios is improved from 83% to 92%, an improvement of 9 percentage points.

[0054] 2. The non-contact tomato harvesting control method based on a vision-language-action model described in this invention constructs a fruit-calyx association detection network based on CenterNet, recovers the three-dimensional coordinates of the fruit center and calyx using depth images, and uses a unit vector pointing from the calyx to the fruit as the harvesting direction. Then, a fruit radius is extrapolated along this direction to determine the harvesting point, achieving precise three-dimensional positioning and optimal approach direction estimation for stemmed tomato fruits. Experimental results show that the root mean square error of fruit center positioning is 8.3 mm, and the harvesting direction angle error is 9.7°, meeting the engineering precision requirements for accurate harvesting.

[0055] 3. The non-contact tomato harvesting control method based on a vision-language-action model described in this invention explicitly introduces a collision penalty term into the reward function and assigns it the highest weight. Simultaneously, it combines distance reward, orientation reward, success reward, and time penalty, enabling the policy network to prioritize learning contact-avoidance behavior patterns during PPO reinforcement learning, thus achieving precise non-contact approach. Simulation tests show that after adopting the reward function of this invention, the branch and leaf collision rate is only 4.7%, significantly lower than the 12.0% baseline of pure vision PPO, greatly reducing mechanical damage to the plant.

[0056] 4. The non-contact tomato harvesting control method based on a vision-language-action model described in this invention constructs a high-fidelity parametric tomato plant simulation environment based on IsaacLab and performs comprehensive domain randomization on three aspects: vision (lighting, texture, camera noise), physics (mass, friction, elasticity), and dynamics (motor delay, joint friction, sensor noise), achieving efficient strategy transfer from simulation to reality. After fine-tuning with only 100 real trajectories, the approach success rate on a real robot platform reaches 86.5%, and the collision rate is reduced to 7.5%, significantly better than the 72.5% success rate and 15.0% collision rate of zero-sample transfer, verifying the practicality and generalization ability of the method of this invention.

[0057] 5. The non-contact tomato harvesting control method based on a vision-language-action model described in this invention employs a two-stage training framework of behavior cloning pre-training and PPO reinforcement learning fine-tuning. First, it uses collision-free expert trajectories generated by RRT-Connect for supervised learning to obtain an initial policy, and then combines this with an online optimization function, achieving a balance between training efficiency and final performance. This training strategy enables the policy network to converge within approximately 2 × 10^7 steps (8 hours), which is about 3 times faster than pure reinforcement learning training from scratch. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings described below are some embodiments of the present invention. For those skilled in the art, it is obvious that other drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 This is a diagram of the architecture of the VLA non-contact tomato harvesting control system described in this invention.

[0060] Figure 2 This is a schematic diagram illustrating the working principle of the cross-attention fusion module described in this invention. Detailed Implementation

[0061] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0062] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "axial," "radial," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0063] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0064] This invention provides a non-contact tomato harvesting control method based on a Vision-Language-Action (VLA) model, implemented by deploying it on a real tomato harvesting robot system. The robot system includes: a 7-DOF collaborative robotic arm (e.g., Franka Emika Panda, 855mm reach) mounted on a wheeled differential chassis (e.g., Clearpath Husky); an RGB-D depth camera (e.g., Intel RealSense D435i, 640×480 resolution, 30fps) mounted at the end of the robotic arm; and a high-performance embedded computing platform (e.g., NVIDIA Jetson AGX Orin, 64GB RAM) serving as an edge computing unit. All software modules utilize the Robot Operating System (ROS2) to achieve communication and unified timing control between distributed nodes. Figure 1 The system architecture diagram shown illustrates the data interaction relationships between the modules.

[0065] The tomato harvesting control method based on the VLA model described in this invention includes the following steps:

[0066] S01: Establish a high-fidelity virtual simulation environment for tomato plants, as detailed below:

[0067] Because the morphology, light, and physical characteristics of tomato plants vary greatly in real agricultural environments, and because direct reinforcement learning training on real robots is costly and risky, this embodiment first constructs a training platform in a simulation environment. Specifically, the NVIDIA Isaac Sim platform and its IsaacLab extension module are used to establish a high-fidelity virtual simulation environment for tomato plants.

[0068] The plant models are generated parametrically to simulate tomato plant shapes at different growth stages. The trunk model is a parametric cylinder with a height randomly set between 1.2 meters and 1.8 meters, and a diameter between 2 centimeters and 4 centimeters. The branching models are generated from the trunk, with lengths ranging from 10 centimeters to 30 centimeters. The leaf models are constructed based on a 3D scanned mesh of real tomato leaves, with each plant containing more than 50 leaves, and the area of ​​a single leaf ranging from 50 square centimeters to 200 square centimeters. The fruit models use ellipsoids to simulate two common types: cherry tomatoes (2-3 centimeters in diameter) and regular tomatoes (5-8 centimeters in diameter).

[0069] To narrow the gap between simulation and real-world environments, this embodiment introduces a comprehensive domain randomization strategy during simulation training, specifically including:

[0070] Visual randomization: The light intensity is randomly varied between 500 lux and 50,000 lux, and the color temperature is randomly set between 3,000 K and 6,500 K; the color of the fruit is continuously randomized between green and dark red; at the same time, 10 different leaf textures are randomly changed, the background image is changed, and random perturbations are added to camera parameters (such as exposure time and gain).

[0071] Physical randomization: The quality of the fruit varies randomly within ±30% of the nominal value, the coefficient of friction between the components varies randomly within ±20% of the nominal value, and the elastic modulus varies randomly within ±25% of the nominal value.

[0072] Dynamic randomization: The motor response delay of the chassis is randomly set between 0 and 50 milliseconds, the friction force of each joint of the robotic arm varies randomly within ±40% of the nominal value, and the sensor measurement noise is set to a position standard deviation of 1 mm and an angle standard deviation of 0.5 degrees.

[0073] Through the randomization process described above, the policy network is trained in a highly diverse environment, thereby learning feature representations that are not robust to various parameter changes, laying the foundation for efficient transfer from simulation to reality.

[0074] S02: Construct a deep learning-based 3D pick-up point estimation network, as follows:

[0075] This deep learning-based 3D picking point estimation network uses CenterNet as its base architecture and EfficientNet-B3 as its backbone to extract multi-scale features. To enhance the response capability to key regions of tomato fruit and calyx in scenarios with foliage occlusion, a Convolutional Block Attention Module (CBAM) attention mechanism is introduced after the backbone network. The CBAM attention mechanism includes channel attention sub-branch and spatial attention sub-branch, which can adaptively recalibrate the importance weights of each channel and position in the feature map.

[0076] The network's detection head has three parallel output branches: the first branch is a center point heatmap branch, which uses a fully convolutional structure to output a heatmap proportional to the input image size, where each pixel value represents the probability of the fruit center or calyx center being located at that position; the second branch is a size regression branch, which estimates the width and height of the target bounding box through convolutional layers; and the third branch is an offset regression branch, used to compensate for discretization coordinate errors introduced by multiple downsampling operations. The network is pre-trained under supervised supervision on a labeled tomato image dataset. The loss function is a weighted sum of the loss terms of each branch, with the center point heatmap branch using focal loss and the size and offset branches using L1 loss.

[0077] In practical applications (whether in simulation or real-world environments), a 224×224×3 RGB image captured by an RGB-D camera is input into the network. After forward propagation, the two-dimensional pixel coordinates (u) of the fruit's center are obtained. f ,v f ) and the two-dimensional pixel coordinates of the calyx center (u c ,v c Then, the depth values ​​d at these two pixel locations are read from the corresponding depth image. f and d c Using the pre-calibrated camera intrinsic parameter matrix K, the two-dimensional pixel coordinates are back-projected to three-dimensional coordinates in the camera coordinate system using the following formula:

[0078] ;

[0079] In the formula, and , respectively, are the three-dimensional coordinates of the fruit center and the calyx center in the camera coordinate system; K is a 3×3 intrinsic parameter matrix, and its inverse matrix K −1 Used to convert homogeneous pixel coordinates to normalized camera coordinates.

[0080] Subsequently, the hand-eye transformation matrix obtained through prior calibration was used. (A 4×4 SE(3) group element, including rotation and translation components), transform these two points to the robot base coordinate system, denoted as and .

[0081] Based on the above three-dimensional coordinate information, the picking direction vector n pick Defined as a unit vector pointing from the center of the calyx to the center of the fruit, it is calculated using the following formula:

[0082] ;

[0083] in, This represents the Euclidean norm.

[0084] The picking point location p used to guide the robotic arm. pick Set at a position on the fruit surface, offset outwards by one fruit radius along the harvesting direction: ;

[0085] Here, r f The fruit radius estimate is output from the size regression branch; in actual calculations, it is usually taken as half the width of the bounding box.

[0086] Through the above design, the deep learning-based 3D picking point estimation network can directly output the 3D picking point coordinates and approach direction from a single RGB-D image, which can be used by downstream control strategies, providing accurate spatial targets for subsequent non-contact proximity control.

[0087] S03: To achieve joint understanding and decision-making of visual images, natural language commands, and robot body state information, a multimodal VLA policy model is constructed. This multimodal VLA policy model consists of four parts: a visual encoding network, a language encoding network, a cross-modal feature fusion mechanism, and an action generation network, as detailed below:

[0088] 1. Visual coding networks

[0089] The visual encoding network employs the ViT-Large (Vision Transformer Large) architecture. Taking an input RGB image of size 224×224×3 as an example, the image is first divided into 16×16 pixel blocks, resulting in 196 blocks. Each block is converted into an embedding vector through a linear mapping layer, and a learnable positional encoding is superimposed on it to preserve the spatial location information of each block in the original image. These 196 patch tokens, along with an additional class token, constitute the input sequence, which is fed into a network consisting of 24 stacked Transformer encoder layers, each containing a multi-head self-attention mechanism and a feedforward network.

[0090] To fully utilize depth information, this embodiment employs an early fusion strategy: the original RGB image and the depth image are directly concatenated along the channel dimension to form a 224×224×4 four-channel RGB-D tensor. The RGB-D tensor is first processed through a convolutional layer with a 3×3 kernel, a stride of 2, and 3 output channels, thus mapping the four-channel input to a three-channel feature map. This three-channel feature map is then input into the ViT-Large network. The final output of the visual encoding network is a feature tensor z with dimensions of 197×1024. v 197 corresponds to 196 image patch tokens plus 1 classification token, and 1024 is the feature dimension of each token. The two-dimensional feature tensor zv is denoted as the visual feature.

[0091] 2. Language coding network

[0092] The language encoding network employs the T5-Large (Text-to-Text Transfer Transformer Large) model. The operator inputs natural language commands through a human-computer interaction interface, such as "pick the reddest tomato from the plant on the right front." This command string is first processed by a tokenizer, converting it into a sequence of words, and then fed into the T5 encoder. The mean pooling result of the last hidden state of the T5 encoder is used as the semantic vector representation z of the entire command sentence. l Its dimension is 1024. To enhance the model's adaptability to different language expressions, this embodiment introduces an instruction randomization strategy during training: for the same target tomato fruit, multiple different wordings of instruction expressions are predefined (such as "pick the red one," "pick the red tomato on the right," "pick the red fruit in the right front," etc.), and one of them is randomly selected to be paired with the visual scene in each training iteration, so that the model learns the essential semantics of language instructions without overly relying on specific word combinations. Semantic vector zl These are denoted as linguistic features.

[0093] 3. Cross-modal feature fusion mechanism

[0094] To achieve guided weighted aggregation of visual features based on linguistic information, this embodiment designs a feature fusion mechanism based on cross-attention, such as... Figure 2 As shown. It should be noted that the visual feature zv is a sequence containing 197 locations (196 image patches plus 1 classification token), with each location having a feature dimension of 1024; while the language feature zl is a single 1024-dimensional vector. To enable cross-attention calculation between the two, this embodiment treats the language feature vector zl as a query sequence of length 1, reshaping its shape from 1024 to 1×1024. Subsequently, attention is calculated using this query sequence and the visual feature sequence zv (as key and value), with the specific calculation formula as follows:

[0095]

[0096] In the formula, CrossAttn is the cross-attention function, which realizes cross-modal fusion of linguistic and visual features; W Q To query the projection matrix, dimension 1024×64; W K W is the key projection matrix, with dimensions 1024×64; V The projection matrix is ​​d, with dimensions 1024×64. k For the attention head dimension, the example takes a value of 64; The scaling factor is used to prevent the dot product result from becoming too large and causing softmax gradient saturation; softmax(⋅) is a normalized exponential function used to convert the attention score into a probability distribution form.

[0097] After the above calculations, a weighted aggregation result of 1×1024 is output, which is the language-guided visual summarization vector. This language-guided visual summarization vector is then passed through layer normalization operations and a feedforward network (consisting of two fully connected layers, with a middle layer dimension of 4096 and an output layer dimension of 1024) to finally obtain the cross-modal fusion multimodal vector. .

[0098] To enhance the model's ability to perceive three-dimensional spatial location, this embodiment introduces spatial location information during the aforementioned fusion process. Specifically:

[0099] The three-dimensional coordinates p of the target picking point output by the aforementioned three-dimensional picking point estimation network are... pickThe expression (x, y, z) is encoded using a learnable positional encoder, which is a fully connected layer with 1024 output neurons. The input is three-dimensional coordinates, and the output is a positional embedding vector. Meanwhile, the robot encoder acquires 20 dimensions of proprioceptive information, including angle values ​​for 7 joints, velocity values ​​for 7 joints, and 6 chassis state variables. This proprioceptive information is mapped to a proprioceptive feature vector of the same dimension through another fully connected layer. Finally, the multimodal vector z vl 3D position embedding vector e pos and ontology-aware feature vector z prop By concatenating along the channel dimension, a complete state vector is formed: The state vector s has a dimension of 3072 and contains the visual semantics, spatial target and self-state information required for the current task.

[0100] 4. Action Generation Network

[0101] The action generation network employs a multilayer perceptron (MLP) architecture. Its input is a state vector *s*, and its output is a Gaussian distribution of parameters in the action space. The network contains three hidden layers, each with 256 neurons, and uses the ReLU activation function. The output layer has two branches: the first branch outputs the mean vector *μ*. θ (s), the second branch outputs the log-standard deviation vector logσ. θ (s), where θ represents all trainable parameters of the action generation network. The action vector a is a 10-dimensional continuous vector, with the first 3 dimensions used to control the linear velocity (v) of the wheeled chassis. x ,v y The motion vector a is calculated using the motion vector ω and angular velocity ω, with the latter seven dimensions used to control the target angles of the seven joints of the robotic arm. During the training phase, the motion vector a is sampled from a Gaussian distribution using a reparameterization technique.

[0102]

[0103] In the formula: a is the action vector obtained from sampling; It follows a multivariate Gaussian (normal) distribution; μ θ (s) is the mean vector; logσ θ (s) is the log-standard deviation vector; θ represents all trainable parameters of the action generation network; Construct a function for a diagonal matrix, converting the logarithmic standard deviation into a diagonal covariance matrix; It is the square of the standard deviation (variance), and the logarithm of the standard deviation ensures that it is positive.

[0104] In actual sampling, reparameterized equivalent calculations are used: ,in This represents standard normally distributed noise. During testing or actual deployment, to improve determinism, the mean action a = μ is typically taken directly. θ (s) is used as an output instruction.

[0105] This embodiment constructs an end-to-end VLA strategy model that can directly generate underlying robot motion control commands from raw visual input and natural language instructions.

[0106] S04: Determination of the non-contact proximity reward function.

[0107] To guide the action generation network in learning collision-free and precise approach behaviors, this embodiment designs a non-contact approach reward function. The non-contact approach reward function is calculated once at each time step when the robot performs an approach action, and is determined by a distance reward r. d Orientation towards reward r o Collision penalty r c Success reward r s and time penalty r t It consists of five parts, and its weighted sum is as follows:

[0108] award ,

[0109] The following details the distance reward r. d Orientation towards reward r o Collision penalty r c Success reward r s and time penalty r t Determination:

[0110] Distance reward r d This is used to encourage the robotic arm's end effector to quickly approach the target picking point. Distance reward r d Defined as the current position p of the end effector ee With the target picking point p pick The negative exponential function of the Euclidean distance between them:

[0111] ,

[0112] in, σ represents the Euclidean norm; d The attenuation coefficient is 0.1m in this example. The negative exponential function reward approaches 0 when the end point is far from the target. When the distance between the end point and the target is less than 0.1m, the reward value rapidly increases to close to 1, thereby guiding the strategy to prioritize shortening the spatial distance.

[0113] w d The weight of the distance reward is set to 1.0 in this embodiment.

[0114] Towards reward r o This is used to encourage the end effector to align with the picking direction. Let n ee Let n be the orientation unit vector of the end effector. pick Given the unit vector for the picking direction calculated in step two, the orientation reward is defined as follows:

[0115] ,

[0116] When n ee With n pick When the two directions are completely aligned, the dot product is 1, and the reward reaches its maximum value of 1; when the angle between the two is greater than 90 degrees, the dot product is negative, and the reward is zero. The truncation is set to 0; the squaring process makes the reward value increase more rapidly when the included angle is small.

[0117] Wo represents the orientation reward weight, which is set to 0.5 in the example.

[0118] Collision penalty r c The most critical design element for achieving non-contact approach is the collision detection mechanism (implemented in simulation by detecting the penetration between the robotic arm mesh and the plant branch mesh; in a real robot, this can be achieved using joint torque-based collision detection or flexible tactile sensors) when it detects contact between any part of the robotic arm and the plant branches. c =−1.0; otherwise r c =0. To ensure the strategy prioritizes collision avoidance, this embodiment assigns the highest weight to the collision penalty, setting the collision penalty weight w. c =2.0. This weighting setting makes it worthwhile to sacrifice some approach speed or heading accuracy in order to bypass obstacles.

[0119] Success reward r s This serves as a sparse but strong positive reward signal, used to explicitly inform the strategy that the task has been completed. When the robotic arm's end effector is close to the target picking point p... pick When the Euclidean distance is less than 0.02 meters and the angle between the tip and the picking direction is less than 15 degrees, it is considered a successful approach. s =+10.0; otherwise r s =0. Success reward weight w s Set to 1.0.

[0120] Time penalty r t This strategy encourages completing tasks in as few steps as possible, avoiding unnecessary wandering or detours. A small negative reward is applied at each time step: r t =−0.05. Time penalty weight w t Set to 1.0.

[0121] The non-contact proximity reward function is used in the subsequent reinforcement learning training phase (step five) to calculate the immediate reward at each time step. The policy network learns the optimal proximity behavior by maximizing the cumulative discounted reward.

[0122] S05: Two-stage policy training. The policy network is trained in the domain-randomized simulation environment constructed in S01. A two-stage training framework is adopted, combining behavioral cloning (BC) pre-training with proximal policy optimization (PPO) reinforcement learning fine-tuning, as detailed below:

[0123] Phase 1: Behavioral Cloning Pre-training

[0124] Traditional collision-free motion planning algorithms were used to generate expert demonstration data in a simulation environment. This embodiment employed the RRT-Connect (Rapidly-exploring Random Tree Connect) algorithm, which, given complete map information in the simulation environment, can plan a collision-free path from a random starting pose, guiding the robotic arm's end effector from the starting point to the target picking point output in step two. By running this planning algorithm, this embodiment collected 5000 expert trajectories. Each trajectory includes a series of state-action pairs. , where state s i Constructed by the method in step S03 (including visual features, linguistic features, 3D location embedding, and ontological perception features), the expert in state s i Actions performed The control commands output by the RRT-Connect algorithm (10-dimensional, including chassis speed commands and robotic arm joint angle commands).

[0125] During the pre-training phase, the parameters θ of the action generation network are updated by minimizing the following mean squared error loss function:

[0126] ,

[0127] In the formula, L BC Here, μ represents the behavioral cloning loss function; N is the total number of samples in all expert trajectories, and μ... θ (si) represents the policy network in state s. i The average value output at the location; Indicates that the expert is in state s i The action performed forces the policy network to mimic the behavior of an expert, thus quickly acquiring an initial policy capable of avoiding basic collisions in the early stages of training. After behavior cloning training is completed, a policy network with preliminary approximation capabilities is obtained, which can accelerate the convergence of subsequent PPO reinforcement learning.

[0128] Phase Two: Fine-tuning of PPO reinforcement learning

[0129] Based on behavioral cloning pre-training, the PPO algorithm is used for online fine-tuning in a simulation environment. During the reinforcement learning phase, the state vector s at each discrete time step t... t and the generated action a t The definitions are completely consistent with those of the state vector s and action a in step three above (s t That is, the 3072-dimensional state vector constructed in step three, a t This refers to the 10-dimensional action command output in step three. The subscript 't' represents the time step number during the interaction process, which differs from the subscript 'i' used to enumerate expert samples in the first stage of behavior cloning. In this stage, the policy network determines the action command based on the current state 's'. t Output the parameters of the action distribution, and obtain action a through sampling. t And execute it; the environment returns an immediate reward R based on the reward function defined in step four. t At the same time, it transitions to the next state s t+1 The PPO algorithm further optimizes its strategy by collecting a large number of such interaction trajectories, with the goal of maximizing the cumulative discount reward. discount factor .

[0130] The core idea of ​​the PPO algorithm is to limit the magnitude of each policy update to avoid training instability. To this end, it employs a clipping mechanism to handle the importance sampling ratio. The objective function of PPO is:

[0131] ,

[0132] ,

[0133] In the formula: This is the ratio of the probability densities of the old and new policies, used to update the new policy using data sampled from the old policy; Indicates the policy network in state s t Output action a t The probability density; This indicates that the old policy network in the previous update is in state s. t-1 Output action a t-1 The probability density; For the expected experience at time steps; A t Generalized advantage estimation (GAE) is used to measure the advantage in state s. t Choose action a t Its degree of good or bad relative to the average level depends on the estimation of state value by the value network (trained in parallel with the policy network); For the pruning function, the importance sampling ratio is limited to... Within the range, cutting factor To prevent the policy update from being too large and causing training instability; the min() operation takes the smaller value between the original target and the pruned target, forming a pessimistic estimate, thereby avoiding overly aggressive policy updates.

[0134] The PPO hyperparameter settings used in this embodiment are as follows: the policy network learning rate is 3×10. −4 The value network learning rate is 1×10. −3 Both training methods used the Adam optimizer; the discount factor γ = 0.99, and the GAE parameter λ = 0.95; each training epoch was iterated 10 times, with a batch size of 2048 (i.e., collecting 2048 steps of interaction data from a parallel environment at once) and a mini-batch size of 64 (i.e., using 64 samples per gradient update); the entropy regularization coefficient was set to 0.01 to encourage the policy to maintain moderate exploratory behavior and prevent premature convergence to a suboptimal policy; 1024 parallel simulation environments were used, significantly improving sampling efficiency by leveraging the parallel computing power of the GPU; the total number of training steps was set to 2 × 10⁷. Training was conducted on a workstation equipped with an NVIDIA RTX 4090 GPU (24GB VRAM) and an Intel Core i9-13900K processor, taking approximately 8 hours.

[0135] After the two-stage training process described above, the action generation network learned to accurately approach the target picking point with an extremely low collision rate in a highly random tomato plant environment. Simulation test results showed that the approach success rate reached 92.3%, and the collision rate with branches and leaves was only 4.7%.

[0136] S06: Sim2real migration and real-world harvesting execution, details are as follows:

[0137] The VLA policy model (including visual encoding network, language encoding network, cross-attention fusion mechanism, and action generation network) trained in the simulation environment, along with the 3D picking point estimation network, was transferred to a real tomato picking robot platform. The real platform includes: a wheeled differential motion chassis (maximum linear velocity 1.0 m / s, maximum angular velocity 1.0 rad / s), a 7-DOF collaborative robotic arm (arm span 855 mm, load 3 kg), an Intel RealSense D435iRGB-D depth camera (640×480 resolution, 30fps) mounted at the end of the robotic arm, and an NVIDIA Jetson AGX Orin 64GB edge computing unit. All software nodes communicate in a distributed manner via ROS2 topics. The edge computing unit is responsible for running all neural network inference tasks and coordinating control commands between the chassis and the robotic arm through a unified clock synchronization mechanism.

[0138] This embodiment supports two methods for migrating strategies from simulation to reality:

[0139] (1) Zero-shot transfer: The model parameters obtained from simulation training are directly loaded onto the real robot without any additional training in the real environment. In actual operation, after the operator starts the system, the camera acquires images in real time, the 3D picking point estimation network outputs the target picking point, the VLA policy model generates action commands based on the current state and language instructions, and the robot performs the approach. The test results show that the approach success rate of zero-shot transfer is 72.5%, and the collision rate is 15.0%. The failure analysis shows that the main influencing factors include the difference between the lighting conditions in the real environment and the simulation environment, the measurement noise of the depth sensor under strong light, and the deviation between the physical flexibility of the real plant and the simulation model.

[0140] (2) Fine-tuning Transfer: To improve performance in real-world environments, this embodiment further employs a fine-tuning transfer strategy. Specifically, a small number of trajectories (e.g., 100) are collected on a real robot platform. Each trajectory includes an RGB-D image sequence, a language command, the executed action, and the final approximation result (success or failure). These real data are used to fine-tune the model. During fine-tuning, the parameters of the visual encoding network and the language encoding network are kept fixed; only the projection matrix in the cross-attention fusion mechanism and the parameters of the action generation network are updated to prevent overfitting under small sample sizes. The fine-tuning process uses a combination of behavioral cloning loss and a small number of PPO updates, reducing the learning rate to one-tenth of that used in simulation training (i.e., the policy network learning rate is set to 3×10). −5After fine-tuning, the success rate of approaching in real-world environments increased to 86.5%, while the collision rate decreased to 7.5%. Statistical analysis of failed cases showed that the main reasons for failure included: depth measurement errors under strong light (accounting for 35% of failures), differences in physical interaction caused by dense foliage (accounting for 30%), and fruits being completely obscured and unable to be detected by the 3D picking point estimation network (accounting for 20%).

[0141] like Figure 1 As shown, the actual harvesting process is as follows:

[0142] The system is powered on and initialized. The edge computing unit loads all model parameters and starts each sensor.

[0143] The operator inputs natural language commands via voice or text (e.g., "Pick the reddest tomato on the right front"). An RGB-D camera captures an image of the current field of view, and a 3D picking point estimation network processes the image to output the target picking point p. pick and picking direction n pick .

[0144] Visual coding networks extract visual features z from RGB-D images v The language coding network encodes instructions into semantic vectors z. l The two obtain the multimodal vector z through a cross-attention fusion mechanism. vl Then embed e with three-dimensional position pos and ontological perception features z prop The states are concatenated to form the state vector s.

[0145] The motion generation network outputs chassis speed commands and robotic arm joint angle commands based on 's'. The robot executes these commands and approaches the picking point. During the approach, a collision detection mechanism monitors in real time whether contact occurs. The robot terminates when the end effector successfully reaches a distance p. pick When the distance to the fruit is less than 0.02 meters and the orientation deviation is less than 15 degrees, the system determines that the fruit is close enough to be picked. The system then triggers the picking actuator (such as a vacuum suction cup or a three-finger gripper) to complete the fruit picking, and then transfers the fruit to the collection basket. The robotic arm returns to its initial position to prepare for the next picking. Throughout the picking cycle, the operator can switch targets or adjust task descriptions at any time using natural language commands.

[0146] Through the above process, the method described in this embodiment organically integrates simulation environment construction, 3D perception, multimodal fusion, non-contact optimization training, and real-world transfer into a complete technical solution, solving the prominent problems of existing tomato picking robots in terms of semantic understanding, positioning accuracy, and collision rate.

[0147] This embodiment also provides a non-contact tomato picking control system based on a vision-language-action model, including a data acquisition module, a three-dimensional picking point estimation module, a visual encoding module, a language encoding module, a multimodal fusion module, an action generation module, a simulation training and transfer module, and a non-contact proximity execution module;

[0148] The data acquisition module is used to control the RGB-D depth camera installed at the end of the robotic arm to acquire 224×224×4 RGB-D images, receive natural language picking instructions through the human-machine interface, and obtain robot body perception information through joint encoders and wheel encoders. The body perception information includes 7-dimensional robotic arm joint angles, 7-dimensional joint velocities, and 6-dimensional chassis status.

[0149] The 3D picking point estimation module is equipped with a 3D picking point estimation network based on the CenterNet architecture. It uses EfficientNet-B3 as the backbone network and introduces the CBAM attention mechanism to detect the two-dimensional key points of the center of the tomato fruit and the center of the calyx from the RGB image. It combines the depth image and the camera intrinsic parameter matrix to recover the three-dimensional coordinates of the two, and then calculates the picking direction vector and the picking point position.

[0150] The visual encoding module is equipped with a ViT-Large visual encoder, which is used to segment the RGB-D image into 16×16 image blocks and process them through a Transformer encoder to output a visual feature vector z. v .

[0151] The language encoding module is equipped with a T5-Large language encoder, used to encode natural language instructions into semantic feature vectors z. l And execute the instruction randomization strategy during training.

[0152] The multimodal fusion module is used to implement the cross-attention fusion mechanism, using language features z l As a query, visual feature z v As keys and values, cross-modal attention weights are calculated, and the fused features are embedded with the 3D location in e. pos and ontological perception features z prop Concatenate the data to construct the state vector s.

[0153] The action generation module employs a multilayer perceptron policy network, with the input being a state vector s and the output being the mean μ of a Gaussian distribution. θ (s) and log-standard deviation logσ θ (s) generates 10-dimensional motion commands through sampling, the motion commands including 3-dimensional chassis speed commands and 7-dimensional target joint angles of the robotic arm.

[0154] The simulation training and transfer module is used to build a high-fidelity tomato plant model in the NVIDIA IsaacLab simulation environment and perform domain randomization. It adopts a two-stage training method of behavior cloning and PPO reinforcement learning to jointly train the visual encoding module, language encoding module, multimodal fusion module and action generation module, and transfers the trained parameters to the real robot platform.

[0155] The non-contact proximity execution module is used to control the movement of the wheeled chassis and the robotic arm according to the instructions output by the motion generation module. It monitors the contact between the robotic arm and the plant in real time through a collision detection mechanism, and triggers the picking actuator to complete the fruit picking when the end effector reaches the target picking point.

[0156] The data acquisition module, 3D picking point estimation module, visual encoding module, language encoding module, multimodal fusion module, action generation module, simulation training and transfer module, and non-contact proximity execution module can be integrated into the edge computing unit NVIDIA Jetson AGX Orin during actual deployment, and data communication and unified timing control can be achieved through the ROS2 topic mechanism.

[0157] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0158] The detailed descriptions listed above are merely specific illustrations of feasible embodiments of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A non-contact tomato harvesting control method based on a vision-language-action model, characterized in that, Includes the following steps: RGB-D images are acquired by an RGB-D depth camera installed at the end of the robotic arm, natural language picking instructions are received through a human-machine interface, and robot body perception information is obtained through an encoder. The body perception information includes the joint angles, joint speeds, and chassis status of the robotic arm. The RGB-D image is input into the visual encoder to extract the visual feature vector, and the natural language instruction is input into the language encoder to extract the semantic feature vector. RGB-D images are input into a 3D picking point estimation network to detect 2D key points at the center of the tomato fruit and the center of the calyx. The 3D coordinates are recovered by combining the depth image, and the picking direction vector and picking point position are calculated. Using semantic feature vectors as queries and visual feature vectors as keys and values, a state vector is constructed by fusing 3D position embedding and ontology-aware features through a cross-attention mechanism. The state vector is input into the policy network, and the parameters of the action distribution are output. The action is sampled and obtained. The action includes the chassis speed command and the target joint angle of the robotic arm. The robot moves its chassis and robotic arm according to motion commands, monitors its contact with branches and leaves in real time through collision detection, and determines that it is close to success when the end effector reaches the target picking point.

2. The non-contact tomato harvesting control method based on a vision-language-action model according to claim 1, characterized in that, The visual encoder adopts the ViT-Large architecture, which divides the input RGB image into multiple image blocks of a preset size, and then inputs them into a multi-layer Transformer encoder after linear embedding and positional encoding. The depth information adopts an early fusion strategy, which concatenates the RGB image and the depth image in the channel dimension to form an RGB-D multi-channel input, which is then mapped to a three-channel feature map through a convolutional layer before being input into ViT-Large. The language encoder adopts a T5-Large architecture, takes the mean pooling result of the last hidden state of the encoder as the semantic feature vector, and adopts an instruction randomization strategy during training to generate multiple natural language instructions with different expressions for the same operation target.

3. The non-contact tomato harvesting control method based on a vision-language-action model according to claim 1, characterized in that, The three-dimensional picking point estimation network is based on CenterNet architecture, using EfficientNet-B3 as the backbone network, and embedding a CBAM attention module after the backbone network to enhance attention to occluded targets; the detection head includes a center point heatmap branch, a size regression branch, and an offset regression branch, which output the two-dimensional coordinates (u) of the fruit center and the calyx center, respectively. f ,v f ) and (u c ,v c The three-dimensional coordinate recovery uses the following formula: ， In the formula, and , respectively, represent the three-dimensional coordinates of the fruit center and calyx center in the camera coordinate system; K is the camera intrinsic parameter matrix; d f d represents the depth value at the center pixel of the fruit. c This represents the depth value at the center pixel position of the calyx. Through hand-eye transformation matrix Transform the center of the fruit and the center of the calyx into the robot's base coordinate system to obtain... and ; Picking direction vector n pick Calculate using the following formula: , Picking point location p pick Calculate using the following formula: , In the formula: Represents the Euclidean norm; r f The radius of the fruit is determined by half the width of the bounding box output by the size regression branch.

4. The non-contact tomato harvesting control method based on a vision-language-action model according to claim 1, characterized in that, The cross-attention mechanism calculates the fusion features using the following formula: ， In the formula: z l z is a semantic feature vector; v W is a visual feature vector. Q To query the projection matrix; W K W is the key projection matrix; V The projection matrix is ​​d; k For attention head dimension; The fused features are then processed by layer normalization and a feedforward network to obtain the multimodal vector z. vl The three-dimensional position embedding e pos The coordinates p of the picking point output by the 3D picking point estimation network pick The ontology-aware feature z is obtained through encoding via a fully connected layer. prop This includes the robotic arm joint angles, joint speeds, and chassis status; The multimodal vector z vl 3D position embedding vector e pos and ontology-aware feature vector z prop By concatenating the data along the channel dimension, a complete state vector is formed: .

5. The non-contact tomato harvesting control method based on a vision-language-action model according to claim 1, characterized in that, The policy network employs a multilayer perceptron architecture, including multiple hidden layers, each with a predetermined number of neurons, and the activation function is ReLU; the policy network outputs a Gaussian distribution with mean μ. θ (s) and log-standard deviation logσ θ (s), the action is obtained by sampling using the reparameterization technique: ， in The noise is a standard normal distribution, and I is the identity matrix; θ is a trainable parameter of the policy network, and action a includes chassis speed command and target joint angle of the robotic arm.

6. The non-contact tomato harvesting control method based on a vision-language-action model according to claim 1, characterized in that, Training is guided by a reward function derived from a non-contact approach strategy, wherein the reward function is expressed as: ， Among them, the distance reward r d The reward is a negative exponential function of the distance between the end effector and the target pickup point; the closer the distance, the greater the reward. Orientation reward r o The end effector orientation is a square function of the direction of picking; collision penalty r c A negative value is assigned when contact is detected, otherwise zero; a successful reward of r is given. s A positive reward is given when the end effector reaches the vicinity of the target picking point and the orientation deviation is less than a preset threshold; otherwise, the reward is zero. A time penalty r is applied. t The negative constant applied at each time step; the weight w of each reward item. d w o w c w s w t All values ​​are preset positive values, with the collision penalty weight set to the highest value among all values.

7. The non-contact tomato harvesting control method based on a vision-language-action model according to claim 1, characterized in that, The policy network adopts a two-stage training method: the first stage is behavior cloning pre-training, which uses a collision-free motion planning algorithm to generate multiple expert trajectories, and supervises learning by minimizing the mean square error loss between the policy network output and the expert actions. The second stage involves fine-tuning the near-end strategy, with the objective function being: ， ， in, This is the ratio of the probability densities of the new and old strategies; Indicates the policy network in state s t Output action a t The probability density; This indicates that the old policy network in the previous update is in state s. t-1 Output action a t-1 The probability density; For the expected experience at time steps; A t For generalized advantage estimation; This is the clipping function; This is the cutting factor.

8. The non-contact tomato harvesting control method based on a vision-language-action model according to claim 1, characterized in that, It also includes transferring policies trained in a simulation environment to a real robot platform, with transfer methods including zero-shot transfer or fine-tuning transfer. When fine-tuning the transfer, a small number of trajectories are collected in the real environment for parameter fine-tuning. The parameters of the visual encoder and language encoder are kept fixed, and only the cross-attention projection matrix and policy network parameters are updated. The learning rate is set to the preset multiple during simulation training.

9. A control system for a non-contact tomato harvesting control method based on a vision-language-action model according to any one of claims 1-8, characterized in that, include: The data acquisition module is used to control the RGB-D depth camera installed at the end of the robotic arm to acquire RGB-D images, receive natural language picking instructions through the human-machine interface, and obtain robot body perception information through the encoder; The 3D picking point estimation module is used to detect 2D key points at the center of the tomato fruit and the center of the calyx from RGB-D images, recover 3D coordinates by combining depth images, and calculate the picking direction vector and picking point position. The visual encoding module is used to input RGB-D images into the visual encoder to extract visual feature vectors; The language encoding module is used to input natural language instructions into the language encoder to extract semantic feature vectors; The multimodal fusion module is used to construct a state vector by fusing 3D position embedding and ontology-aware features through a cross-attention mechanism, using semantic feature vectors as queries and visual feature vectors as keys and values. The motion generation module is used to input the state vector into the policy network, output the parameters of the motion distribution, and sample the motion including the chassis speed command and the target joint angle of the robotic arm. The non-contact proximity execution module is used to control the movement of the chassis and robotic arm according to the action command. It monitors the contact with branches and leaves in real time through collision detection and determines that the approach is successful when the end effector reaches the target picking point.

Images