VLA task space control method and system for multi-degree-of-freedom mechanical arm

Abstract

The invention discloses a VLA task space control method and system for a multi-degree-of-freedom mechanical arm, and relates to the technical field of intelligent robot control, visual information and language instructions when the mechanical arm executes a task are obtained, and a trained VLA model is adopted to obtain a target pose of an end effector in a task space; an IK solver corresponding to the mechanical arm configuration is adopted, and according to the target pose, multiple sets of joint angle solutions of all joints of the mechanical arm are obtained; selecting an optimal joint angle of each joint from the multiple groups of joint angle solutions by adopting a preset optimization criterion; and after the optimal joint angle sequence of the mechanical arm is subjected to track smoothing processing, the mechanical arm is driven to move. Mechanical arm motion control is carried out in combination with a VLA model and an IK solver, the problems that the generalization ability between different mechanical arm configurations is poor, the data collection cost is high and the model interpretability is insufficient are solved, and the generalization ability and data efficiency of mechanical arm control are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot control technology, and in particular to a VLA task space control method and system for multi-degree-of-freedom robotic arms. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of artificial intelligence technology, deep learning-based robot control methods, especially Vision-Language-Action (VLA) models, have shown great potential in achieving complex robotic tasks. VLA models can directly learn and generate robot actions from visual input and verbal commands, achieving end-to-end control. For example, Google's RT-X series and OpenVLA models, trained on large-scale datasets, enable robotic arms to perform various operational tasks. These models typically employ an end-to-end learning paradigm, directly mapping perceptual inputs (such as images and text) to control commands (such as joint angles and torques) in the joint space.

[0004] However, existing VLA models face significant challenges in practical applications, particularly in their ability to generalize across robotic platforms. For example, while Google's Robotics Transformer (RT-1) and subsequent RT-2 models, proposed in patent US20230182295A1, demonstrate powerful task execution capabilities, their training data typically comes from specific types and degrees of freedom (e.g., 7-DOF) robotic arms. This means that if the training data comes from a specific configuration (e.g., 7-DOF robotic arm), the model's generalization ability when controlling other configurations (e.g., 6-DOF or 5-DOF robotic arms) will be severely limited. Each time the robotic arm hardware platform is changed, a large amount of data needs to be collected again and the model retrained, resulting in high costs and low deployment efficiency, significantly hindering the widespread adoption and application of VLA models across diverse robotic platforms.

[0005] Furthermore, some studies have explored combining learning methods with traditional robot control. For example, some research utilizes deep learning to solve inverse kinematics problems or maps learned policies to task space objectives. However, these methods generally suffer from two limitations: first, the research scope is too narrow, focusing only on single problems such as inverse kinematics (IK) solutions; second, even when some studies adopt learning paradigms to achieve more general capabilities, they usually remain superficial, failing to fully utilize the core advantages of VLA models in advanced task understanding and semantic reasoning, and failing to overcome the challenge of cross-platform generalization. For example, some task space-based learning methods, while outputting task space objectives, may still require fine-tuning of their VLA models for specific robots, or lack intelligent policies when dealing with multiple solutions to inverse kinematics. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a VLA task space control method and system for multi-degree-of-freedom robotic arms. By combining a VLA model with an IK solver for robotic arm motion control, this method solves the problems of poor generalization ability among different robotic arm configurations, high data acquisition costs, and insufficient model interpretability, thereby improving the generalization ability and data efficiency of robotic arm control.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a VLA task space control method for multi-degree-of-freedom robotic arms, comprising: The visual information and verbal commands of the robotic arm during task execution are acquired, and the trained VLA model is used to obtain the target pose of the end effector in the task space. Using an IK solver corresponding to the robotic arm configuration, multiple joint angle solutions for each joint of the robotic arm are obtained based on the target pose. Using a preset optimization criterion, the optimal joint angle for each joint is selected from multiple joint angle solutions; The optimal joint angle sequence of the robotic arm is smoothed through trajectory processing before driving the robotic arm to move.

[0008] As an alternative implementation, a VLA model is trained using visual information and verbal commands as input and the pose of the robotic arm's end effector in the task space as output; pose refers to the three-dimensional position and orientation of the end effector in Cartesian coordinates; during training, the loss function is to minimize the error between the predicted pose and the true pose. ; in, and These are the position error terms. and attitude error term The weight, and These represent the predicted 3D position and the actual 3D position, respectively. and These represent the predicted quaternion pose and the true quaternion pose, respectively.

[0009] As an alternative implementation, the VLA model employs a Transformer vision-language fusion architecture, including: The input layer is used to receive the embedded vectors of processed visual features and language instructions; The fusion module is used to fuse visual and linguistic features through a multi-head cross-attention mechanism; The task space pose prediction head is used to output the target pose; The IK solver obtains multiple joint angle solutions for each joint of the robotic arm based on the kinematic model of the robotic arm and the target pose of the end effector in the task space.

[0010] As alternative implementation methods, optimization criteria include minimizing joint motion, avoiding singularities, maintaining specific configurations, and incorporating VLA model context and task-specific constraints.

[0011] As an alternative implementation, the trajectory smoothing process for the optimal joint angle sequence of the robotic arm includes: using cubic spline interpolation and a Butterworth low-pass filter with a cutoff frequency of 10-50Hz for trajectory smoothing.

[0012] As an alternative implementation, after driving the robotic arm to move, the VLA model output or IK solver is also corrected online based on the actual joint angle and end-effector pose after the movement, so as to form closed-loop control.

[0013] Secondly, the present invention provides a VLA (Virtual Ability Assist) mission space control system for multi-degree-of-freedom robotic arms, comprising: The pose prediction module is configured to acquire visual information and verbal instructions when the robotic arm performs a task, and uses a trained VLA model to obtain the target pose of the end effector in the task space. The IK solution module is configured to use an IK solver corresponding to the robotic arm configuration to obtain multiple joint angle solutions for each joint of the robotic arm based on the target pose. The optimization module is configured to select the optimal joint angle for each joint from multiple joint angle solutions using preset optimization criteria. The control module is configured to drive the robotic arm to move after the optimal joint angle sequence of the robotic arm has been smoothed by trajectory processing.

[0014] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0015] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0016] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a VLA (Variable Ability Positioning) task space control method and system for multi-degree-of-freedom (DOF) robotic arms. The VLA model does not directly depend on the joint configuration of a specific robotic arm; its output is a general task space pose. Therefore, the same VLA model can seamlessly adapt to robotic arms of different degrees of freedom, sizes, and brands, fundamentally solving the problem of strong hardware coupling in existing VLA models and greatly improving the model's generalization ability and cross-platform application potential. Furthermore, when training the VLA model, only the task space pose data of the robotic arm's end effector needs to be collected, rather than the precise angle data of all joints. Task space pose data is easier to acquire, standardize, and reuse across platforms, significantly reducing the cost of data acquisition and model training.

[0018] This invention proposes a VLA (Variable Alignment) task space control method and system for multi-degree-of-freedom robotic arms. The VLA model outputs the task space objective, providing clear constraints for inverse kinematics (IK) solutions. In the presence of multiple solutions, an intelligent optimization mechanism is introduced. Combining the VLA model's understanding of the task or pre-defined optimization criteria, the optimal joint configuration is intelligently selected, avoiding ambiguities or undesirable movements that may occur in traditional end-to-end methods. This significantly improves the accuracy, smoothness, and safety of robotic arm control. Through multi-solution optimization and trajectory smoothing, the continuity, stability, and executability of robotic arm motion are ensured, avoiding abrupt changes and jitter.

[0019] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a flowchart of the VLA task space control method for a multi-degree-of-freedom robotic arm provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the preferred solution of multiple inverse kinematics provided in Embodiment 1 of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0026] Based on the problems in the background art, the existing technology mainly has the following shortcomings: Insufficient generalization ability and strong hardware coupling: Many existing end-to-end VLA robotic arm control methods rely heavily on specific robot kinematic configurations for their learning and inference, resulting in a tight coupling between the model and the physical parameters of a specific robotic arm (such as degrees of freedom, link lengths, joint limits, etc.). This coupling makes it difficult to directly generalize the model to robotic arms with different degrees of freedom, sizes, or brands. When changing robot platforms, the model is often difficult to apply directly and requires expensive retraining, which greatly limits its application scope and deployment flexibility. This limitation stems from the fact that VLA models learn and output directly in joint space, which is directly bound to the characteristics of the robot hardware.

[0027] High cost and low efficiency of data acquisition: Re-acquiring large amounts of joint space data for VLA model training for robotic arms with different configurations is a time-consuming, labor-intensive, and costly process. High-quality teaching data usually requires manual operation or complex automated systems to acquire, and needs to cover various task scenarios and robot configurations. This high data cost and low data reuse efficiency hinder the rapid popularization and application of VLA models on diverse robotic platforms. Data acquisition becomes a bottleneck, especially for long-tail tasks and new environments.

[0028] Poor model interpretability and debugging difficulties: End-to-end VLA models are often considered "black boxes," with their decision-making processes difficult to interpret and debug. When control anomalies occur (such as uneven robotic arm movement, collisions, or inability to complete tasks), it is difficult to pinpoint the source of the problem: sensor input, the internal decision-making logic of the VLA model, or the underlying execution level. This low interpretability increases the difficulty of system development and maintenance, and reduces system reliability.

[0029] The VLA (Vertical Inverse Kinematics) model faces challenges in effectively handling multiple solutions: Traditional VLA may involve multiple joint configurations that satisfy the end-effector pose conditions (i.e., multiple solutions). Current VLA models directly output joint angles, failing to effectively leverage the advantages of VLA in task space planning and also hindering the control uncertainties arising from multiple solutions. For example, the model might choose a joint solution that leads to the robotic arm entering a singularity, self-collision, or an undesirable configuration, impacting task success and safety. Furthermore, due to the lack of explicit understanding of task space semantics, VLA models struggle to intelligently select the best solution from multiple feasible options to meet task constraints or optimize motion trajectories.

[0030] Based on this, this invention provides a VLA task space control method for multi-degree-of-freedom robotic arms, which is a highly generalizable robotic arm control method based on a combination of vision-language-motion models and inverse kinematics. Its core idea is to decouple the VLA model from the strongly coupled mode of directly outputting joint angles, allowing it to focus on understanding high-level tasks and outputting the target pose of the robotic arm's end effector in the task space, while the specific joint kinematics calculation tasks are handled by a traditional inverse kinematics solver. This hierarchical decoupling architecture significantly improves the generalization ability, data efficiency, and interpretability of robotic arm control.

[0031] Example 1 like Figure 1 As shown, this embodiment provides a VLA task space control method for multi-degree-of-freedom robotic arms, including: The visual information and verbal commands of the robotic arm during task execution are acquired, and the trained VLA model is used to obtain the target pose of the end effector in the task space. Using an IK solver corresponding to the robotic arm configuration, multiple joint angle solutions for each joint of the robotic arm are obtained based on the target pose. Using a preset optimization criterion, the optimal joint angle for each joint is selected from multiple joint angle solutions; The optimal joint angle sequence of the robotic arm is smoothed through trajectory processing before driving the robotic arm to move.

[0032] In this embodiment, visual information (images / videos), verbal commands (text), and the end effector's task space pose (position and orientation) obtained through a high-precision external tracking system, forward kinematics calculation, or visual positioning system are simultaneously acquired during the robotic arm's task execution. This pose is independent of the joint angles of a specific robotic arm. Subsequently, the multimodal data is time-synchronized and labeled to form (visual input, verbal commands, target end effector pose) data pairs for supervised learning of the VLA model. Finally, the data is cleaned, denoised, and normalized to improve data quality and ensure the stability of model training.

[0033] Specifically: (1) Data types mainly include: Visual information: Images or video streams of the robotic arm's working environment, which may come from RGB cameras, depth cameras, or multi-view cameras. This data is used by the VLA model to understand the scene and the target.

[0034] Language instructions: Task instructions in natural language form, such as "pick up the red cube" or "place the cup on the edge of the table." These instructions, along with visual information, serve as input to the VLA model.

[0035] End-effector task space pose: Unlike traditional methods that acquire joint angles, this embodiment acquires the three-dimensional position (X, Y, Z) and three-dimensional pose of the robotic arm's end-effector in a Cartesian coordinate system (e.g., represented by Euler angles, quaternions, or rotation matrices). This data is obtained through the following methods: High-precision external tracking systems, such as the OptiTrack Prime series, with position accuracy <0.1mm and attitude accuracy <0.1°, or electromagnetic tracking systems, directly measure the real-time position and attitude of the end effector.

[0036] Robot forward kinematics calculation: If the robotic arm is equipped with a high-precision encoder, the pose of the end effector can be calculated in real time by reading the joint angles and combining them with the known kinematic model of the robotic arm. The forward kinematics calculation here is only used for data acquisition and not for control.

[0037] Visual positioning system: It uses an external camera and target recognition technology to identify specific markers or features on the end effector, thereby estimating its pose.

[0038] As alternative implementations, in addition to Cartesian coordinates and Euler angles / quaternions for representing task space pose, spiral coordinates or Lie group / Lie algebra representations can also be used. These representations have more natural mathematical properties in robot kinematics and dynamics, and may provide a simpler and more robust mathematical framework, especially when dealing with continuous motion and optimization problems, which helps VLA models learn more efficient task space trajectories.

[0039] As an alternative implementation, the VLA model can output the relative pose change relative to a reference point (such as the target object, robot base, or previous end-effector pose), rather than the absolute pose. This representation helps the model learn task-related relative motions better and is more robust to absolute position changes in the environment.

[0040] (2) Data annotation: The collected visual information, language commands and end effector task space poses are annotated synchronously to form data pairs of (visual input, language command, target end effector pose). For example, when the user issues the command "pick up the pen", the corresponding visual frame sequence and the precise pose sequence of the robotic arm end effector before, during and after grasping the pen are recorded.

[0041] (3) Data cleaning and normalization: The collected data is cleaned to remove outliers and noise. The pose data is normalized to the range of [-1, 1] and the pose data is uniformly converted into a unit quaternion representation.

[0042] (4) Data augmentation strategy: In addition, data augmentation techniques can be used during data collection and training, such as small-scale perturbation of end pose data, introduction of visual data with different backgrounds, and geometric transformations such as random cropping and rotation, to improve the robustness of VLA model.

[0043] In this embodiment, a VLA model is trained using visual information and language commands as input, and the pose of the robotic arm's end effector in the task space as output. The VLA model does not directly predict joint angles, fundamentally decoupling it from the specific kinematic configuration of the robotic arm and significantly improving cross-platform generalization capabilities. The VLA model employs a Transformer vision-language fusion architecture and is optimized for continuous pose prediction in the task space, such as by adding a dedicated pose prediction head to the output layer or enhancing the model's understanding of the geometric relationships in the task space through self-supervised learning tasks. During training, by minimizing the error between the predicted pose and the true pose, the VLA model can accurately understand semantic intent and generate precise task space trajectories.

[0044] Specifically: (1) Model architecture: The VLA model is built on the existing advanced deep learning model Transformer. Its core is that it can integrate visual and linguistic information and output continuous task space pose data.

[0045] Input layer: Receives processed visual features (e.g., image features extracted via convolutional neural networks) and embedding vectors of language instructions (e.g., encoded via pre-trained language models such as BERT).

[0046] Fusion Module: This module fuses visual and linguistic features using a multi-head cross-attention mechanism. Visual features serve as the Key and Value, while linguistic features serve as the Query. The number of attention heads is set to 8-16, enabling the model to understand the meaning of linguistic instructions in the visual scene.

[0047] Task-space pose prediction head: The model output layer is no longer joint angles, but is specifically designed to predict the target position (3 floating-point numbers) and pose (e.g., 4 floating-point numbers representing quaternions) of the robotic arm's end effector in task space. This prediction head can be a fully connected layer or a more complex sequence prediction network to handle continuous pose trajectories.

[0048] (2) Training objective: The training objective of the model is to minimize the weighted error between the end pose predicted by the VLA model and the actual end pose acquired; The loss function is: ; in, and These are the position error terms. and attitude error term The weighting coefficients, and These represent the predicted 3D position and the actual 3D position, respectively. and These represent the predicted quaternion pose and the true quaternion pose, respectively.

[0049] (3) Training strategy: Supervised learning is carried out by behavior cloning, and reinforcement learning is combined to further optimize the model’s performance in complex tasks.

[0050] As an alternative implementation, besides the traditional Transformer or RNN architectures, the VLA model architecture can also employ a diffusion model. The diffusion model excels at generating high-quality, diverse sequence data and can be used to generate smoother, more physically consistent task space pose trajectories. Its advantages lie in its ability to better handle the continuity and multimodality of task space trajectories, while also better integrating visual and linguistic information through conditioned generation.

[0051] As an alternative implementation, for edge computing or resource-constrained robotic platforms, VLA models can be optimized using pruning, quantization, knowledge distillation, or more compact network structures (such as variants of MobileNet and EfficientNet) to obtain lightweight or efficient VLA models, thereby reducing computational resource consumption and inference latency while maintaining performance.

[0052] In this embodiment, visual information and verbal commands are acquired when the robotic arm performs a task. A trained VLA model is used to obtain the target pose of the end effector in the task space. For the target pose of the end effector in the task space, an independent IK solver built for a specific robotic arm is used to perform inverse kinematics (IK) calculations, obtaining multiple sets of joint angle solutions for each joint of the robotic arm. This solver is implemented using analytical solutions, numerical solutions, or learning methods. As an adaptation layer between the VLA model and specific robotic arm hardware, by pre-establishing or configuring corresponding IK solvers for different robotic arm platforms, seamless switching and deployment of the VLA model on robotic arm platforms with different degrees of freedom and configurations are effectively achieved without retraining the VLA model.

[0053] Specifically: (1) Solver construction: For each type of robotic arm that needs to be adapted, its corresponding IK solver is pre-built or configured. The IK solver calculates multiple sets of joint angle solutions for each joint of the robotic arm based on the kinematic model of the robotic arm (usually described by the DH parameter table) and the target pose of the end effector in the task space.

[0054] in, Analytical solution: For a 6-DOF industrial robotic arm, there is usually an analytical solution, which is fast and accurate.

[0055] Numerical solution: For robotic arms with redundant degrees of freedom (>6 degrees of freedom) or certain special configurations, numerical solution methods (such as the Jacobian matrix pseudo-inverse method and iterative method) are required. Although the computational load is slightly larger, it can still meet the real-time requirements.

[0056] Learning-based IK: In some cases, deep learning-based IK solvers (such as neural network IK) can be used to learn the mapping from end-effector pose to joint angles in a data-driven manner. This is particularly suitable for robotic arms with complex kinematic models, high redundant degrees of freedom, or nonlinear constraints. Learning-based IK solvers can be pre-trained or fine-tuned during deployment to adapt to the characteristics of a specific robotic arm.

[0057] Optimized IK solver: Employs more advanced optimization algorithms (such as Sequential Quadratic Programming (SQP) and interior point method) to solve IK problems, especially when multiple constraints (such as joint limits, speed limits, and obstacle avoidance) need to be satisfied simultaneously, providing more flexible and robust solutions.

[0058] (2) Interface design: The target pose data (X, Y, Z, quaternions / Euler angles) output by the VLA model is transmitted to the IK solver through a standard interface. After receiving the target pose, the IK solver immediately performs calculations and outputs one or more sets of joint angle solutions.

[0059] In this embodiment, multiple solution optimization and trajectory smoothing are employed: After obtaining multiple feasible solutions for inverse kinematics, an intelligent multiple solution optimization mechanism is introduced. This mechanism uses preset optimization criteria such as minimum joint motion, avoiding singularities, maintaining a specific configuration, incorporating VLA context information, and obstacle avoidance / task-specific constraints to select the optimal joint configuration. This effectively solves the control uncertainty problem caused by multiple IK solutions, which is difficult for traditional end-to-end VLA models to handle. Subsequently, the optimized joint angle sequence undergoes trajectory smoothing processing (such as spline interpolation, low-pass filtering, or motion planning) to enhance the continuity, stability, and executability of the robotic arm's motion, avoid abrupt changes and jitter, and comprehensively improve motion quality. Figure 2 As shown.

[0060] Specifically: (1) Multiple solution processing: Inverse kinematics calculation may result in multiple joint configurations that satisfy the end-effector pose conditions. Therefore, a multiple solution optimization mechanism is introduced to ensure the rationality and safety of the robotic arm's motion.

[0061] Optimization criteria, including but not limited to: Minimum joint motion: Select the solution that minimizes the change in the current joint angle to ensure smooth motion.

[0062] Avoid singularities: Prevent the robotic arm from entering singular configurations to prevent control instability.

[0063] Maintain a specific configuration: for example, maintain an "elbow up" or "elbow down" configuration to avoid collisions with the environment.

[0064] Incorporating VLA context: While predicting pose, the VLA model outputs additional contextual information (such as the relative position of the target object and information about environmental obstacles), which serves as the basis for selecting the best IK solution. For example, if the VLA model "knows" that the target is to the left of the robotic arm, it will choose a joint configuration that extends the robotic arm to the left.

[0065] Task-specific constraints: based on specific task requirements, such as avoiding self-collision, maintaining tool posture, etc.

[0066] As an alternative implementation method, the multiple-solution optimization strategy also includes: Multiple-solution optimization based on reinforcement learning: A reinforcement learning (RL) agent is trained, taking the task space pose output by the VLA model and multiple joint solutions from the IK solver as input. Through interaction with the environment, it learns the optimal joint configuration selection strategy. The RL agent optimizes its selection based on task rewards (such as success rate, smoothness, and safety), enabling it to adapt to more complex task scenarios and dynamic environments.

[0067] Context-based multi-solution optimization: In addition to the extra contextual information provided by the VLA model, richer environmental semantic information (such as object attributes, scene topology, and task stage) can be introduced to guide multi-solution optimization. For example, graph neural networks (GNNs) can be used to model the relationships between objects in the scene and use them as input to the optimization strategy.

[0068] (2) Trajectory smoothing: The selected joint target angle sequence is smoothed by using cubic spline interpolation and a Butterworth low-pass filter with a cutoff frequency of 10-50Hz to ensure the continuity, stability and executability of the robotic arm motion trajectory and avoid sudden changes and jitter.

[0069] As an alternative implementation method, trajectory smoothing and motion planning alternatives include: Trajectory optimization based on higher-order kinematic constraints: In addition to simple spline interpolation or low-pass filters, trajectory optimization algorithms that consider robotic arm dynamics and higher-order kinematic constraints (such as acceleration and jerk limits) can be used to generate motion trajectories that are more in line with physical laws, more efficient and smoother.

[0070] Online motion planning: Combining online motion planning algorithms (such as Fast Random Tree Search (RRT) and Probabilistic Route Graph (PRM)) to handle dynamic obstacles or unknown environments enables the robotic arm to adjust its trajectory in real time during task execution, thereby improving the robustness and adaptability of the system.

[0071] In this embodiment, the robotic arm control involves sending the optimized and smoothed target joint angles to the robotic arm's underlying controller, driving precise movement of each joint to ensure the end effector accurately reaches the target pose specified by the VLA model. Simultaneously, to further enhance system performance, real-time feedback based on vision, force sensing, or joint encoders is introduced to perform online correction of the VLA model output or IK solver input, forming a closed-loop control. This mechanism effectively enhances control accuracy and robustness and supports abnormal state detection and autonomous recovery.

[0072] (1) Low-level controller: The optimized and smoothed joint target angle is sent to the low-level controller built into the robotic arm. The low-level controller usually contains a PID controller or other advanced joint-level controller, which is responsible for precisely driving each joint of the robotic arm to the specified angle.

[0073] (2) Closed-loop feedback: The actual joint angles and end-effector poses of the robotic arm are fed back in real time through encoders and sensors. This feedback information can be used for: VLA Model Calibration: Adjust the output of the VLA model online to cope with environmental changes or execution errors.

[0074] IK Solver Calibration: Adjusts the input to the IK solver in real time to compensate for accumulated errors.

[0075] Anomaly detection and recovery: Monitor whether the movement of the robotic arm is in accordance with expectations, and trigger alarms or execute recovery strategies when anomalies occur.

[0076] The method described in this embodiment achieves the following beneficial effects by decoupling the VLA model from the inverse kinematics solver: Significantly improved generalization ability: The VLA model no longer directly depends on the joint configuration of a specific robotic arm; its output is a general task space pose. This means that the same VLA model can seamlessly adapt to robotic arms of different degrees of freedom, sizes, and brands, as long as these robotic arms have the corresponding inverse kinematics solution capabilities. This fundamentally solves the problem of strong hardware coupling in existing VLA models, greatly improving the model's generalization ability and cross-platform application potential, making it one of the most outstanding innovations.

[0077] Significantly reduced data acquisition and training costs (significant economic benefits): When training a VLA model, only the task space pose data of the robotic arm's end effector needs to be collected, rather than the precise angle data of all joints. Task space pose data is easier to acquire, standardize, and reuse across platforms. For example, task space pose data generated through a simulation environment, or data collected on a small number of real robotic arms, can be used to train a general-purpose VLA model, significantly reducing the cost of data acquisition and model training and accelerating the commercialization of VLA technology.

[0078] Enhanced model interpretability and debuggability (improved system reliability): The control flow is clearly divided into "high-level task understanding and pose planning" (VLA model) and "low-level kinematics calculation and execution" (IK solver). This modular, layered architecture makes the system more interpretable, facilitating problem localization and debugging. When control anomalies occur, it can be clearly identified whether the problem lies with the VLA model planning, IK solving, or low-level execution, thereby improving system reliability and maintenance efficiency.

[0079] Effectively handling multiple solutions in inverse kinematics (improved control accuracy and safety): The VLA model outputs the task space objective, providing explicit constraints for inverse kinematics solutions. In cases of multiple solutions, an intelligent optimization mechanism is introduced, combining the VLA model's understanding of the task or pre-defined optimization criteria (such as minimum joint motion, avoiding singularities, maintaining a specific configuration, and avoiding self-collisions) to intelligently select the optimal joint configuration. This avoids ambiguity or undesirable movements that may occur with traditional end-to-end methods, significantly improving the accuracy, smoothness, and safety of robotic arm control.

[0080] Accelerating the deployment and application of new robots (enhancing market competitiveness): For new robotic arm platforms, only the corresponding inverse kinematics solver needs to be configured or developed, without the need to retrain the expensive and time-consuming VLA large model. This greatly shortens the development and deployment cycle of new applications, enabling VLA technology to adapt to market demands and new hardware iterations more quickly, thus enhancing the market competitiveness of products.

[0081] Improving motion quality and robustness: Through multi-solution optimization and trajectory smoothing, the continuity, smoothness, and executability of the robotic arm's motion are ensured, avoiding abrupt changes and jitter. Simultaneously, a closed-loop feedback mechanism further enhances the system's robustness to environmental disturbances and execution errors, ensuring stable task completion.

[0082] The following example illustrates the task of "grabbing the red square on the desktop and putting it into the basket".

[0083] Data acquisition: A 7-DOF robotic arm was used for teaching to acquire visual images, verbal commands such as "grab the red square" and "put it in the basket", as well as the 3D position and orientation data of the robotic arm end effector during the grasping and placement process.

[0084] VLA Model Training: Train the VLA model so that it can predict the grasping pose of the red square and the placement pose of the basket based on visual input and verbal instructions.

[0085] Deployment to a 6-DOF robotic arm: The trained VLA model is deployed onto a 6-DOF robotic arm. This 6-DOF robotic arm is pre-configured with its own IK solver.

[0086] Task Execution: When the user issues the command "Grab the red square," the VLA model identifies the red square based on visual input and outputs its grasping pose. This pose is passed to the IK solver of the 6-DOF robotic arm to calculate the joint angles. If multiple solutions exist, the optimal solution is selected according to the preset "minimum joint motion" principle. The joint angles are smoothed and then sent to the robotic arm for execution. After grasping, the VLA model outputs the placement pose, the IK solver recalculates the joint angles, and the robotic arm places the square into the basket.

[0087] The above implementation method enables the generalization of the VLA model on robotic arms with different degrees of freedom, without the need to retrain the VLA model for a 6-DOF robotic arm.

[0088] Example 2 This embodiment provides a VLA (Virtual Ability Assist) mission space control system for multi-degree-of-freedom robotic arms, including: The pose prediction module is configured to acquire visual information and verbal instructions when the robotic arm performs a task, and uses a trained VLA model to obtain the target pose of the end effector in the task space. The IK solution module is configured to use an IK solver corresponding to the robotic arm configuration to obtain multiple joint angle solutions for each joint of the robotic arm based on the target pose. The optimization module is configured to select the optimal joint angle for each joint from multiple joint angle solutions using preset optimization criteria. The control module is configured to drive the robotic arm to move after the optimal joint angle sequence of the robotic arm has been smoothed by trajectory processing.

[0089] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0090] In further embodiments, the following is also provided: An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0091] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0092] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0093] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0094] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0095] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0096] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0097] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0098] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0099] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0100] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A VLA task space control method for multi-degree-of-freedom robotic arms, characterized in that, include: The visual information and verbal commands of the robotic arm during task execution are acquired, and the trained VLA model is used to obtain the target pose of the end effector in the task space. Using an IK solver corresponding to the robotic arm configuration, multiple joint angle solutions for each joint of the robotic arm are obtained based on the target pose. Using a preset optimization criterion, the optimal joint angle for each joint is selected from multiple joint angle solutions; The optimal joint angle sequence of the robotic arm is smoothed through trajectory processing before driving the robotic arm to move.

2. The VLA task space control method for a multi-degree-of-freedom robotic arm as described in claim 1, characterized in that, The VLA model is trained by taking visual information and language commands as inputs and the pose of the end effector of the robotic arm in the task space as outputs; the pose is the three-dimensional position and three-dimensional orientation of the end effector in the Cartesian coordinate system. During training, the loss function is to minimize the error between the predicted pose and the true pose. ； in, and These are the position error terms. and attitude error term The weight, and These represent the predicted 3D position and the actual 3D position, respectively. and These represent the predicted quaternion pose and the true quaternion pose, respectively.

3. The VLA task space control method for a multi-degree-of-freedom robotic arm as described in claim 1, characterized in that, The VLA model employs a Transformer vision-language fusion architecture, including: The input layer is used to receive the embedded vectors of processed visual features and language instructions; The fusion module is used to fuse visual and linguistic features through a multi-head cross-attention mechanism; The task space pose prediction head is used to output the target pose; The IK solver obtains multiple joint angle solutions for each joint of the robotic arm based on the kinematic model of the robotic arm and the target pose of the end effector in the task space.

4. The VLA task space control method for a multi-degree-of-freedom robotic arm as described in claim 1, characterized in that, The optimization criteria include minimizing joint motion, avoiding singularities, maintaining specific configurations, and combining VLA model context with task-specific constraints.

5. The VLA task space control method for a multi-degree-of-freedom robotic arm as described in claim 1, characterized in that, The trajectory smoothing process for the optimal joint angle sequence of the robotic arm includes: using cubic spline interpolation and a Butterworth low-pass filter with a cutoff frequency of 10-50Hz for trajectory smoothing.

6. The VLA task space control method for a multi-degree-of-freedom robotic arm as described in claim 1, characterized in that, After driving the robotic arm to move, the system also includes online correction of the VLA model output or IK solver based on the actual joint angles and end-effector pose after the movement, in order to form closed-loop control.

7. A VLA (Virtual Ability Assist) mission space control system for multi-degree-of-freedom robotic arms, characterized in that, include: The pose prediction module is configured to acquire visual information and verbal instructions when the robotic arm performs a task, and uses a trained VLA model to obtain the target pose of the end effector in the task space. The IK solution module is configured to use an IK solver corresponding to the robotic arm configuration to obtain multiple joint angle solutions for each joint of the robotic arm based on the target pose. The optimization module is configured to select the optimal joint angle for each joint from multiple joint angle solutions using preset optimization criteria. The control module is configured to drive the robotic arm to move after the optimal joint angle sequence of the robotic arm has been smoothed by trajectory processing.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

Images