Mechanical arm control method and system, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of robot control, and discloses a mechanical arm control method and system, electronic equipment and a readable storage medium, and the method comprises the steps: obtaining a visual image and a natural language task instruction of a working environment of a mechanical arm, and extracting time features and space features of the environment through a multi-modal large model; the task is decomposed into a plurality of sub-tasks, and a mechanical arm action sequence is generated; the multi-modal space-time aided decision-making model extracts relation features of the time features and the space features in real time, and detects an action execution error in combination with a preset time accumulation error threshold value and a space deviation threshold value; if it is detected that the action execution error exceeds the threshold value, the current action is paused, the mechanical arm is controlled to return to the previous safe position, and an error correction action generation mechanism is triggered; and in combination with a self-attention mechanism of a multi-modal large model, a visual image, a natural language instruction and mechanical arm state information are fused, a corrected action sequence is generated, the mechanical arm is controlled again to execute, and the generalization ability and task success rate of the model are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a mechanical arm control method, system, electronic device and readable storage medium, and particularly relates to an imitation learning control method and system combining a motion block algorithm of a multi-modal large model. BACKGROUND

[0002] At present, the mechanical arm control method in the related art mainly has the following several ways: Li H Y, Zhao Z L, Gu L, et al. Mechanical arm control method based on deep reinforcement learning [J]. Journal of Systems Simulation, 2019, 31(11): 2452-2457. Deep reinforcement learning constantly explores and tries in the environment, and adjusts the neural network parameters through the reward function. According to the target setting state variable and reward mechanism, the deep deterministic policy gradient algorithm is trained in the model, the mechanical arm is controlled through the deep reinforcement learning algorithm, the gripper is moved to the lower side of the object, the adaptability of the control algorithm is improved, and the debugging time is shortened. The experimental results show that the deep learning algorithm can reach convergence in a shorter time, and the control of the mechanical arm is realized. However, there are problems such as the need to retrain when changing the scene and target object, limited long-time sequence capture ability for complex continuous tasks, and difficulty in ensuring the smoothness and accuracy of the action sequence. This method depends on a large amount of environment interaction training, and the training process is time-consuming and consumes a lot of computing resources.

[0003] Liu J, Pu J X, Niu X Y. Mechanical arm trajectory tracking control method based on neural network sliding mode [J]. Computer Engineering and Design, 2019, 40(07): 1934-1938. In view of the problem of control performance decline caused by modeling error and external disturbance in mechanical arm trajectory tracking control, an improved adaptive neural sliding mode control method is proposed. State feedback and improved neural network sliding mode method are selected to control the deterministic part and uncertain part of the system. The nonlinear mapping ability of the neural network is used to adaptively learn the unknown upper bound of the system uncertainty, and its output is used as the dynamic compensation item of the sliding mode controller. The neural network weight update law is derived by Lyapunov function method. In order to ensure the effectiveness of neural network mapping and improve the convergence speed, genetic algorithm is used to optimize the neural network structure parameters. This method mainly compensates for the mechanical arm dynamics modeling error and disturbance, and lacks the understanding ability of complex task semantics and environment information. It depends on accurate dynamic model and parameter optimization, and has limited adaptability and flexibility. It cannot realize intelligent planning based on task instructions.

[0004] Zhu Haiyan. Trajectory planning and control method of seven degrees of freedom humanoid robot arm. Zhejiang University of Technology, 2012. The recursive Newton-Euler algorithm is used to solve the dynamic equation of the robot arm, and the calculation results of the joint driving torque of the algorithm are instantiated. In view of the shortcomings of the trajectory curve planned by the previous trajectory planning method, the symmetric combination sine function method is used for trajectory planning of the robot arm, the curve adjustment parameter is designed, and the smooth and continuous acceleration curve is obtained. Based on the study of adaptive iterative learning control theory, an improved AILC control strategy is designed by adding a parameter uncertainty compensator and introducing a saturation function, which reduces the influence of parameter uncertainty and control torque chattering. And the improved AILC control algorithm is applied to the control of the robot arm. The trajectory planning method improves the smoothness of the trajectory, but still depends on the predefined trajectory, and lacks real-time response capability to dynamic changes in tasks. It is difficult to handle long-term dependencies and multi-task switching in complex continuous tasks, and the task planning capability is limited. SUMMARY

[0005] In order to solve the above technical problems, the present application provides a robot arm control method, system, electronic device and readable storage medium, which is suitable for efficient planning and accurate execution of robot arm continuous tasks.

[0006] The first aspect of the present application provides a robot arm control method, comprising the following steps: Obtain the visual image of the working environment of the robot arm and the natural language task instruction, and extract the time feature and space feature of the environment through the multi-modal large model; Based on the multi-modal large model, the task is high-level planned and decomposed into multiple sub-tasks, and the robot arm action sequence corresponding to each sub-task is generated through the action block algorithm; During the execution of the robot arm, the relationship features of the time features and the space features are extracted in real time through the multi-modal spatio-temporal auxiliary decision model, and the action execution error is detected in combination with the preset time cumulative error threshold and the space deviation threshold; If the action execution error is detected to exceed the threshold, the current action is paused, the robot arm is controlled to return to the previous safe position, and the error correction action generation mechanism is triggered; Model the error correction action based on Markov decision process, combine the self-attention mechanism of the multi-modal large model, fuse the visual image, the natural language instruction and the robot arm state information, generate the corrected action sequence and control the robot arm to execute again.

[0007] Optionally, the action block algorithm uses a conditional variational autoencoder, and the input includes: a multi-channel two-dimensional matrix corresponding to the visual image, joint state information of the robot arm and a task instruction label; The output is a sequence of k action blocks corresponding to the current observation condition, and each action block corresponds to a set of joint position control instructions of the robot arm.

[0008] Optionally, the setting of the time accumulation error threshold comprises: Based on the similarity calculation of the sliding window of the historical motion trajectory of the mechanical arm and the current execution trajectory, the current trajectory is compared with the reference trajectory in the training demonstration, and the time error is accumulated according to the similarity distance.

[0009] Optionally, the setting of the spatial deviation threshold comprises: Based on the spatial relationship of the current position of the mechanical arm, the target position of the task and the objects in the environment, the spatial distance deviation of the current execution trajectory and the expected trajectory is calculated.

[0010] Optionally, after generating the mechanical arm motion sequence corresponding to each sub-task, the following steps are further included: using a flow matching method to correct the motion sequence, generating a smooth and accurate motion vector field, and controlling the mechanical arm to execute.

[0011] Optionally, the flow matching method comprises: The noisy motion sequence is supervised trained through a conditional flow matching loss function, learning to correct the noisy motion to a correct motion vector field; In the inference stage, starting from random noise, a smooth motion sequence is gradually generated through forward Euler integration.

[0012] Optionally, the error correction motion generation mechanism comprises: The multi-modal large model is used to generate a basic motion prediction and a memory prompt motion prediction based on historical demonstration; The weights of the two are calculated by Softmax normalization, and the final error correction motion sequence is dynamically fused and generated.

[0013] The second aspect of the present application provides a mechanical arm control system, comprising: A multi-modal perception module is used to acquire visual images and natural language instructions; A task decomposition and planning module is used to perform task decomposition and motion sequence generation based on a multi-modal large model; An error detection and correction module is used to detect motion errors in real time and trigger a correction mechanism; The system is used to implement the mechanical arm control method described above.

[0014] The third aspect of the present application provides an electronic device, comprising: At least one processor; A memory storing a computer program; When the computer program is executed by the processor, the mechanical arm control method described above is implemented.

[0015] The fourth aspect of the present application provides a computer readable storage medium storing a computer program, which, when executed by a processor, implements the mechanical arm control method described above.

[0016] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages: The mechanical arm control method, system, electronic device and readable storage medium provided by the embodiments of the present application first extract the time and space features of the environment from the visual image and natural language task instruction through the multi-modal large model, so that the system has semantic understanding ability of the environment, instead of relying on fixed models or predefined trajectories. Then, the multi-modal large model is used for task high-level planning, the complex continuous task is decomposed into multiple sub-tasks, and the corresponding mechanical arm action sequence is generated through the action block algorithm, thereby enhancing the adaptability and generalization ability to diversified tasks. The system simultaneously processes the visual image and the natural language instruction, realizes semantic-level understanding of the task intention and the environment structure, extracts the relationship between the time and space features in real time through the multi-modal space-time auxiliary decision model, detects the execution error in combination with the preset threshold, and realizes intelligent task execution monitoring. The system detects the action error in real time during the execution of the mechanical arm, and once the threshold is exceeded, the execution is paused, the safe position is returned, and the error correction action generation mechanism is triggered. Then, based on the error correction modeling of the Markov decision process, the multi-modal information is fused in combination with the self-attention mechanism to generate the corrected action sequence, thereby realizing dynamic and adaptive task execution and error correction. The mechanical arm control method, system, electronic device and readable storage medium provided by the present application apply the multi-modal large model to the mechanical arm control, realize a control method based on action-language-action, control the mechanical arm action in the current period by combining the action in the past time and space of the mechanical arm and the action sequence logic to be realized in the future, enhance the task understanding and environment adaptation ability by introducing the visual and language multi-modal information, improve the accuracy and smoothness of the action sequence, reduce the error accumulation, and improve the generalization ability and task success rate of the model. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a mechanical arm control method provided by the embodiments of the present application is shown in the figure; Figure 2 A sampling output process diagram provided by the embodiments of the present application is shown in the figure; Figure 3 A multi-modal space-time information fusion flowchart provided by the embodiments of the present application is shown in the figure; Figure 4 A task planning generation flowchart provided by the embodiments of the present application is shown in the figure; Figure 5 A flowchart of training of the predicted mechanical arm motion sequence through the neural network provided by the embodiments of the present application is shown in the figure; Figure 6The simulation experimental environment provided in this embodiment of the invention includes a tracking diagram of the actual execution angle of the model. Figure 7 The simulation experimental environment provided in this embodiment of the invention includes a tracking diagram of the model's predicted angle. Figure 8 The teaching trajectory diagram is provided for the experimental environment provided in this embodiment of the invention when the initial positions of the objects are the same. Figure 9 The model predicts the trajectory when the initial positions of objects in the experimental environment provided in this embodiment of the invention are the same. Detailed Implementation

[0018] The following detailed description of a specific embodiment of the present invention is provided in conjunction with the accompanying drawings. However, it should be understood that the scope of protection of the present invention is not limited to the specific embodiment.

[0019] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the technical solution of this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0020] The present invention will be described below through several specific embodiments. To keep the following description of the embodiments clear and concise, detailed descriptions of known functions and components may be omitted. When any component of an embodiment of the present invention appears in more than one drawing, the component may be represented by the same reference numerals in each drawing.

[0021] like Figure 1 As shown, the first embodiment of the present invention provides a robotic arm control method, including the following steps: The system acquires visual images and natural language task instructions of the robotic arm's working environment, and encodes them using a multimodal large model to extract the temporal and spatial features of the environment. The spatiotemporal features are obtained based on the robotic arm's historical and current states during the task. The temporal features include the task objectives and completion status connected by time relationships. The spatial features are the relationships between the robotic arm and the task space and task items, which are used to generate reasoning, determine whether the action is executed correctly, and understand and generate task planning.

[0022] Task decomposition and action block generation mechanism: based on the multi-modal large model, the task is planned at a high level, the execution action and movement are planned in sequence, and the task is decomposed into multiple sub-tasks, and the action block algorithm is used to generate the corresponding robot arm action sequence, and the multi-modal large model guides the generated action sequence; Multi-modal spatio-temporal information fusion asynchronous reasoning decision assistance: during the execution of the robot arm, the multi-modal spatio-temporal auxiliary decision model is used to extract the relationship between the time characteristics and the space characteristics in real time, and the preset time cumulative error threshold and the space deviation threshold are combined to detect the action execution error; If the action execution error is detected to exceed the threshold, the current action is paused, the robot arm is controlled to return to the previous safe position, and the error correction action generation mechanism is triggered to ensure the continuity and accuracy of the action; Error correction action generation: based on Markov decision process to model the error correction action, combine reward function to optimize action strategy, combine multi-modal large model self-attention mechanism, according to the space characteristics, compare the real-time space characteristics with the unused action sequence space characteristics and the expected space characteristics, fuse the visual image, natural language instruction and robot arm state information, generate the corrected action sequence and control the robot arm to execute again, realize the deep understanding and dynamic adjustment to the task target.

[0023] This invention provides a robotic arm control method. First, it extracts temporal and spatial features of the environment from visual images and natural language task instructions using a multimodal large-scale model, enabling the system to semantically understand the environment, rather than relying on fixed models or predefined trajectories. Then, it performs high-level task planning using the multimodal large-scale model, decomposing complex continuous tasks into multiple sub-tasks. A motion segmentation algorithm generates corresponding robotic arm motion sequences, enhancing adaptability and generalization capabilities for diverse tasks. The system simultaneously processes visual images and natural language instructions, achieving semantic-level understanding of task intent and environmental structure. A multimodal spatiotemporal auxiliary decision-making model extracts the relationship between temporal and spatial features in real time, and combined with preset thresholds to detect execution errors, achieving intelligent task execution monitoring. During robotic arm execution, the system detects motion errors in real time. Once the threshold is exceeded, execution is paused, the system returns to a safe position, and an error correction motion generation mechanism is triggered. Subsequently, error correction modeling based on Markov decision processes, combined with a self-attention mechanism to fuse multimodal information, generates a corrected motion sequence, achieving dynamic and adaptive task execution and error correction. This invention provides a robotic arm control method, system, electronic device, and readable storage medium that applies a multimodal large model to robotic arm control, realizing a control method based on action-language-action. By combining the logic of the robotic arm's past actions in time and space with the predicted future action sequence, the robotic arm's actions in the current time period are controlled. By introducing visual and linguistic multimodal information, the task understanding and environmental adaptability are enhanced, the accuracy and smoothness of the action sequence are improved, error accumulation is reduced, and the model's generalization ability and task success rate are improved.

[0024] Traditional methods struggle to plan long-cycle, multi-step task action sequences, have limited ability to capture long-term dependencies in complex continuous tasks, and directly output raw, high-dimensional joint control sequences, which can easily lead to jitter and instability. They also struggle to ensure the smoothness and accuracy of action sequences, lack the ability to generalize to diverse tasks and environments, require retraining when changing scenes and target objects, and are disconnected from high-level semantic instructions and visual scenes. Furthermore, they lack the ability to understand the semantics and environmental information of complex tasks.

[0025] Based on the above problems, in the embodiments of the present invention, reference is made to Figure 4 The action segmentation algorithm uses a conditional variational autoencoder, whose inputs include: a multi-channel two-dimensional matrix corresponding to the visual image, robotic arm joint state information, and task instruction labels. The output is a sequence of k action blocks corresponding to the current observation conditions, with each action block corresponding to a set of joint position control commands for the robotic arm.

[0026] A complete task decomposition and action block generation mechanism includes: The task decomposition mechanism of designing the mechanical arm in the method is based on the multi-modal large model processing the input task instruction according to the set logic, which can split the task instruction into specific executable verbs. The executable verb is generated as an executable action block. The verb mechanical arm action generation core lies in the action block generation algorithm, which uses a conditional variational autoencoder derived from an improved variational autoencoder. It can introduce additional information in the generation process to control certain attributes of the generated results. The conditional variational autoencoder learns the relationship between the conditional information and the data by passing the conditional information and the input data to the encoder and the decoder together. The conditional information is the current angle and speed state information of each joint of the mechanical arm corresponding to each image. As shown in Figure 2 The process of sampling the output is shown: when the data is input into the encoder, the data content and its label are merged together; when the encoded input is decoded, the encoded content and the data label are merged together. Therefore, when generating data, the label of the data to be generated can be sampled from the normal distribution first, and then sent into the decoder together. The generated data is similar to the input label, and the generated data is the state data of the mechanical arm corresponding to the execution action under the characteristics of the input two-dimensional matrix.

[0027] The action chunking algorithm based on CVAE is not simply regressing joint angles, but constructing an action planning core with strong generative ability, interpretability and generalization. The structured and modular mapping from high-level semantics to low-level actions is achieved. Instead of directly outputting a long sequence of continuous joint angles, the algorithm generates a sequence of k action chunks. Each action chunk is a well-packaged, semantically clear motion primitive. Complex continuous tasks (such as "making coffee") are decomposed into action chunks such as "move to coffee jar", "grab", "move to cup", "pour", etc. These chunks can be reused and recombined in different tasks (such as "making tea" also requires "grab" and "pour"), greatly improving code reusability and learning efficiency. It solves the long-time sequence dependence problem. The model does not need to remember the details of each step, but learns how to select and concatenate suitable action chunks under the correct observation conditions. This reduces the difficulty of modeling long sequences. Each action chunk can be pre-designed or learned as a smooth trajectory (such as using polynomials or DMPs), ensuring that the actions within the chunk are natural and smooth, avoiding the jitter that point-by-point control may bring. It integrates multi-modal observation conditions to achieve context-based action generation. The input of CVAE integrates vision (multi-channel two-dimensional matrix), proprioceptive perception (joint state) and task intent (instruction label). This makes the generated action sequence highly conditional and accurately matches the current context. The model sees a specific scene image (such as scattered building blocks), rather than an abstract state. Therefore, the "grab red building block" action chunk it generates will adjust the grab pose and path based on the actual position, pose and occlusion of the red building block in the image. The current joint state is input to enable the model to generate a coherent and dynamically feasible next action block from the current state, achieving closed-loop planning. Different task instruction labels guide the model to search for completely different action sequence distributions in the latent space. The same scene (a table with a cup and a kettle), "pouring water" and "cleaning the table" will evoke completely different action chunk sequences. By learning a smooth latent variable (z) space, CVAE can generate action sequences that have never been seen during training but are reasonable. For a new instruction "put the object between point A and point B", the model can naturally generate an action sequence that puts the object in an intermediate position by interpolating the latent vectors corresponding to "put it at point A" and "put it at point B" in the latent space, without the need for retraining. When the visual features such as object position and lighting change, as long as the changes are within the distribution of the training data, the encoder can map them to the corresponding area in the latent space, and the decoder can still generate the correct action. This provides robustness to scene changes. For the same observation condition, by sampling different latent variables z, multiple different but effective action strategies (such as grabbing from the left or right) can be generated, providing the system with a reserve of strategies to deal with uncertainty.The action block sequence is an ideal intermediate representation layer between the original joint trajectory and the natural language instruction, and provides an operable intermediate representation for high-level error correction and planning. When the error detection module alarms, the system can locate which action block executes the problem (such as the spatial deviation of the “grasp” block exceeds the standard), so as to replace or adjust the action block in a targeted manner, rather than re-plan the entire long action sequence, so that the error correction is more efficient. The high-level task decomposition module (such as a large model) only needs to output the semantic label sequence of the action block (such as [MOVE_TO, GRASP, MOVE_TO, POUR]), and the CVAE is responsible for filling in the specific and instantiated action parameters under each label. This realizes the clear division of labor and efficient cooperation between high-level semantic planning and low-level operation generation.

[0028] Traditional methods (such as deep reinforcement learning) are difficult to maintain consistency in long-time and continuous action sequences, lack the ability to capture long-time sequence dependence of complex continuous tasks, and once the traditional trajectory planning method deviates from the preset trajectory, it is difficult to dynamically evaluate the cumulative effect of errors, which is easy to cause small errors to accumulate into big problems.

[0029] Based on the above problems, the setting of the time accumulation error threshold in the embodiment of the present application includes: comparing the current trajectory with the reference trajectory in the training demonstration based on the sliding window similarity calculation of the historical motion trajectory of the mechanical arm and the current execution trajectory, and accumulating the time error according to the similarity distance. The time accumulation error threshold based on the sliding window similarity calculation is not only a detection index, but also an error measurement method with time sequence perception ability, and its effect is reflected in the following aspects: the execution rhythm and process consistency are monitored, which not only checks whether the mechanical arm reaches the correct position, but also checks whether it passes through the key nodes of the task in the correct way and within the correct time window; early warning ability is provided to prevent error accumulation, through the sliding window similarity distance calculation and the time error accumulation, the system can issue a warning when a slight deviation appears and has not caused a significant error in space position, a slight deviation at a single time point may not be a concern, but if the similarity of multiple time windows is decreasing (error is accumulating), it indicates that the mechanical arm is systematically and continuously deviating from the ideal execution mode. This accumulation judgment is more forward-looking than the absolute position judgment at a single moment, and can trigger error correction before the problem becomes irreparable (such as hitting an obstacle); the robustness to dynamic interference and model uncertainty is enhanced, the current trajectory is compared with the reference trajectory in the training demonstration, which is essentially the optimal experience of expert demonstration as a standard, when slight disturbances occur in the environment (such as slight sliding of objects, slight slipping of joints), the real-time trajectory of the mechanical arm will be compared with the statistical characteristics (such as average trajectory or probability distribution) of multiple demonstration trajectories. This data-driven comparison is more tolerant of uncertainty than relying on an accurate physical model, as long as the real-time trajectory is still within the "reasonable" experience distribution range, there will be no false alarm; a more interpretable input is provided for the error correction mechanism, the output time accumulation error is a quantitative, time sequence context value, which provides key information for subsequent error correction action generation, the error correction model, i.e. Markov decision process, receives signals such as low trajectory similarity in the middle of the task, which is more helpful for decision-making than receiving a position deviation of X centimeters. It can infer that: the current may encounter unforeseen friction or the grasping posture may have shifted, thereby generating a more targeted correction strategy (such as adjusting the grip force or re-aligning).

[0030] Traditional control methods (such as sliding mode control) only focus on the trajectory of the mechanical arm itself, ignoring the semantic space relationship between it and the task target, environmental objects, and lack the ability to understand complex task semantics and environmental information. It may only ensure smooth joint trajectory, but cannot guarantee the accuracy of the relative pose of the end effector and the target object at all times, making it difficult to ensure the smoothness and accuracy of the action sequence, and the pre-defined trajectory cannot adapt to changes in the position of the target object or the appearance of new obstacles in the environment. Relying on pre-defined trajectories, it lacks response to dynamic environments.

[0031] Based on the above problems, in the embodiment of the present application, the setting of the spatial deviation threshold value comprises: calculating the spatial distance deviation of the current execution trajectory and the expected trajectory based on the spatial relationship of the current position of the mechanical arm, the task target position and the objects in the environment. The spatial deviation threshold value calculated based on the spatial relationship is a spatial constraint with environment perception and task guidance, and its effect is reflected in the following aspects: the precision control related to the task rather than the absolute coordinates is realized, and the core of the deviation calculation is the spatial relationship of the task target position and the objects in the environment, which means that the system does not monitor whether the mechanical arm strictly follows a preset spatial curve, but whether the execution action meets the geometric conditions required to complete the current subtask. For example, in the subtask of "inserting the plug into the socket", the key spatial relationship is the alignment degree of the plug axis and the socket hole axis, as well as the distance between them. Even if the trajectory of the mechanical arm does not completely coincide with the demonstration trajectory, as long as this relative relationship (alignment degree and distance) is within the allowable threshold, it is not considered an error. On the contrary, even if the trajectory is smooth, but the plug is finally skewed, it will trigger error correction due to the excessive spatial relationship deviation. The adaptability to dynamic environment and target changes is enhanced, by calculating the relationship with the current task target and the objects in the environment in real time, the system can cope with the movement of the target position or the change of the environmental layout, if the target object is moved by a person, the spatial deviation from the current position to the target position detected by the system will continue to exist and may exceed the standard. At this time, the system will not rigidly require the mechanical arm to return to the original trajectory, but will trigger error correction based on the new spatial relationship (new target position) to re-plan a path to reach the new position. This gives the system the ability to adapt to environmental changes online. Collision and unsafe operation are prevented, by including the spatial relationship of the objects in the environment in the calculation, the system can actively avoid collisions with non-target objects (such as obstacles, other equipment, workbench edges), and the spatial deviation threshold value can include safety distance constraints. For example, in the task of "getting around the obstacle to get the object", the system will calculate the distance between the mechanical arm and the obstacle in real time. If the current trajectory causes this distance to be less than the safety threshold (i.e. the spatial relationship becomes dangerous), even if the trajectory is correct in terms of pointing to the target, it will trigger error correction due to the violation of the safe spatial relationship, guiding the mechanical arm away from the obstacle. Precise stage switching criteria are provided for multi-stage tasks, in a continuous task composed of multiple subtasks (such as "approach-grab-move-place"), the completion mark of each subtask is essentially the achievement of a spatial relationship state, the completion mark of the "grab" subtask may be that the distance between the end effector and the object surface is less than 1 cm and the attitude is aligned. By continuously monitoring this spatial relationship, the system can accurately determine that the current subtask has been completed when the condition is met, and smoothly switch to the next "move" subtask. This is more reliable and accurate than simply relying on time or joint position to determine.

[0032] In the embodiment of the present application, the complete multi-modal spatio-temporal information fusion asynchronous reasoning decision assistance is as follows: Prior to this, a multimodal large-scale model is used to encode the visual images and natural language commands of the robotic arm's working environment, extracting temporal and spatial features to achieve a deep understanding of the task objectives and environmental states. Simultaneously, an action segmentation algorithm is used to generate action sequences. Leveraging the planning and reasoning capabilities of the multimodal large-scale model, a biomimetic neural cerebellum structure is constructed. The cerebellum uses the ACT algorithm to quickly generate action sequence predictions for rapid action, while the brain is a multimodal spatiotemporal auxiliary decision-making model that first generates high-level task planning, breaking down the task into step-by-step plans for execution actions and movement. Furthermore, the spatiotemporal continuous understanding capability of the multimodal large-scale model guides the current action. The specific process is as follows: Figure 3 As shown:

[0033] When a task begins, the multimodal spatiotemporal auxiliary decision-making mechanism generates a task plan, and then the action segmentation algorithm generates and executes precise actions. During execution, overfitting behavior of the action segmentation algorithm and erroneous behavior caused by incorrect data acquisition are detected by the inference of the multimodal spatiotemporal auxiliary decision-making mechanism (the system continuously inputs visual images, robotic arm state data, and current action labels during operation; then the multimodal large model infers the relationship between the target and the robotic arm in the image, predicting action labels and robotic arm state information. This is then compared, and if there is a discrepancy between the inference and the actual behavior for a period of time, an error correction mechanism is activated to determine whether to take over). The current error is then interrupted, and the multimodal spatiotemporal auxiliary decision-making mechanism takes over action control. Previously, the action segmentation algorithm controlled the robotic arm using the imitation learning data corresponding to the instructions generated by the multimodal large model, moving the robotic arm to the inferred reasonable execution position and posture before using the action segmentation algorithm to generate the action prediction sequence.

[0034] To effectively correct errors during task execution, the system must correlate the robot's current behavior with historical demonstrations. This is addressed by comparing the similarity between the robot's current trajectory and the training demonstrations. A fixed window size W is defined, and the robot's most recent W state steps at deadline t are represented as follows: For each demonstration trajectory Perform a sliding window search to calculate Recent clips The similarity between candidate segments. For Each candidate index in Calculate the distance between the current trajectory window and the corresponding reference window. Distance as a cost .

[0035] , This segment represents the current demonstration trajectory. The cost of space evaluation in this index The motion trajectory of the current state represents the “current state”.

[0036] Since long-term tasks usually produce long trajectories, sliding window search can become computationally expensive, and similar trajectory windows can appear in different task stages. To solve this problem, a hierarchical strategy is adopted by limiting the candidate index to those whose associated stage labels and the most recently identified stage label satisfy , thereby ensuring that the search is limited to adjacent task stages. Then, the reference index and the reference trajectory are selected to minimize the distance:

[0037] , The task stage related to is then used to update the current task stage and retrieve the corresponding from the memory prompt library. This retrieved prompt reflects the expert behavior memory in the most similar historical situation encountered from the demonstration.

[0038] Considering that traditional methods such as trajectory optimization and point-by-point regression often have a trade-off between smoothness and accuracy, it is difficult to ensure the smoothness and accuracy of the action sequence, and the model-based method requires a lot of manual adjustment and is sensitive to model errors, relies on accurate dynamic models and parameter tuning, and has limited adaptability and flexibility. Generation models such as CVAE, whose output may be limited to the distribution of training data, lack the ability to further optimize from approximately correct to global optimal, and the deterministic bias or pattern collapse of action generation models.

[0039] ​In the embodiment of the present application, after the mechanical arm action sequence corresponding to each subtask is generated, the training of the predicted mechanical arm motion sequence by the neural network is further included: the flow matching method is used to correct the action sequence, a smooth and accurate action vector field is generated, and the mechanical arm is controlled to execute. The flow matching method is introduced, which is not a simple smoothing filtering of the generated action, but a rough-to-fine action distillation based on the probability flow theory. First, it realizes global optimal smoothing instead of local average smoothing. Traditional smoothing methods (such as low-pass filtering and spline interpolation) average data in a local window, which can remove noise but may damage the accuracy of critical path points (for example, making the grasping point vague). Flow matching learns a deterministic vector field from a noisy distribution to a real data distribution. During inference, it starts from a random noise and integrates along the vector field (i.e. probability flow). This process naturally finds a trajectory from the starting point to the ending point that satisfies the maximum probability (i.e. the most consistent with the demonstration data rules) and the minimum energy. The result is a trajectory that is extremely smooth overall and can accurately pass through all critical nodes, fundamentally solving the contradiction between smoothing and accuracy. Second, it does not require an accurate robot dynamics model. By learning from successful demonstration data, it can implicitly master the physical and geometric constraints that “good actions” should follow (such as continuous acceleration, avoiding singular points, and energy saving), providing a model-free action optimizer. The vector field learned by flow matching is essentially the “consensus direction” of all successful trajectories in the demonstration data. Following it, the generated trajectory is the most probable in the data sense, i.e. the most like the expert's demonstration and the most likely to succeed. The action sequence generated in the previous step (such as CVAE) may be only one of the feasible solutions, even with slight imperfections. As a post-processing optimizer, flow matching can attract these feasible solutions to the optimal solution manifold that is smoother, more robust, and higher in probability quality, which is equivalent to a smart purification of the action. Finally, it has strong error correction and repair capabilities. Even if the initially generated action sequence has local disturbances, non-physical mutations, or slight deviations, the flow matching process can effectively pull it back to the reasonable trajectory manifold. During training, the flow matching model has seen how various noisy and imperfect trajectories evolve into perfect trajectories. Therefore, during inference, it can identify the unreasonable parts in the current sequence and correct them based on the learned vector field. This adds a strong fault tolerance and robustness barrier to the system.

[0040] Specifically, the flow matching method includes: Supervised training of the noisy action sequence by the conditional flow matching loss function to learn a vector field that corrects the noisy action to the correct action; In the inference phase, a smooth action sequence is gradually generated by forward Euler integration starting from random noise.

[0041] Efficient, one-step-to-multi-step sampling inference is achieved. As mentioned above, the precision and computational cost of generating trajectories can be flexibly controlled by forward Euler integration. Compared with diffusion models that require iterative sampling, flow matching can usually generate high-quality samples with fewer steps (even theoretically one step), significantly reducing the computational delay when executed online, meeting real-time requirements. The number of integration steps becomes a tuning knob. In the stage where fast response is required, a small number of steps can be used to generate a rough but immediate trajectory; in the critical stage where fine operation is required, the number of steps can be increased to generate a more accurate and smooth trajectory. This provides a flexible trade-off between computational resources and action quality.

[0042] The complete steps of training the predicted robot motion sequence by the neural network are as follows: In the implementation of the algorithm, first, the robot collects teaching data through teleoperation, and the data input of each data frame is: two RGB images, each image has a resolution of 480*640, and the joint angle position data of the robot, each robot has six joints, and the degree of freedom is 6-dimensional. The output of the generated Batch size is the absolute joint position of the current robot, a 6-dimensional vector. Therefore, in the action block strategy, a k*6-dimensional vector is output under the current observation condition, and the k batches corresponding to the k*6-dimensional vector are the current block action sequence. Secondly, the style variable Z is generated. First, the initialized learning weight CLS is composed, and then a linear layer is used to project the robot to the joint position of the embedding dimension, expand the 6-dimensional to 512-dimensional, and obtain the joint dimension sequence. Then, another linear layer is used to project the k*6-dimensional action sequence to the embedding dimension of the action sequence, and the dimension of the k*6-dimensional action sequence is upgraded to k*512-dimensional. The above three outputs finally form a sequence with dimension embedding. Finally, only the first output is taken, corresponding to the [class] label, and another linear network is used to predict the mean and variance of the distribution of Z, which is parameterized as a diagonal Gaussian distribution. Finally, the reparameterization is used to obtain the sample of Z. The specific generation process is shown in Figure 5 Then, the generated action sequence is issued to the robot for execution using the flow matching method. Flow matching: a training method for generative models, aiming to let the model learn to correct "noisy actions" back to "correct actions". This method forms a "smooth vector field" to help the model predict actions more stably and accurately. Their model can control the robot data distribution Modeling, where , as the future action sequence for the action block Each action in the action block has a corresponding (I think noisy) action token, which is converted by the action expert, represents the input noisy action, which is equivalent to the action with noise, where the subscriptt Indicates the robot's time step, superscript Indicates the time step of stream matching. It is an observation, consisting of multiple RGB images, a voice command, and the robot's body state, namely:

[0043] , in It is the first Zhang (There are 2 images) It is a series of language tokens. The vectors of joint angles are trained using a conditional flow matching loss function to supervise these action tokens. , The former in the above formula For learning the network, it is equivalent to the noise in the prediction, the latter The learning objective is equivalent to adding real noise, i.e., training the former to approximate the latter. For parameters The loss function is a variable whose goal is to minimize the prediction error by adjusting the parameters. It is the weighted average of the joint expectation of the probability distribution over all possible action sequences and their sampled transformations.

[0044] When flow matching is combined with a simple linear Gaussian probability path, powerful empirical results can be achieved, as given by the following expression: , in, It is a conditional probability distribution, representing the probability distribution under a given sequence of actions. Under the condition of action sequence Sampling, This represents a multivariate Gaussian distribution with mean 1. The covariance matrix is .

[0045] The first step is usually for the network to randomly sample noise that conforms to a normal distribution. Conduct training to calculate "noisy movements". ; Step 2: Then train the network output. This is the vector field representation of the action block. The noise representing the prediction is used to match the denoised vector field. , This represents the added real noise, hence the loss function mentioned above. Essentially, after obtaining the added real noise, we can use this formula... Calculations yielded At inference time, actions are generated by integrating the learned vector field from to , starting from random noise , using the forward Euler integration rule where are the input and output of the integration step, and the input tokens correspond to the noise action block , with the number of tokens equal to the action prediction horizon, the model uses transformer outputs corresponding to H noise actions, which are decoded into by a linear projection.

[0046] For each noisy action , the corresponding embedded expression input to the transformer is: where is a sinusoidal position encoding function, , d is the action dimension, w is the embedding dimension of the action expert, W 1 is a weight matrix 1, W 2 is a weight matrix 2, W 3 is a weight matrix 3.

[0047] The present application significantly improves the action accuracy and generalization ability of the robot arm in continuous tasks by fusing multi-modal spatio-temporal information and action block generation mechanism. Experimental verification shows that this method can effectively alleviate the overfitting problem existing in traditional imitation learning, reduce the occurrence rate of incorrect actions, improve the task success rate and execution stability. The automatic action correction mechanism enhances the adaptability of the robot arm to environmental changes, improves the robustness and practicality of the robot arm in complex environments, and is suitable for various application scenarios such as industrial manufacturing and service robots.

[0048] Specifically, the error correction action generation mechanism includes: using a multi-modal large model to generate a basic action prediction and a memory prompt action prediction based on historical demonstrations; calculate the weight of the two by Softmax normalization, and dynamically integrate to generate the final error correction action sequence.

[0049] The relationship feature between the time feature and the space feature is extracted by the multi-modal spatio-temporal decision-making model, a time cumulative error threshold is set, and based on the history actions that have occurred in the time feature, a dynamic weight evaluation is performed for the current action and the expectation. The action prediction weight evaluation process is: based on the current action sequence and the expected action sequence marked for evaluation. This action sequence encapsulates the action prior knowledge about the best matching demonstration future action under similar circumstances. According to the difference between the current action and the expected action, it is compared with the predicted demonstration action, and the dynamic weight coefficient is calculated by softmax normalization to evaluate the weight. According to the spatial feature, the spatial deviation of the current action and the expected action is dynamically evaluated; if the action error is detected to exceed the error threshold, the current action is paused, the robot arm returns to the previous action position, and the multi-modal auxiliary decision-making re-predicts the correction action block to take control.

[0050] The complete error correction action generation is as follows: Due to the strong benchmark performance of LLaMA and the ability to generate complete sequence embeddings, the LLaMA model is used to generate embeddings from natural language instructions. During the pre-training process, the LLaMA model combines masked language modeling and permutation language modeling, thereby realizing embedding that captures global and local semantic information.

[0051] In order to enable the action block algorithm model to interpret and execute tasks based on natural language instructions, first, the text embedding needs to be converted to the input dimension of the encoder, = 512. This is achieved by projecting the text embedding to the required dimension using a linear projection layer, denoted as:

[0052] where, is the original embedding of the natural language instruction, is the weight of the projection layer, is the bias of the projection layer, is the mapped vector, which serves as the input for subsequent calculations of the model. is a real matrix with element dimension = 512, is the dimension before mapping.

[0053] In addition, the position embedding of the model needs to be updated to adapt to the new input. The ACT model applies position information to the encoder input, where one position encoding is used for visual features and another position encoding is used for joint positions. A third position encoding is introduced for the text embedding to ensure that the model can distinguish between input types during processing, denoted as:

[0054] where, ​​is a natural language instruction translation action pose encoding. , is a code matrix for expanding the position encoding.

[0055] Finally, the projected text embeddings are incorporated into the Transformer encoder by concatenating them with the existing input sequence. Specifically, the text embeddings are concatenated to the input sequence, which includes the latent variables from the CVAE and the proprioceptive data of the robotic arm

[0056] , where, is the mapped vector, z is the latent style variable generated by the CVAE, denotes the features generated by the visual encoder, is the input sequence, is the current joint position of the robotic arm. The combined input sequence is then processed by the Transformer encoder to generate a latent representation, which is used by the decoder to generate the action sequence.

[0057] By projecting the text embeddings to the appropriate dimension, appending them to the encoder input, and applying modified position embeddings, it is possible to successfully integrate natural language instructions into the action segmentation model. These modifications enable the model to learn to associate natural language instructions with corresponding tasks and to adjust the generated action sequence according to the task instructions.

[0058] Memory cues provide valuable memories from past demonstrations for the current execution phase, while base cues are trained throughout the task, providing more extensive task-level generalization capabilities. However, neither party alone is sufficient to meet the demand. Retrieved memories are susceptible to retrieval inaccuracies, ambiguous execution phases, and mismatched training demonstration phases. In contrast, base cues are based solely on current observations and task instructions and are not affected by such errors but lack a historical basis and long-term memory. To reconcile these complementary advantages, a memory-aware cue integration mechanism is introduced that dynamically combines the phase specificity of memory cues with the task generalization capabilities of base cues. At each time step , the frozen action segmentation algorithm model produces two action predictions: one using base cues , and one using retrieved phase-specific memory cues . Simultaneously, in the demonstration , the reference action sequence is retrieved from the index starting at the index ​The action sequence encapsulates valuable action prior knowledge about the best matching demonstration future action in similar situations. However, it is infeasible to directly execute due to the discrepancy between the current task and the historical demonstration. Instead, it is used as an action prior to dynamically weight and . Specifically, the predictions are compared with the retrieved demonstration action and the dynamic weight coefficients are calculated by softmax normalization:

[0059] , where is the weight coefficient, is the action prediction under the specific prompt, is the action prediction generated by the base model. is the demonstration action for reference. is the exponential function.

[0060] Using the calculated , the final execution time-dependent action is given by: , where is the action sequence finally executed at time step t, is the weight coefficient, is the action prediction under the specific prompt, is the action prediction generated by the base model.

[0061] This prompt set and column mechanism encourages the strategy to tend to be closer to the prediction of the retrieved expert action, effectively utilizing the future action prior in the demonstration, while not sacrificing the adaptability of the model to the current scene. By dynamically balancing the task-level generalization of the base prompt and the stage-specificity of the retrieval prompt, the model can maintain robustness to retrieval inaccuracy, improve tolerance to ambiguous stage boundaries, and maintain adaptability to the scene at the dynamic test time, thereby achieving more stable and accurate behavior during task execution.

[0062] The second part of the embodiment of the application provides a robot arm control system, comprising: a multi-modal perception module for acquiring visual images and natural language instructions; a task decomposition and planning module for task decomposition and action sequence generation based on a multi-modal large model; an error detection and correction module for real-time detection of action errors and triggering of a correction mechanism; The system is used to implement the robot arm control method described above.

[0063] The third part of the embodiment of the application provides an electronic device, comprising: at least one processor; a memory storing a computer program; When the computer program is executed by the processor, the mechanical arm control method described above is realized.

[0064] The fourth part of the embodiment of the application provides a computer readable storage medium storing a computer program, when the computer program is executed by the processor, the mechanical arm control method described above is realized.

[0065] Simulation results: As Figure 6 and Figure 7 shown, the simulation experiment environment shows the tracking comparison of the actual execution angle and the predicted angle of the model in the real experiment environment, and the results show that the two are highly consistent, reflecting the accuracy and stability of the model in the actual execution. Figure 8 and Figure 9 respectively show the comparison of the teaching trajectory and the model predicted trajectory under the condition that the initial positions of the objects are the same. Although there is a certain difference between the trajectories, the tasks are successfully completed, which reflects the good generalization ability and adaptability to environmental changes of the model.

[0066] Detailed explanation of the experiment: In the real environment experiment, four-channel recording of the joint angles, angular velocity and torque of the guide arm and slave arm is used, and the control frequency is 1000Hz. At the same time, each camera acquires images at a frequency of 100Hz. The success rate of the task and the average time to complete the task are used to evaluate the model index.

[0067] The evaluation task content is as follows: using the same data set and parameters for training, one minute of task execution is carried out in the same environment, the task is composed of two consecutive tasks, and it is required to complete them in order. Record the joint angle curve under the change of time. In the continuous grasping and placing task, the mechanical arm using the method of the present application has higher success rate, fewer error actions, smooth and stable action trajectory compared with the traditional algorithm, which verifies the effectiveness and superiority of the method.

[0068] The application provides a mechanical arm control method, system, electronic equipment and readable storage medium, which adopts a hierarchical system control mechanism to efficiently manage and execute complex tasks of the mechanical arm. The mechanism divides the mechanical arm control task into multiple levels of cooperative work, each level undertakes different functions and responsibilities, and through information interaction between upper and lower levels, intelligent planning, precise execution and dynamic error correction of the mechanical arm are realized. Specifically, the highest level is a high-level planning layer, responsible for fusing multi-modal perception information including visual images, natural language instructions and mechanical arm states, using a multi-modal large model to deeply understand and semantically analyze the task target, complete the decomposition and high-level action planning of the complex task, and generate action block instructions for step-by-step execution. The lower layer is an execution layer, responsible for specific action control and trajectory planning of the mechanical arm joints, real-time collection of state information such as joint angle, speed and torque of the mechanical arm and environmental feedback, and feedback of the execution result to the upper layer. The error correction auxiliary layer across layers continuously monitors the correctness of action execution, compares and predicts the action through a multi-modal spatio-temporal auxiliary decision model, discovers deviations and errors in time, suspends the current action block if necessary, triggers the auxiliary decision module to take control, re-plans reasonable actions, guides the mechanical arm to restore to a safe posture and continues to execute the task. The hierarchical system control mechanism realizes multi-layer cooperation with clear division of labor and clear responsibilities, which not only guarantees the global vision of high-level task planning, but also takes into account the real-time response ability of low-level action generation and the fine control of the execution layer, and at the same time, the dynamic error correction mechanism is supplemented, which significantly improves the adaptability, robustness and task completion rate of the mechanical arm in complex environments.

[0069] The above application is only a few specific embodiments of the application, but the embodiments of the application are not limited thereto, and any changes that can be thought of by those skilled in the art should fall within the protection scope of the application.

Claims

1. A robot arm control method characterized by, The method comprises the following steps: acquiring a visual image of a working environment of a robot arm and a natural language task instruction, and extracting time features and space features of the environment through a multi-modal large model; performing high-level planning on the task based on the multi-modal large model, decomposing the task into multiple sub-tasks, and generating a robot arm action sequence corresponding to each sub-task through an action block algorithm; in the execution process of the robot arm, real-time extraction of relationship features of the time features and the space features is performed through a multi-modal spatio-temporal auxiliary decision model, and a preset time cumulative error threshold and a space deviation threshold are combined to detect an action execution error; if the action execution error is detected to exceed the threshold, the current action is paused, the robot arm is controlled to return to a previous safe position, and a correction action generation mechanism is triggered; a Markov decision process is used to model the correction action, a self-attention mechanism of the multi-modal large model is combined, visual image, natural language instruction and robot arm state information are fused, a corrected action sequence is generated, and the robot arm is controlled to execute again.

2. The robot control method according to claim 1, wherein The action block algorithm uses a conditional variational autoencoder, and the input includes a multi-channel two-dimensional matrix corresponding to the visual image, joint state information of the robot arm and a task instruction label; the output is a k action block sequence corresponding to the current observation condition, and each action block corresponds to a group of joint position control instructions of the robot arm.

3. The robot control method according to claim 1, wherein The setting of the time cumulative error threshold comprises: a sliding window similarity calculation is performed based on a historical action trajectory of the robot arm and a current execution trajectory, the current trajectory is compared with a reference trajectory in a training demonstration, and a time error is accumulated according to a similarity distance.

4. The robot control method of claim 1, wherein The setting of the space deviation threshold comprises: based on the current position of the robot arm, the target position of the task and the spatial relationship of the objects in the environment, the spatial distance deviation of the current execution trajectory and the expected trajectory is calculated.

5. The robot control method of claim 1, wherein After the robot arm action sequence corresponding to each sub-task is generated, the following steps are further included: a flow matching method is used to correct the action sequence, a smooth and accurate action vector field is generated, and the robot arm is controlled to execute.

6. The robot control method according to claim 5, wherein The flow matching method comprises: a conditional flow matching loss function is used to supervise the training of the action sequence with noise, and a vector field for correcting the action with noise to a correct action is learned; in the inference stage, a smooth action sequence is gradually generated through forward Euler integration starting from random noise.

7. The robot control method of claim 1, wherein The correction action generation mechanism comprises: a multi-modal large model is used to generate a basic action prediction and a memory prompt action prediction based on historical demonstrations; the weights of the two are calculated through Softmax normalization, and a final correction action sequence is dynamically fused and generated.

8. A robot control system, characterized by It comprises: a multi-modal perception module for acquiring a visual image and a natural language instruction; a task decomposition and planning module for task decomposition and action sequence generation based on a multi-modal large model; an error detection and correction module for real-time detection of action errors and triggering of a correction mechanism; The system is used to implement the robot arm control method in any one of claims 1 to 7.

9. An electronic device, comprising: It comprises: at least one processor; a memory storing a computer program; when the computer program is executed by the processor, the robot arm control method in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that, The computer program is stored, and when the computer program is executed by a processor, a mechanical arm control method as claimed in any one of claims 1 to 7 is implemented.

Images