Mechanical arm control method, server, equipment, system and medium

Abstract

The invention provides a mechanical arm control method, a server, equipment, a system and a medium, and relates to the technical field of industrial automation. According to the method, a cloud server responds to an input target task operation instruction, and a full-view observation image of a mechanical arm operation scene is obtained; the target task operation instruction and the full-view-angle observation image are input into a preset multi-mode large model, and a mechanical arm action planning instruction is obtained; the mechanical arm action planning instruction is sent to terminal equipment; the terminal equipment responds to a mechanical arm action planning instruction sent by the cloud server and obtains a current observation image of the mechanical arm; the mechanical arm action planning instruction and the current observation image of the mechanical arm are input into a preset action prediction model, and a corresponding action sequence is obtained; and controlling the mechanical arm to execute the target task according to the action sequence. According to the method, the generalization, migration ability and accuracy of mechanical arm control are improved through end-cloud cooperative control.

Description

Technical Field

[0001] The present application relates to the field of industrial automation technology, and in particular to a robot arm control method, server, device, system and medium. Background Art

[0002] Multimodal Large Language Model (MLLM) is an artificial intelligence model that can process and generate multiple types of data (such as text, images, videos, audio, etc.). It is widely used in various fields, such as robotic arm control in the industrial field.

[0003] At present, the control of robotic arms in the industrial field only relies on the teaching pendant, and technicians need to use programming to define the fixed robotic arm movement path and each path point. However, this method has low programming, poor adaptability and high requirements on the ability of technicians. Therefore, the relevant technology introduces a multimodal large model to control the robotic arm. The multimodal large model loaded in the computing device can use methods such as deep reinforcement learning to learn new strategies from examples of robotic arm movement, and combine it with visual, language and other modalities for task planning and scene understanding, predicting the atomic task sequence of the robotic arm, and the computing device can control the robotic arm according to the predicted atomic task sequence.

[0004] However, large multimodal models based on deep reinforcement learning require a stable reward function, while the reward function usually needs to be adjusted in actual tasks, and the model pre-training process is complicated. The smaller model size cannot learn a large number of strategies, resulting in poor generalization, migration ability and accuracy of robotic arm control. Summary of the invention

[0005] Embodiments of the present application provide a robotic arm control method, server, device, system and medium, which can improve the generalization, migration capability and accuracy of robotic arm control.

[0006] In a first aspect, an embodiment of the present application provides a robotic arm control method based on a multi-modal large model, which is applied to a cloud server, including:

[0007] In response to the input target task operation instruction, a full-view observation image of the robot arm operation scene is obtained;

[0008] Inputting the target task operation instruction and the full-view observation image into a preset multimodal large model to obtain a robot arm motion planning instruction;

[0009] The robot arm motion planning instruction is sent to the terminal device, and the robot arm motion planning instruction is used to instruct the terminal device to generate a corresponding action sequence according to the robot arm motion planning instruction and the current observation image of the robot arm, and control the robot arm to perform the target task based on the action sequence.

[0010] In a possible implementation, it further includes:

[0011] Step S1: after the robot arm motion planning instruction is sent to the terminal device for a preset time, a full-view feedback image of the robot arm operation scene is obtained;

[0012] Step S2: inputting the target task operation instruction, the full-view feedback image and the robot arm motion planning instruction into the multimodal large model to obtain an updated robot arm motion planning instruction;

[0013] Step S3: Send the updated robot arm motion planning instruction to the terminal device, and repeat the above steps S1-S2.

[0014] In a possible implementation, the resolution of the full-view observation image / full-view feedback image is greater than the resolution of the current observation image of the robotic arm.

[0015] In a second aspect, an embodiment of the present application provides another method for controlling a robotic arm based on a multimodal large model, which is applied to a terminal device, including:

[0016] In response to the robot arm motion planning instruction sent by the cloud server, the current observation image of the robot arm is obtained, wherein the robot arm motion planning instruction is obtained by the cloud server responding to the target task operation instruction, obtaining the full-view observation image of the robot arm operation scene, and inputting the target task operation instruction and the full-view observation image into a preset multi-modal large model;

[0017] Inputting the robot arm motion planning instruction and the current observation image of the robot arm into a preset motion prediction model to obtain a corresponding motion sequence;

[0018] The robotic arm is controlled to perform a target task according to the action sequence.

[0019] In a possible implementation manner, the action prediction model is trained in the following manner:

[0020] Acquire sample data of robot arm operation, wherein the sample data is obtained based on multiple robot arm fine-tuning tasks;

[0021] The initial model is transferred and trained according to the sample data of the robot arm operation and the low-rank adaptive strategy to obtain an action prediction model. The initial model is an end-to-end visual language action large model.

[0022] In a possible implementation, the acquiring sample data of the robot arm operation includes:

[0023] Obtaining fine-tuning task instructions and a motion trajectory of the robotic arm corresponding to each fine-tuning task of the robotic arm, wherein the motion trajectory of the robotic arm includes a motion trajectory obtained by performing a robotic arm demonstration in a virtual environment and a motion trajectory obtained by performing a robotic arm demonstration in a real environment;

[0024] Performing semantic expansion on the fine-tuning task instruction to obtain an expanded task instruction;

[0025] Acquire a scene observation image of the end effector of the robot arm according to different camera postures, objects to be operated, and scene environments;

[0026] Sample data of the robot arm operation is generated according to the robot arm motion trajectory, the augmented task instructions and the scene observation image.

[0027] In a third aspect, an embodiment of the present application provides a cloud server, including:

[0028] A first processor, and a first memory communicatively connected to the first processor;

[0029] The first memory is used to store computer-executable instructions;

[0030] The first processor is used to execute computer-executable instructions stored in the first memory, so that the first processor executes the above-mentioned first aspect and / or various possible implementations of the first aspect.

[0031] In a fourth aspect, an embodiment of the present application provides a terminal device, including:

[0032] a second processor, and a second memory communicatively connected to the second processor;

[0033] The second memory is used to store computer-executable instructions;

[0034] The second processor is used to execute the computer-executable instructions stored in the second memory, so that the second processor executes the above-mentioned second aspect and / or various possible implementations of the second aspect.

[0035] In a fifth aspect, an embodiment of the present application provides a terminal-cloud collaborative control system, comprising: a cloud server as described in the third aspect above, and a terminal device as described in the fourth aspect above.

[0036] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the robotic arm control method based on a multimodal large model as described in the first aspect or the second aspect above.

[0037] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the robotic arm control method based on a multimodal large model as described in the first aspect or the second aspect above.

[0038] The embodiment of the present application provides a robot control method, server, device, system and medium, and provides a robot control solution based on a multimodal large model for end-cloud collaboration. In response to the input target task operation instruction, the cloud server can input the target task operation instruction and the full-view observation image into the preset multimodal large model for scene understanding and task planning, obtain the staged robot action planning instruction, and send the robot action planning instruction to the terminal device. In response to the robot action planning instruction sent by the cloud server, the terminal device can input the robot action planning instruction and the current observation image of the robot into the preset action prediction model for action prediction, and obtain a finer-grained action sequence. Through such a setting, in the process of end-cloud collaborative control based on the multimodal large model, the multimodal large model set by the cloud server can complete complex scene understanding and task planning, improve the accuracy of robot control, and the multimodal action prediction model set by the terminal device can realize fast migration and fine-tuning according to the current environment of the robot, and does not rely on reward functions and a large number of strategies. The models on both sides of the end and cloud are trained separately, which reduces the complexity of model training, improves the generalization and migration ability of robot control, and enables robot control to adapt to various scenarios and tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] Figure 1 A system architecture diagram of an embodiment of the present application;

[0041] Figure 2 This is a flow chart of a robotic arm control method based on a multi-modal large model according to an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the structure of a cloud server according to an embodiment of the present application;

[0043] Figure 4A schematic diagram of the structure of a terminal device according to an embodiment of the present application.

[0044] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0045] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0046] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0048] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0049] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0050] The robot arm control method, server, device, system and medium of the present application can be used in the field of industrial automation technology, and can also be used in any field outside the field of industrial automation technology, such as the field of robot arm control, etc. The application field of the robot arm control method, server, device, system and medium of the present application is not limited.

[0051] The robot arm control method, server, device, system and medium of the present application can be applied to scenarios in the industrial field where multimodal large models are used to control robot arms. The robot arm control method, server, device, system and medium of the present application can be applied to related scenarios in the industrial field where multimodal large models are used to control any robot arm and complete any industrial task.

[0052] First, the terms involved in this application are explained:

[0053] Multimodal Large Language Model (MLLM) is an artificial intelligence model that can process and generate multiple types of data (such as text, images, videos, audio, etc.). Unlike traditional single-modal large language models (such as the GPT series) that focus on text generation, multimodal models integrate multiple modal information such as vision and hearing, and achieve richer expression and interaction capabilities by understanding and associating different types of input. MLLM is based on large-scale pre-training methods and is usually trained with massive cross-modal data. The model can learn to establish connections between different modalities, such as converting image content into text descriptions, or generating corresponding images based on text. Typical multimodal models include GPT-4o, which have demonstrated outstanding performance in visual understanding and generation tasks. MLLM has been widely used in image annotation, text generation, intelligent assistants and other fields, promoting the innovation and development of cross-modal tasks. With the advancement of technology, MLLM will further enhance its ability to perceive and understand multiple data types, bringing more possibilities to human-computer interaction, content creation and other fields.

[0054] Currently, the control of robotic arms in the industrial field only relies on the teach pendant, which is a device used to program and control the movement of industrial robots (robotic arms). It is usually used in industrial automation environments so that operators can intuitively guide and program the robotic arms to perform specific tasks.

[0055] However, this method has some obvious disadvantages. First, the programming efficiency is low, and the operator needs to define each path point step by step. For complex tasks, this method is extremely cumbersome and time-consuming. Secondly, the programming of the teach pendant lacks flexibility and can usually only record fixed motion paths. Once the task or environment changes, the entire path may need to be rewritten, resulting in poor adaptability. In addition, the operation of the teach pendant relies on manual intervention and requires the operator to have certain professional skills, which is particularly limited in some tasks that require high precision and high complexity.

[0056] Therefore, the relevant technology introduces a multimodal large model to control the robotic arm. The multimodal large model loaded in the computing device can use methods such as deep reinforcement learning to learn new strategies from robotic arm motion examples, and combine it with visual, language and other modalities for task planning and scene understanding, and predict the atomic task sequence of the robotic arm. The computing device can control the robotic arm according to the predicted atomic task sequence.

[0057] However, the large multimodal model based on deep reinforcement learning requires a stable reward function, while the reward function usually needs to be adjusted in actual tasks, and the model pre-training process is complicated. The smaller model size cannot learn a large number of strategies, resulting in poor generalization, transferability and accuracy of robotic arm control across environments and multiple tasks. At the same time, it also lacks sufficient semantic understanding and complex reasoning capabilities.

[0058] Based on the above technical problems, the inventive concept of this application is: how to provide a robotic arm control solution based on a multimodal large model that can improve the generalization, migration ability and accuracy of robotic arm control.

[0059] The embodiments of the present application provide a robot control method, server, device, system and medium, which can improve the generalization, migration capability and accuracy of robot control through end-cloud collaborative control based on a multimodal large model. Specifically, the multimodal large model set up by the cloud server can complete complex scene understanding and task planning, improve the accuracy of robot control, and the multimodal action prediction model set up by the terminal device can achieve rapid migration and fine-tuning according to the current environment of the robot, and does not rely on reward functions and a large number of strategies. The models on both sides of the end cloud are trained separately, which reduces the complexity of model training and improves the generalization and migration capability of robot control.

[0060] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0061] Figure 1 This is a system architecture diagram of an embodiment of the present application, such as Figure 1 As shown, when the robot needs to be controlled to complete a task, the operator can send the target task operation instruction to the cloud server through the operation terminal. After receiving the instruction, the cloud server uses camera 1 to obtain the full-view observation image of the robot operation scene; inputs the target task operation instruction and the full-view observation image into the preset multimodal large model to obtain the robot action planning instruction; and sends the robot action planning instruction to the terminal device. The terminal device responds to the robot action planning instruction sent by the cloud server, and uses camera 2 to obtain the current observation image of the robot; inputs the robot action planning instruction and the current observation image of the robot into the preset action prediction model to obtain the corresponding action sequence; the terminal device generates a control signal according to the action sequence to control the robot to perform the target task.

[0062] Figure 2 This is a flowchart of a robotic arm control method based on a multimodal large model according to an embodiment of the present application. This embodiment uses a terminal-cloud collaborative control system as an execution subject to illustrate the robotic arm control method based on a multimodal large model. The terminal-cloud collaborative control system includes a cloud server and a terminal device. Figure 2 As shown, the robotic arm control method based on the multi-modal large model may include the following steps:

[0063] S201: The cloud server obtains a full-view observation image of the robot arm operation scene in response to the input target task operation instruction.

[0064] In this embodiment, the target task operation instruction may be an instruction for operating the target object that is remotely input by an operator to the cloud server, such as “spray component A”.

[0065] In this embodiment, a first camera device capable of capturing full-view observation images of the scene may be provided in the robot arm operation scene. After receiving the input target task operation instructions, the cloud server may utilize the first camera device to obtain full-view observation images of the robot arm operation scene.

[0066] S202: The cloud server inputs the target task operation instructions and the full-view observation image into the preset multimodal large model to obtain the robot arm motion planning instructions.

[0067] In this embodiment, the full-view observation image can be a full-view image of the robotic arm operation scene, including information such as the operating environment, the robotic arm, and the object to be operated. The multimodal large model preset in the cloud server can perform comprehensive and accurate scene understanding and reasoning based on the full-view observation image, and perform task planning and decomposition in combination with the target task operation instructions.

[0068] In this embodiment, the multimodal big model preset in the cloud server can be the existing GPT-4o big model or other multimodal big models, without any limitation.

[0069] In this embodiment, the multimodal large model does not rely on the visual information representation method, has higher robustness, and has a more accurate understanding of the image. Compared with directly performing task reasoning and scene understanding on the client side, the cloud-side model can better utilize its own large model's inherent knowledge to make better planning for tasks, and optimize the cloud-side model based on the embodied reasoning chain, which optimizes the thinking structure of the large model to a certain extent and helps with long-term reasoning tasks.

[0070] In this embodiment, the robot arm motion planning instruction may be a staged motion planning instruction of the robot arm in the target task operation. For example, the target task operation may be planned into several stages with finer granularity, such as "moving left", "moving down", "spraying operation", etc., and the robot arm motion planning instruction may be a text instruction of the "moving left" stage.

[0071] S203: The cloud server sends the robot arm motion planning instructions to the terminal device.

[0072] In this embodiment, the terminal device can be a control device for controlling the movement of the robotic arm in the robotic arm operation scenario, such as a computer, mobile phone and other devices. The specific type of the terminal device can be flexibly set by technical personnel in this field and is not limited here.

[0073] S204: The terminal device obtains the current observation image of the robotic arm in response to the robotic arm motion planning instruction sent by the cloud server.

[0074] In this embodiment, a second camera device capable of capturing images of the robot arm (end effector) may also be provided in the robot arm operation scene. After the terminal device receives the robot arm motion planning instruction sent by the cloud server, the second camera device may be used to obtain the current observation image of the robot arm.

[0075] S205: The terminal device inputs the robot arm motion planning instruction and the current observation image of the robot arm into a preset motion prediction model to obtain a corresponding motion sequence.

[0076] In this embodiment, the motion prediction model can predict the 6D relative position and gripper state of the robot end effector based on the robot motion planning instruction, and can obtain the corresponding motion sequence based on inverse kinematics planning.

[0077] In this embodiment, the action prediction model preset in the terminal device can be an end-to-end multimodal large model, such as an OpenVLA large model, an Octo large model, etc. The specific type of the action prediction model can be flexibly set by technical personnel in this field, as long as the action prediction model is an end-to-end multimodal large model.

[0078] In this embodiment, the action space of the action prediction model is the end effector posture, which is independent of the atomic task and can be quickly migrated through end-to-end fine-tuning.

[0079] In this embodiment, the current observation image of the robotic arm may be an image that only includes the robotic arm (end effector). The current action and specific environment of the robotic arm can be determined based on the current observation image of the robotic arm. The motion prediction model preset in the terminal device can perform environmental migration and fine-tuning on the robotic arm motion planning instructions based on the current observation image of the robotic arm, and predict a finer-grained and specific action sequence.

[0080] S206: The terminal device controls the robotic arm to perform the target task according to the action sequence.

[0081] In this embodiment, after the action prediction model predicts the action sequence corresponding to the robot arm action planning instructions at each stage, the terminal device can output a corresponding control signal according to the action sequence to control the robot arm to perform the target task corresponding to the target task operation instruction.

[0082] In this embodiment, a multimodal large model-based robotic arm control solution for end-cloud collaboration is provided. In response to the input target task operation instruction, the cloud server can input the target task operation instruction and the full-view observation image into the preset multimodal large model for scene understanding and task planning, obtain the staged robotic arm motion planning instruction, and send the robotic arm motion planning instruction to the terminal device. In response to the robotic arm motion planning instruction sent by the cloud server, the terminal device can input the robotic arm motion planning instruction and the current observation image of the robotic arm into the preset motion prediction model for motion prediction to obtain a finer-grained motion sequence. Through such a setting, in the process of end-cloud collaborative control based on the multimodal large model, the multimodal large model set by the cloud server can complete complex scene understanding and task planning, improve the accuracy of robotic arm control, and the multimodal motion prediction model set by the terminal device can realize fast migration and fine-tuning according to the current environment of the robotic arm, and does not rely on reward functions and a large number of strategies. The models on both sides of the end and cloud are trained separately, which reduces the complexity of model training, improves the generalization and migration ability of robotic arm control, and enables robotic arm control to adapt to various scenarios and tasks.

[0083] In a possible implementation, the method may further include:

[0084] Step S1: After sending the robot arm motion planning instruction to the terminal device for a preset time, the cloud server obtains a full-view feedback image of the robot arm operation scene.

[0085] Step S2: The cloud server inputs the target task operation instructions, the full-view feedback image, and the robot arm motion planning instructions into the multimodal large model to obtain updated robot arm motion planning instructions.

[0086] Step S3: The cloud server sends the updated robot arm motion planning instructions to the terminal device and repeats the above steps S1-S2.

[0087] In this embodiment, the preset time length can be flexibly set by those skilled in the art according to actual conditions, and no limitation is imposed here. For example, the cloud server can obtain a full-view feedback image of the robot arm operation scene 15 seconds after sending the robot arm motion planning instruction to the terminal device.

[0088] In this embodiment, the full-view feedback image can be a full-view image of the robot arm operation scene, including information such as the operating environment, the robot arm, and the object to be operated. The full-view feedback image and the full-view observation image can be collected by the same camera device. After the cloud server sends the robot arm motion planning instruction to the terminal device for a preset time, it can use the first camera device to obtain the full-view feedback image of the robot arm operation scene.

[0089] In this embodiment, each time the cloud server sends a robotic arm motion planning instruction to the terminal device, it will obtain a full-view feedback image of the robotic arm operation scene after a preset time period, and continuously adjust the next robotic arm motion planning instruction based on the feedback image after the actual action is completed.

[0090] In this embodiment, after the cloud server sends the robot arm motion planning instruction to the terminal device for a period of time, it can also obtain the full-view feedback image after the robot arm performs the corresponding action according to the robot arm motion planning instruction, and generate the next robot arm motion planning instruction according to the target task operation instruction, the full-view feedback image and the robot arm motion planning instruction. Through such a setting, the cloud server can continuously adjust the robot arm motion planning instruction sent to the terminal device according to the feedback image after the actual action is completed, thereby achieving the effect of closed-loop feedback of the cloud server-terminal device-cloud server, and further improving the accuracy of robot arm control.

[0091] In a possible implementation manner, the resolution of the full-view observation image / full-view feedback image is greater than the resolution of the current observation image of the robotic arm.

[0092] In this embodiment, the full-view observation image / full-view feedback image may be the original full-view image captured by the first camera device, and the cloud server does not process the image captured by the first camera device after acquiring it.

[0093] In this embodiment, the current observation image of the robot arm can be obtained by compressing the observation image of the robot arm collected by the second camera device by the terminal device, and the resolution is relatively low. Since the current observation image of the robot arm only needs to collect the robot arm, even if the resolution of the current observation image of the robot arm is relatively low, the comprehensiveness and observability of the image information can be guaranteed.

[0094] In this embodiment, after the terminal device acquires the observation image of the robotic arm using the camera device, the obtained image can be compressed to obtain a current observation image of the robotic arm with a lower resolution, thereby reducing the amount of data calculation of the motion prediction model in the terminal device and improving calculation efficiency.

[0095] In a possible implementation, the action prediction model in step S205 may be trained in the following manner:

[0096] S11: The terminal device obtains sample data of the robot arm operation, where the sample data is obtained based on multiple robot arm fine-tuning tasks.

[0097] S12: The terminal device performs migration training on the initial model based on the sample data of the robot arm operation and the low-rank adaptive strategy to obtain an action prediction model. The initial model is an end-to-end visual language action large model.

[0098] In this embodiment, the initial model can be an end-to-end visual language action large model, such as an OpenVLA model, or an Octo model, etc. Those skilled in the art can set it flexibly without any limitation, as long as the initial model is an end-to-end visual language action large model.

[0099] Exemplarily, the initial model can be an OpenVLA model. OpenVLA is pre-trained based on a large amount of robot trajectory data. Its large number of parameters and training samples make it a strong basic model for robot manipulation, with better multi-task generalization and cross-robot generalization capabilities. Its action space is the end effector posture, which does not depend on atomic tasks and can be quickly migrated through end-to-end fine-tuning. It has advanced cross-environment fine-tuning and multi-task processing capabilities. OpenVLA can be quickly migrated and fine-tuned (LoRA) based on imitation learning, so it does not rely on reward functions.

[0100] In this embodiment, the end-to-end visual language action model such as OpenVLA is pre-trained based on a large amount of robotic arm trajectory data. Therefore, it is only necessary to use multiple robotic arm fine-tuning tasks to quickly migrate and fine-tune the initial model to complete the migration of the end-side model to the new environment and new robotic arm, which greatly reduces the complexity of model training.

[0101] In this implementation, the low-rank adaptive strategy (low-rank matrix decomposition technology) can reduce the computational complexity and memory requirements during model training while maintaining or even improving the model's ability to handle complex data. Through this strategy, we can effectively adjust and optimize the key parameters of the model, such as learning rate and number of training rounds, without sacrificing model performance. When the low-rank adaptive strategy is used to perform migration training on the initial model, it does not change the model training process, but only changes the related calculations of the model training process.

[0102] In this embodiment, the action prediction model preset in the terminal device can be obtained by migration training the initial model based on the sample data of the robot arm operation and the low-rank adaptive strategy. The end-to-end visual language action large model as the initial model does not rely on atomic tasks, can be quickly migrated through end-to-end fine-tuning, and does not rely on the reward function, so that the trained action prediction model has good generalization and migration capabilities. In addition, based on the sample data obtained from multiple robot arm fine-tuning tasks, the action prediction model can better adapt to specific scenarios and robot arm task requirements. The use of low-rank adaptive strategies can reduce the computational complexity and memory requirements during model training, while maintaining or even improving the model's ability to process complex data. Without sacrificing performance, it effectively adjusts and optimizes key training parameters, thereby improving the performance of the action prediction model.

[0103] In a possible implementation, obtaining sample data of the robot arm operation in the above step S11 may include:

[0104] S21: Obtain fine-tuning task instructions and robot arm motion trajectories corresponding to each robot arm fine-tuning task. The robot arm motion trajectory includes a motion trajectory obtained by performing robot arm demonstration in a virtual environment and a motion trajectory obtained by performing robot arm demonstration in a real environment.

[0105] S22: semantically amplify the fine-tuning task instruction to obtain an amplified task instruction.

[0106] S23: Acquire a scene observation image of the end effector of the robot arm according to different camera postures, objects to be operated, and scene environments.

[0107] S24: Generate sample data of the robot arm operation according to the robot arm motion trajectory, the augmented task instructions and the scene observation image.

[0108] In this embodiment, the technician can set multiple robot arm fine-tuning tasks according to the task requirements and difficulty of the robot arm, including fine-tuning task instructions and robot arm motion trajectories. The fine-tuning task instructions can be instructions for the robot arm to perform different operations on different objects, and the robot arm motion trajectories can be robot arm motion trajectories corresponding to each robot arm fine-tuning task. The fine-tuning task instructions and the robot arm motion trajectories can be a many-to-many relationship, that is, one fine-tuning task instruction can correspond to one or more robot arm motion trajectories, and one robot arm motion trajectory can also correspond to one or more fine-tuning task instructions.

[0109] In this embodiment, after the technician sets up the robot arm fine-tuning task, the robot arm motion trajectory can be collected in the virtual environment and on the real robot arm at the same time. The virtual environment can rely on the Mujoco simulation environment for data collection; the real scene can be completed by teaching on the robot arm.

[0110] In this embodiment, after the technician sets the robot arm fine-tuning task, the technician can perform semantic expansion on the fine-tuning task instructions to obtain multiple fine-tuning task instructions with different expressions but the same essence, thereby improving the comprehensiveness of the fine-tuning task instructions.

[0111] In this embodiment, technicians can also perform data augmentation on the robot arm operation scene, such as adjusting the posture (position and angle) of the camera in the scene, adjusting the lighting angle in the scene, changing the objects to be operated in the scene, etc., so as to improve the comprehensiveness of the scene observation image.

[0112] In this embodiment, technicians can set up multiple robotic arm fine-tuning tasks according to specific scene requirements and task requirements, and perform data amplification on information such as fine-tuning task instructions, camera posture, objects to be operated, and scene environment, so that the sample data can comprehensively and accurately characterize the relevant information of the specific scene and robotic arm task, so that the motion prediction model trained according to the sample data can better adapt to the specific scene and robotic arm task requirements, thereby improving the accuracy of motion prediction of the motion prediction model.

[0113] The following is an explanation of the multi-modal large model-based robotic arm control method of the present application with reference to a specific embodiment.

[0114] In a specific embodiment, a factory introduces a new robotic arm, which is used to operate parts in the factory. The factory is equipped with a first camera device that can capture full-view observation images of the scene, and a second camera device that can capture images of the robotic arm. The operator uses the end-cloud collaborative control system to control the robotic arm. The end-cloud collaborative control system includes a computer that controls the operation of the robotic arm and a cloud server. The specific robotic arm control process is as follows:

[0115] In the first step, the operator sets fine-tuning task instructions for multiple robot arm fine-tuning tasks according to the task requirements and difficulty of the robot arm, and demonstrates the corresponding robot arm motion trajectories on the virtual environment and the real robot arm. The fine-tuning task instructions and robot arm motion trajectories corresponding to each robot arm fine-tuning task are input into the computer.

[0116] In the second step, the computer semantically augments the fine-tuning task instructions to obtain the augmented task instructions. According to different camera postures, parts to be operated, and scene environments, the computer obtains the scene observation image of the robot arm end effector, and generates sample data of the robot arm operation based on the robot arm motion trajectory, the augmented task instructions, and the scene observation image.

[0117] In the third step, the computer performs migration training on the end-to-end OpenVLA model based on the sample data of the robotic arm operation and the low-rank adaptive strategy to obtain the action prediction model.

[0118] In the fourth step, the operator sends the target task operation instruction to the cloud server through the operation terminal. After receiving the instruction, the cloud server uses the first camera device to obtain a full-view observation image of the robot arm operation scene.

[0119] In the fifth step, the cloud server inputs the target task operation instructions and full-view observation images into the preset GPT-4o large model, obtains the robotic arm motion planning instructions, and sends the robotic arm motion planning instructions to the computer.

[0120] In the sixth step, after receiving the robot arm motion planning instruction sent by the cloud server, the computer obtains the image captured by the second camera device and compresses the image to obtain the current observation image of the robot arm.

[0121] In the seventh step, the computer inputs the robot arm motion planning instructions and the current observation image of the robot arm into a preset motion prediction model, obtains the corresponding motion sequence, and controls the robot arm to perform the target task according to the motion sequence.

[0122] In the eighth step, 15 seconds after the cloud server sends the robot arm motion planning instruction to the computer, the cloud server uses the first camera device to obtain a full-view feedback image of the robot arm operation scene.

[0123] In the ninth step, the cloud server inputs the target task operation instructions, full-view feedback images, and robotic arm motion planning instructions into the GPT-4o large model to obtain updated robotic arm motion planning instructions.

[0124] In the tenth step, the cloud server sends the updated robot arm motion planning instructions to the terminal device and repeats the above steps eight to nine until the robot arm completes the operation corresponding to the target task operation instruction.

[0125] Figure 3 This is a schematic diagram of the structure of a cloud server according to an embodiment of the present application. Figure 3 As shown, the cloud server includes: a first processor 301, and a first memory 302 communicatively connected to the first processor 301; the first memory 302 stores computer execution instructions; the first processor 301 executes the computer execution instructions stored in the first memory 302, so that the first processor executes the robot arm control method based on the multimodal large model as shown in the above steps S201-S203.

[0126] In the above-mentioned cloud server, the first memory 302 and the first processor 301 are electrically connected directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines, such as through a bus connection. The first memory 302 stores computer-executable instructions for implementing the data access control method, including at least one software function module that can be stored in the first memory 302 in the form of software or firmware. The first processor 301 executes various functional applications and data processing by running the software programs and modules stored in the first memory 302.

[0127] Figure 4 This is a schematic diagram of the structure of a terminal device according to an embodiment of the present application. Figure 4 As shown, the terminal device includes: a second processor 401, and a second memory 402 communicatively connected to the second processor 401; the second memory 402 stores computer execution instructions; the second processor 401 executes the computer execution instructions stored in the second memory 402, so that the second processor executes the robotic arm control method based on the multimodal large model as shown in the above steps S204-S206.

[0128] In the above terminal device, the second memory 402 and the second processor 401 are electrically connected directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines, such as through a bus connection. The second memory 402 stores computer-executable instructions for implementing the data access control method, including at least one software function module that can be stored in the second memory 402 in the form of software or firmware. The second processor 401 executes various functional applications and data processing by running the software programs and modules stored in the second memory 402.

[0129] The memory 302 / 402 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory 302 / 402 is used to store programs, and the processor 301 / 401 executes the program after receiving the execution instruction. Furthermore, the software programs and modules in the above-mentioned memory 302 / 402 may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.

[0130] The processor 301 / 401 may be an integrated circuit chip having the ability to process signals. The above-mentioned processor 301 / 401 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The disclosed methods, steps and logic block diagrams in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0131] An embodiment of the present application also provides a terminal-cloud collaborative control system, including: Figure 3 The cloud server shown in Figure 4 The terminal equipment shown.

[0132] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the steps of each method embodiment of the present application.

[0133] An embodiment of the present application further provides a computer program product, including a computer program, which implements the steps of each method embodiment of the present application when executed by a processor.

[0134] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0135] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0136] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0137] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0138] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the appended claims.

[0140] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A robotic arm control method based on a multi-modal large model, applied to a cloud server, characterized in that: include: In response to the input target task operation instruction, a full-view observation image of the robot arm operation scene is obtained; Inputting the target task operation instruction and the full-view observation image into a preset multimodal large model to obtain a robot arm motion planning instruction; The robot arm motion planning instruction is sent to the terminal device, and the robot arm motion planning instruction is used to instruct the terminal device to generate a corresponding action sequence according to the robot arm motion planning instruction and the current observation image of the robot arm, and control the robot arm to perform the target task based on the action sequence.

2. The method for controlling a robotic arm based on a multi-modal large model according to claim 1, characterized in that: Also includes: Step S1: after the robot arm motion planning instruction is sent to the terminal device for a preset time, a full-view feedback image of the robot arm operation scene is obtained; Step S2: inputting the target task operation instruction, the full-view feedback image and the robot arm motion planning instruction into the multimodal large model to obtain an updated robot arm motion planning instruction; Step S3: Send the updated robot arm motion planning instruction to the terminal device, and repeat the above steps S1-S2.

3. The method for controlling a robotic arm based on a multi-modal large model according to claim 2, characterized in that: The resolution of the full-view observation image / full-view feedback image is greater than the resolution of the current observation image of the robotic arm.

4. A robotic arm control method based on a multi-modal large model, applied to a terminal device, characterized in that: include: In response to the robot arm motion planning instruction sent by the cloud server, the current observation image of the robot arm is obtained, wherein the robot arm motion planning instruction is obtained by the cloud server responding to the target task operation instruction, obtaining the full-view observation image of the robot arm operation scene, and inputting the target task operation instruction and the full-view observation image into a preset multi-modal large model; Inputting the robot arm motion planning instruction and the current observation image of the robot arm into a preset motion prediction model to obtain a corresponding motion sequence; The robotic arm is controlled to perform a target task according to the action sequence.

5. The method for controlling a robotic arm based on a multi-modal large model according to claim 4, characterized in that: The action prediction model is trained in the following way: Acquire sample data of robot arm operation, wherein the sample data is obtained based on multiple robot arm fine-tuning tasks; The initial model is transferred and trained according to the sample data of the robot arm operation and the low-rank adaptive strategy to obtain an action prediction model. The initial model is an end-to-end visual language action large model.

6. The method for controlling a robotic arm based on a multi-modal large model according to claim 5, characterized in that: The obtaining of sample data of the robot arm operation includes: Obtaining fine-tuning task instructions and a motion trajectory of the robotic arm corresponding to each fine-tuning task of the robotic arm, wherein the motion trajectory of the robotic arm includes a motion trajectory obtained by performing a robotic arm demonstration in a virtual environment and a motion trajectory obtained by performing a robotic arm demonstration in a real environment; Performing semantic expansion on the fine-tuning task instruction to obtain an expanded task instruction; Acquire a scene observation image of the end effector of the robot arm according to different camera postures, objects to be operated, and scene environments; Sample data of the robot arm operation is generated according to the robot arm motion trajectory, the augmented task instructions and the scene observation image.

7. A cloud server, characterized in that: include: A first processor, and a first memory communicatively connected to the first processor; The first memory is used to store computer-executable instructions; The first processor is used to execute the computer-executable instructions stored in the first memory, so that the first processor executes the robotic arm control method based on a multi-modal large model as described in any one of claims 1-3.

8. A terminal device, characterized in that: include: a second processor, and a second memory communicatively connected to the second processor; The second memory is used to store computer-executable instructions; The second processor is used to execute the computer-executable instructions stored in the second memory, so that the second processor executes the robotic arm control method based on a multi-modal large model as described in any one of claims 4-6.

9. A terminal-cloud collaborative control system, characterized in that: include: The cloud server as claimed in claim 7, and the terminal device as claimed in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the robotic arm control method based on a multi-modal large model described in any one of claims 1-3 or 4-6.

Images