Body-fitting intelligent execution and training method and device based on edge-end collaborative large model

Abstract

The invention discloses a body intelligent execution and training method and device based on an edge-end cooperation large model, and the method comprises the steps: transmitting a voice instruction to a robot body, converting the voice instruction into a text instruction through the robot body, and transmitting the text instruction, machine vision feedback and a visual image to an edge side unit; a multi-modal large model is arranged in the edge side unit; reasoning the text instruction, the machine vision feedback and the vision image by the multi-modal large model, obtaining the intention of the text instruction after reasoning, generating a task decomposition instruction according to the intention, and sending the task decomposition instruction to the robot body; the robot body calls a chassis moving API and an atomic skill library API according to the task decomposition instruction to complete a target task; and after the target task is completed, the robot body feeds back process algorithm data and an execution effect to the edge side unit, and the edge side unit carries out post-training upgrading according to the feedback. According to the method, the large model reasoning time delay and cost can be effectively reduced, the load of the cloud server is reduced, and the data security and privacy are improved.

Description

Technical Field

[0001] The present invention relates to the fields of embodied intelligence technology and artificial intelligence technology, and specifically to an embodied intelligence execution and training method and device based on an edge-end collaborative big model. Background Art

[0002] In today's society, intelligent robots have been involved in all aspects of human life and are playing an increasingly important role in high-end manufacturing and fine processing. How to improve the response speed of robots and achieve real-time response has become an important direction for the development of robots.

[0003] At present, in the existing technical solutions, the large language models or multimodal large models called by the robot are deployed in the cloud. During the actual execution of the robot, depending on the network conditions, there are communication delays in data uploading and inference result transmission, which affects the real-time execution response. On the other hand, the user's original data needs to be uploaded to the cloud, and data security and privacy are not well guaranteed. At the same time, traditional robot control technology relies on deterministic fixed code operation processes and can only handle simple tasks. The existing cloud-based solution for deploying large models to control robots will inevitably cause computing power burden and call congestion when controlling a large number of robots, and it is not convenient for the robot terminal to provide computing power support at the hardware level. At the same time, the robot terminal does not have the self-learning ability based on artificial intelligence large model training.

[0004] Therefore, how to invent a robot execution and training method for a large embodied intelligent model has become an urgent problem to be solved. Summary of the invention

[0005] To this end, the present invention provides an embodied intelligent execution and training method based on edge-end collaborative large models, which can output global task planning through the local edge-side multimodal large model, perform local planning when the robot body executes the task, and collaborate with the edge to improve the efficiency of the entire system. At the same time, it effectively reduces the delay and cost of large model reasoning, reduces the load on the cloud server, and at the same time, the user's original data does not leave the local area, improving data security and privacy.

[0006] In order to achieve the above-mentioned purpose, the present invention provides the following technical solution: an embodied intelligent execution and training method based on an edge-end collaborative large model, comprising:

[0007] Sending a voice command to the robot body, the robot body receiving the voice command and converting the voice command into a text command;

[0008] The robot body obtains machine vision feedback and visual images through the perception system; the robot body sends the text instructions, the machine vision feedback and the visual images to the edge side unit; the edge side unit is provided with a trained multimodal large model;

[0009] The multimodal large model infers the text instruction, the machine vision feedback and the visual image, and obtains the intention of the text instruction after inference; the multimodal large model generates a task decomposition instruction according to the intention of the text instruction, and sends the task decomposition instruction to the robot body;

[0010] The robot body calls the chassis movement API and the atomic skill library API according to the task decomposition instructions to complete the target task;

[0011] When the target task is completed, the robot body feeds back the process algorithm data and execution results to the edge side unit; the edge side unit performs post-training and upgrading according to the process algorithm data and execution results.

[0012] As a preferred solution for the embodied intelligent execution and training method based on the edge collaborative big model, the multimodal big model is a vertical big model for specific scenarios, including: a general base model, a home scene big model, a warehousing scene big model, and an industrial scene big model.

[0013] As a preferred solution for the embodied intelligent execution and training method based on the edge-to-edge collaborative big model, the APIs in the chassis movement API and the atomic skill library API are Python function code blocks presented in string format, which are executed by the Python interpreter to drive the robot's walking and movements.

[0014] As a preferred scheme for the embodied intelligent execution and training method based on the edge collaborative big model, during the process of post-training upgrade of the edge side unit according to the process algorithm data and execution results, if there is training improvement content in the task completion process, the edge side unit will allocate hardware computing power, and perform post-training upgrade with the feedback of the process algorithm data and the data feedback from the robot terminal sensor module as the corpus.

[0015] As a preferred solution for the embodied intelligent execution and training method based on the edge-side collaborative large model, the post-training of the edge-side unit includes: alignment training and fine-tuning training.

[0016] The present invention also provides an embodied intelligent execution and training device based on the edge-end collaborative big model, based on the above embodied intelligent execution and training method based on the edge-end collaborative big model, including:

[0017] A voice command receiving and converting module is used to send a voice command to the robot body, and the robot body receives the voice command and converts the voice command into a text command;

[0018] A text instruction uploading module is used for the robot body to obtain machine vision feedback and visual images through the perception system; the robot body sends the text instruction, the machine vision feedback and the visual image to the edge side unit; the edge side unit is provided with a trained multimodal large model;

[0019] A multimodal large model processing module, used for the multimodal large model to infer the text instruction, the machine vision feedback and the visual image, and obtain the intention of the text instruction after inference; the multimodal large model generates a task decomposition instruction according to the intention of the text instruction, and sends the task decomposition instruction to the robot body;

[0020] The target task execution module is used for the robot body to call the chassis movement API and the atomic skill library API according to the task decomposition instructions to complete the target task;

[0021] The multimodal large model post-training upgrade module is used for the robot body to feedback process algorithm data and execution results to the edge side unit after the target task is completed; the edge side unit performs post-training upgrades based on the process algorithm data and execution results.

[0022] As a preferred solution for the embodied intelligent execution and training device based on the edge collaborative big model, in the text instruction uploading module, the multimodal big model is a vertical big model for specific scenarios, including: a general base model, a home scene big model, a warehousing scene big model, and an industrial scene big model.

[0023] As a preferred solution for the embodied intelligent execution and training device based on the edge-to-edge collaborative large model, in the target task execution module, the APIs in the chassis movement API and the atomic skill library API are Python function code blocks presented in string format, which are executed by the Python interpreter to drive the robot's walking and movements.

[0024] As a preferred solution for the embodied intelligent execution and training device based on the edge collaborative big model, in the post-training upgrade module of the multimodal big model, when the edge side unit performs post-training upgrade according to the process algorithm data and execution results, if there is training improvement content in the task completion process, the edge side unit will allocate hardware computing power and perform post-training upgrade with the feedback of the process algorithm data and the data feedback from the robot terminal sensor module as the corpus.

[0025] As a preferred solution for the embodied intelligent execution and training device based on the edge collaborative big model, in the multimodal big model post-training upgrade module, the post-training of the edge side unit includes: alignment training and fine-tuning training.

[0026] The present invention has the following advantages: a voice command is sent to the robot body, the robot body receives the voice command and converts the voice command into a text command; the robot body obtains machine vision feedback and visual images through a perception system; the robot body sends the text command, the machine vision feedback and the visual image to an edge side unit; a trained multimodal large model is provided in the edge side unit; the multimodal large model infers the text command, the machine vision feedback and the visual image, and obtains the intention of the text command after inference; the multimodal large model generates a task decomposition instruction according to the intention of the text instruction, and sends the task decomposition instruction to the robot body; the robot body calls the chassis movement API and the atomic skill library API according to the task decomposition instruction to complete the target task; when the target task is completed, the robot body feeds back the process algorithm data and execution results to the edge side unit; the edge side unit performs post-training upgrade according to the process algorithm data and execution results. The present invention outputs global task planning through a local edge side multimodal large model, and the robot body performs local planning when performing tasks, and the edge and end cooperate to improve the efficiency of the entire system. At the same time, it provides the robot with perception and decision-making capabilities, so that the robot can think about solutions like humans when facing complex tasks. The edge-end combined embodied intelligent large model layout proposed in the present invention can not only guarantee the reasoning computing power requirements at the hardware level, but also provide training data updates for the robot's front-line work scenarios, that is, the robot's self-learning ability. Lightweight and miniaturized edge-side large model devices can be applied to more scenarios. At the same time, the present invention can effectively reduce the delay and cost of large model reasoning, reduce the load on cloud servers, and at the same time, the user's original data does not leave the local area, thereby improving data security and privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0028] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0029] Figure 1This is a flow chart of the embodied intelligent execution and training method based on the edge-end collaborative big model provided in Example 1 of the present invention;

[0030] Figure 2 This is a schematic diagram of the module architecture of the embodied intelligent device in the embodied intelligent execution and training method based on the edge-end collaborative big model provided in Example 1 of the present invention;

[0031] Figure 3 A schematic diagram of a specific execution flow of the robot body in the embodied intelligence execution and training method based on the edge-end collaborative large model provided in Example 1 of the present invention;

[0032] Figure 4 This is a schematic diagram of the architecture of the embodied intelligent execution and training device based on the edge-to-edge collaborative big model provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0033] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Example 1

[0035] See also Figure 1 , Figure 2 and Figure 3 Embodiment 1 of the present invention provides an embodied intelligent execution and training method based on an edge-end collaborative large model, comprising the following steps:

[0036] S1. Sending a voice command to the robot body, which receives the voice command and converts the voice command into a text command;

[0037] S2. The robot body obtains machine vision feedback and visual images through the perception system; the robot body sends the text instruction, the machine vision feedback and the visual image to the edge side unit; the edge side unit is provided with a trained multimodal large model;

[0038] S3, the multimodal large model infers the text instruction, the machine vision feedback and the visual image, and obtains the intention of the text instruction after inference; the multimodal large model generates a task decomposition instruction according to the intention of the text instruction, and sends the task decomposition instruction to the robot body;

[0039] S4, the robot body calls the chassis movement API and the atomic skill library API according to the task decomposition instruction to complete the target task;

[0040] S5. When the target task is completed, the robot body feeds back the process algorithm data and execution results to the edge side unit; the edge side unit performs post-training and upgrading according to the process algorithm data and execution results.

[0041] In this embodiment, in step S1, a voice command is sent to the robot body, and the robot body receives the voice command and converts the voice command into a text command;

[0042] Among them, Figure 2 As shown, the robot body includes a chassis, a robotic arm and a gripper, and a controller. The chassis includes a perception module, an execution module, and a control module. The robotic arm and the gripper include a perception module, an execution module, and a control module. The controller includes a robot model and a communication module. The robot model includes a chassis movement API and an atomic skill library API.

[0043] In this embodiment, taking the voice command "I am thirsty" as an example, after the robot body receives the voice command, it converts it into a text command "I am thirsty".

[0044] In this embodiment, in step S2, the robot body obtains machine vision feedback and visual images through the perception system; the robot body sends the text instruction, the machine vision feedback and the visual image to the edge side unit; the edge side unit is provided with a trained multimodal large model;

[0045] Specifically, the robot body obtains the current machine vision feedback and visual images including the environmental state through the perception system, and transmits text instructions, machine vision feedback and visual images to the edge side unit where the multimodal large model is deployed.

[0046] Among them, the multi-model big model built into the edge side unit is a pre-trained big model, which is different from the cloud big model. It is a vertical big model for specific scenarios, such as a general base model, a home scene big model, a warehousing scene big model, an industrial scene big model, etc. The vertical big model is generally compressed, and the preset model compression method includes at least one of pruning, knowledge distillation, model distillation, and quantization. The edge side robot big model performs reasoning and can perform post-training according to the actual scenario, including alignment and fine-tuning.

[0047] In this embodiment, in step S3, the multimodal large model infers the text instruction, the machine vision feedback and the visual image, and obtains the intention of the text instruction after inference; the multimodal large model generates a task decomposition instruction according to the intention of the text instruction, and sends the task decomposition instruction to the robot body;

[0048] Specifically, the edge-side large model infers and outputs decomposed task instructions based on text instructions, machine vision feedback and visual images, and transmits them to the robot body. For example, the large model infers the target task "take Coca-Cola" from the instruction "I am thirsty" based on the preferences of the person issuing the instruction and the beverage situation in the environment, and then outputs the decomposed task instructions as {go to the refrigerator point - open the refrigerator door - identify Coca-Cola - take Coca-Cola - close the refrigerator door - go to the target point - put down Coca-Cola}.

[0049] In this embodiment, in step S4, the robot body calls the chassis movement API and the atomic skill library API according to the task decomposition instruction to complete the target task;

[0050] Specifically, the robot body needs to complete subtasks such as "go to the refrigerator point - open the refrigerator door - take Coca-Cola - close the refrigerator door - go to the target point - put down Coca-Cola" in sequence according to the task decomposition instructions. This complex operation task including multiple steps has modular characteristics on the one hand, and each step is clearly defined. On the other hand, some steps are universal and can be learned separately as a sub-skill. Then, more complex operations can be achieved through combination, thereby updating the skill library API.

[0051] The robot body calls the API library in sequence according to the decomposed task instructions {go to the refrigerator point - open the refrigerator door - identify Coca-Cola - take Coca-Cola - close the refrigerator door - go to the target point - put down Coca-Cola} until the task is completed.

[0052] The APIs called include chassis movement API and atomic skill library API. The action reserve of the API library is sufficient to enrich the execution capability of the embodied intelligent robot to complex tasks. The API is a Python function code block presented in string format, which can be executed by the Python interpreter to drive the robot's walking and movement.

[0053] Specifically, the robot chassis is a small wheeled robot chassis, and the chassis perception module includes a variety of sensors such as laser radar, camera, and ultrasonic wave. The wheeled chassis has basic movement strategies, such as m0: move to a calibrated point, m4: move to a specific coordinate point, m6: go straight for a given distance, etc. The movement library combines various movement strategies to form a movement library Move = {m0, m1, ..., mn}. Where n is the number of movement strategies. For specific scenarios, the robot controller has a preset global map stored in it, and specific points are calibrated. For example, if the refrigerator point has been calibrated as marker_fri, when executing the decomposition task {to the refrigerator point}, the movement API: m0 (marker_fri) will be directly called. Move according to the plan. During the movement process, if an obstacle outside the global plan is encountered, the sensor module will identify it in time, avoid the obstacle, and continue to plan the route until it reaches the target position.

[0054] The robot arm is a small six-axis collaborative robot arm. The robot has basic action strategies, such as s0: grabbing, s4: plugging and unplugging, s6: pressing, etc. The skill library is a combination of various skill strategies to form a skill library Skill = {s0, s1, ..., sn}. Where n is the number of skill strategies. Of course, the skill library can include more skills, which can be determined by the user as needed to make it universal.

[0055] Among them, the robot arm and gripper perception module includes a depth camera and a force sensor, etc., which are used to collect the interaction data between the robot arm and the gripper and the environment, and use a deep neural network to detect, identify and estimate the unstructured environment. For example, the perception module uses the image captured by the depth camera at the end of the robot arm as input, outputs the target category and the position of the target in the image plane through a preset target detection and recognition algorithm; and outputs the six-degree-of-freedom pose of the target through the PoseCNN pose estimation algorithm. Among them, the preset target detection and recognition algorithm includes at least one of the YOLO method, the FasterRCNN method, and the Mask RCNN method.

[0056] In this embodiment, in step S5, when the target task is completed, the robot body feeds back the process algorithm data and execution results to the edge side unit; the edge side unit performs post-training and upgrading according to the process algorithm data and execution results.

[0057] Specifically, if there is training and improvement content during the task completion process, the edge side unit will allocate hardware computing power and use the feedback of the process algorithm data and the data feedback from the robot terminal sensor module as the corpus for post-training upgrades.

[0058] The robot calls the API library in sequence until the task is completed. After completing the task, the robot side will feedback the algorithm and execution results of the entire process to the edge-side multimodal large model. If there is training and improvement content during the task completion process, the edge-side unit will allocate hardware computing power, and use the feedback of the process algorithm data and the data feedback from the robot terminal sensor module as the corpus for post-training upgrades, including alignment training and fine-tuning training. Post-training can enable robots working in different scenarios to have the ability to solve problems according to local conditions, such as planning different furniture models to result in different opening strategies and opening methods for handle designs, and opening strategies and opening methods for the same furniture under different placements.

[0059] In summary, the present invention sends voice commands to the robot body, and the robot body receives the voice commands and converts the voice commands into text commands; the robot body obtains machine vision feedback and visual images through the perception system; the robot body sends the text commands, the machine vision feedback and the visual images to the edge side unit; the edge side unit is provided with a trained multimodal large model; the multimodal large model infers the text commands, the machine vision feedback and the visual images, and obtains the intention of the text commands after inference; the multimodal large model generates task decomposition instructions according to the intention of the text commands, and sends the task decomposition instructions to the robot body; the robot body calls the chassis movement API and the atomic skill library API according to the task decomposition instructions to complete the target task; when the target task is completed, the robot body feeds back the process algorithm data and execution results to the edge side unit; the edge side unit performs post-training upgrades according to the process algorithm data and execution results. The present invention outputs global task planning through the local edge side multimodal large model, and the robot body performs local planning when performing tasks, and the edge and end collaborate to improve the efficiency of the entire system. At the same time, it provides the robot with perception and decision-making capabilities, so that the robot can think about solutions like humans when facing complex tasks. The edge-end combined embodied intelligent large model layout proposed in the present invention can not only guarantee the reasoning computing power requirements at the hardware level, but also provide training data updates for the robot's front-line work scenarios, that is, the robot's self-learning ability. Lightweight and miniaturized edge-side large model devices can be applied to more scenarios. At the same time, the present invention can effectively reduce the delay and cost of large model reasoning, reduce the load on cloud servers, and at the same time, the user's original data does not leave the local area, thereby improving data security and privacy.

[0060] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0061] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0062] Example 2

[0063] See also Figure 4 Embodiment 2 of the present invention also provides an embodied intelligent execution and training device based on an edge-end collaborative large model, including:

[0064] The voice command receiving and converting module 001 is used to send a voice command to the robot body, and the robot body receives the voice command and converts the voice command into a text command;

[0065] The text instruction uploading module 002 is used for the robot body to obtain machine vision feedback and visual images through the perception system; the robot body sends the text instruction, the machine vision feedback and the visual image to the edge side unit; the edge side unit is provided with a trained multimodal large model;

[0066] The multimodal large model processing module 003 is used for the multimodal large model to infer the text instruction, the machine vision feedback and the visual image, and obtain the intention of the text instruction after inference; the multimodal large model generates a task decomposition instruction according to the intention of the text instruction, and sends the task decomposition instruction to the robot body;

[0067] The target task execution module 004 is used for the robot body to call the chassis movement API and the atomic skill library API according to the task decomposition instruction to complete the target task;

[0068] The multimodal large model post-training upgrade module 005 is used for the robot body to feedback process algorithm data and execution results to the edge side unit after the target task is completed; the edge side unit performs post-training upgrades based on the process algorithm data and execution results.

[0069] In this embodiment, in the text instruction uploading module 002, the multimodal big model is a vertical big model for a specific scene, including: a general base model, a home scene big model, a warehousing scene big model, and an industrial scene big model.

[0070] In this embodiment, in the target task execution module 004, the chassis movement API and the API in the atomic skill library API are Python function code blocks presented in string format, which can be executed by the Python interpreter to drive the robot's walking and movements.

[0071] In this embodiment, in the multimodal large model post-training upgrade module 005, when the edge side unit performs post-training upgrades according to the process algorithm data and execution results, if there is training improvement content during the task completion process, the edge side unit will allocate hardware computing power and perform post-training upgrades using the feedback of the process algorithm data and the data feedback from the robot terminal sensor module as a corpus.

[0072] In this embodiment, in the multimodal large model post-training upgrade module 005, the post-training of the edge side unit includes: alignment training and fine-tuning training.

[0073] It should be noted that the information interaction, execution process and other contents between the modules of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and will not be repeated here.

[0074] Example 3

[0075] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which program code for an embodied intelligent execution and training method based on an edge-to-edge collaborative big model is stored, and the program code includes instructions for executing embodiment 1 or any possible implementation method of the embodied intelligent execution and training method based on an edge-to-edge collaborative big model.

[0076] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0077] Example 4

[0078] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0079] The processor and the memory communicate with each other through a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the embodied intelligent execution and training method based on the edge-to-edge collaborative large model of Example 1 or any possible implementation thereof.

[0080] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor implemented by reading software codes stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0081] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0082] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing system, they can be concentrated on a single computing system, or distributed on a network composed of multiple computing systems, and optionally, they can be implemented by a program code executable by a computing system, so that they can be stored in a storage system and executed by the computing system, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0083] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. The embodied intelligent execution and training method based on the edge-end collaborative big model is characterized by: include: Sending a voice command to the robot body, the robot body receiving the voice command and converting the voice command into a text command; The robot body obtains machine vision feedback and visual images through the perception system; the robot body sends the text instructions, the machine vision feedback and the visual images to the edge side unit; the edge side unit is provided with a trained multimodal large model; The multimodal large model infers the text instruction, the machine vision feedback and the visual image, and obtains the intention of the text instruction after inference; The multimodal large model generates a task decomposition instruction according to the intention of the text instruction, and sends the task decomposition instruction to the robot body; The robot body calls the chassis movement API and the atomic skill library API according to the task decomposition instructions to complete the target task; When the target task is completed, the robot body feeds back the process algorithm data and execution results to the edge side unit; the edge side unit performs post-training and upgrading according to the process algorithm data and execution results.

2. The embodied intelligent execution and training method based on the edge-end collaborative big model according to claim 1 is characterized in that: The multimodal big model is a vertical big model for specific scenarios, including: a general base model, a home scene big model, a warehousing scene big model, and an industrial scene big model.

3. The embodied intelligent execution and training method based on the edge-end collaborative big model according to claim 2 is characterized in that: The APIs in the chassis movement API and the atomic skill library API are Python function code blocks presented in string format and are executed by the Python interpreter to drive the robot's walking and movements.

4. The embodied intelligent execution and training method based on the edge-end collaborative big model according to claim 3 is characterized in that: During the process of post-training and upgrading of the edge side unit based on the process algorithm data and execution results, if there is training improvement content during the task completion process, the edge side unit will allocate hardware computing power and perform post-training and upgrading with the feedback of the process algorithm data and the data feedback from the robot terminal sensor module as the corpus.

5. The embodied intelligent execution and training method based on the edge-end collaborative big model according to claim 4 is characterized in that: The post-training of the edge side unit includes: alignment training and fine-tuning training.

6. An embodied intelligent execution and training device based on an edge-end collaborative big model, adopting the embodied intelligent execution and training method based on an edge-end collaborative big model as described in any one of claims 1 to 5, characterized in that: include: A voice command receiving and converting module is used to send a voice command to the robot body, and the robot body receives the voice command and converts the voice command into a text command; A text instruction uploading module is used for the robot body to obtain machine vision feedback and visual images through the perception system; the robot body sends the text instruction, the machine vision feedback and the visual image to the edge side unit; the edge side unit is provided with a trained multimodal large model; A multimodal large model processing module, used for the multimodal large model to infer the text instruction, the machine vision feedback and the visual image, and obtain the intention of the text instruction after inference; The multimodal large model generates a task decomposition instruction according to the intention of the text instruction, and sends the task decomposition instruction to the robot body; The target task execution module is used for the robot body to call the chassis movement API and the atomic skill library API according to the task decomposition instructions to complete the target task; The multimodal large model post-training upgrade module is used for the robot body to feedback process algorithm data and execution results to the edge side unit after the target task is completed; the edge side unit performs post-training upgrades based on the process algorithm data and execution results.

7. The embodied intelligent execution and training device based on the edge-end collaborative big model according to claim 6 is characterized in that: In the text instruction uploading module, the multimodal big model is a vertical big model for specific scenes, including: a general base model, a home scene big model, a warehousing scene big model, and an industrial scene big model.

8. The embodied intelligent execution and training device based on the edge-end collaborative big model according to claim 7 is characterized in that: In the target task execution module, the chassis movement API and the API in the atomic skill library API are Python function code blocks presented in string format, which are executed by the Python interpreter to drive the robot's walking and movements.

9. The embodied intelligent execution and training device based on the edge-end collaborative big model according to claim 8 is characterized in that: In the multimodal large model post-training upgrade module, when the edge side unit performs post-training upgrades based on the process algorithm data and execution results, if there is training improvement content during the task completion process, the edge side unit will allocate hardware computing power and perform post-training upgrades using the feedback of the process algorithm data and the data feedback from the robot terminal sensor module as the corpus.

10. The embodied intelligent execution and training device based on the edge-end collaborative big model according to claim 9 is characterized in that: In the multimodal large model post-training upgrade module, the post-training of the edge side unit includes: alignment training and fine-tuning training.

Images