An artificial intelligence-based robot control method, device, equipment and medium

Abstract

The application relates to an artificial intelligence technology, which can be applied to a medical health, financial technology and other business system platform, and discloses a robot control method, device, equipment and medium based on artificial intelligence, which comprises the following steps: decomposing a natural language task instruction; generating a planning path instruction of a robot according to the decomposed task; executing the planning path instruction in a simulation environment, and outputting operation track information of the robot and environment state change data; verifying the simulation execution result, and if the verification is successful, storing the natural language task instruction, the decomposed task, the operation track information and a success label as multi-modal information; training an initial diffusion strategy model based on the multi-modal information, generating a robot decision model; and controlling the robot based on the robot decision model. The application significantly improves the learning efficiency and robustness of the robot through an automatic and large-scale data generation process, and can respond to natural language instructions to execute multi-task.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a robot control method, apparatus, device, and medium based on artificial intelligence. Background Technology

[0002] With the development of technology, the application of robots is becoming increasingly widespread, covering fields such as medicine and finance, bringing revolutionary changes to the industry.

[0003] In the medical field, surgical robots, rehabilitation robots, nursing robots, and medical service robots have emerged. Surgical robots combine AI, robotics, and medical technologies to assist or completely replace doctors in surgical procedures. Through high-precision robotic arms and advanced vision systems, surgical robots can accurately locate and remove lesions, reducing surgical risks and recovery time, and improving surgical success rates and patient safety. Rehabilitation robots combine AI, robotics, and rehabilitation medicine theories to provide patients with personalized rehabilitation training programs. By simulating human movement patterns and biomechanical characteristics, rehabilitation robots can precisely train and stimulate joints and muscles, promoting the recovery of motor function. Nursing robots combine AI, robotics, and nursing medicine theories to provide patients with personalized daily care services, such as posture adjustment, medication management, and health monitoring. Medical service robots are mainly used in hospitals for auxiliary tasks such as patient guidance, medication delivery, and disinfection, improving hospital operational efficiency and service quality.

[0004] In the financial sector, chatbots, leveraging natural language processing technology, can provide 24 / 7 customer service, answering inquiries and handling diverse needs such as account checks, fund transfers, and investment product recommendations. This intelligent customer service approach not only improves customer satisfaction but also significantly reduces the labor costs for financial institutions. Based on customer history and preferences, chatbots can offer personalized service suggestions, such as customized investment product recommendations and tailored investment advice, thereby enhancing customer loyalty. Fraud detection and risk management chatbots can also utilize AI to analyze trillions of data points, assessing transaction risks within seconds, improving fraud detection rates, and reducing false alarm rates.

[0005] Existing methods for acquiring robot skills primarily rely on large-scale data collection (such as human instruction and trial-and-error exploration) and effective learning of strategies from massive datasets. However, these methods have significant limitations: human instruction and annotation depend heavily on manpower and are difficult to scale; autonomous exploration or the use of non-robot data often lacks necessary information, has low data quality, or requires complex cross-modal transfer; directly using large language models (LLMs) as the underlying control strategy typically lacks accuracy and robustness in relation to the physical world; and traditional behavior cloning methods heavily depend on difficult-to-obtain high-quality expert data. Therefore, existing robot dataset collection lacks necessary information, dataset quality is poor, and underlying control relies solely on large speech models, resulting in poor accuracy and robustness. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the present invention provides a robot control method, device, equipment and medium based on artificial intelligence, which aims to solve the problems in the prior art where the collection of robot dataset samples lacks necessary information, the dataset quality is poor, and the control of the underlying layer relies only on large voice models, resulting in poor accuracy and robustness.

[0007] The technical solution of the present invention is as follows:

[0008] The first embodiment of the present invention provides a robot control method based on artificial intelligence, the method comprising:

[0009] Obtain natural language task instructions, decompose the natural language task instructions based on a large language model to obtain decomposed tasks, and generate successful detection conditions for the decomposed tasks.

[0010] The decomposed task is converted into robot operation primitives, and the robot's planned path instructions are generated based on the robot operation primitives.

[0011] The planned path instructions are executed in a simulation environment to generate simulation execution results and output the robot's operation trajectory information and environmental state change data.

[0012] The simulation execution results are verified according to the success detection conditions of the decomposed task. If the verification is successful, the natural language task instructions, decomposed tasks, operation trajectory information and success labels are stored in the database as multimodal information.

[0013] Multimodal information is extracted from the database, and the initial diffusion strategy model is trained based on the multimodal information. After training, a robot decision model is generated.

[0014] The robot is controlled based on the robot decision-making model.

[0015] Another embodiment of the present invention provides an artificial intelligence-based robot control device, the device comprising:

[0016] The task decomposition module is used to obtain natural language task instructions, decompose the natural language task instructions based on a large language model to obtain decomposed tasks, and generate successful detection conditions for the decomposed tasks.

[0017] The task conversion module is used to convert the decomposed tasks into robot operation primitives, and generate robot planning path instructions based on the robot operation primitives.

[0018] The path planning simulation module is used to execute the planned path instructions in a simulation environment, generate simulation execution results, and output the robot's operation trajectory information and environmental state change data.

[0019] The verification and storage module is used to verify the simulation execution result according to the success detection conditions of the decomposed task. If the verification is successful, the natural language task instruction, decomposed task, operation trajectory information and success label are stored in the database as multimodal information.

[0020] The diffusion strategy training module is used to extract multimodal information based on the database, train the initial diffusion strategy model based on the multimodal information, and generate a robot decision model after training is completed.

[0021] A control module is used to control the robot based on the robot decision model.

[0022] Another embodiment of the present invention provides a computer device, the computer device including at least one processor; and,

[0023] A memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores instructions that can be executed by the at least one processor, which, when executed, enable the at least one processor to perform the steps of the artificial intelligence-based robot control method described above.

[0025] Another embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the artificial intelligence-based robot control method described above.

[0026] Beneficial Effects: The AI-based robot control method, apparatus, device, and medium of this invention can be used to: acquire natural language task instructions; decompose the natural language task instructions based on a large language model to obtain decomposed tasks and generate success detection conditions for the decomposed tasks; convert the decomposed tasks into robot operation primitive instructions; generate robot planning path instructions based on the robot operation primitive instructions; execute the planning path instructions in a simulation environment, generate simulation execution results, and output robot operation trajectory information and environmental state change data; verify the simulation execution results based on the success detection conditions of the decomposed tasks; if the verification is successful, store the natural language task instructions, decomposed tasks, operation trajectory information, and success labels as multimodal information in a database; extract multimodal information based on the database; train an initial diffusion strategy model based on the multimodal information; generate a robot decision model after training; and control the robot based on the robot decision model. This invention significantly improves the efficiency and robustness of robot learning through an automated and large-scale data generation process. It not only reduces the reliance on human instruction and reward design, but also collects high-quality data containing failure recovery experience by utilizing LLM guidance and a "verification and retry" mechanism. Finally, the robot "brain" refined through a diffusion strategy not only has excellent performance, surpassing the original data collection process in complex, long-term tasks that require common sense reasoning, but more importantly, it inherits strong robust recovery capabilities, can respond to natural language instructions to perform multiple tasks, and shows good potential for transfer from simulation to reality. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram illustrating the application environment of an embodiment of the robot control method based on artificial intelligence according to the present invention;

[0029] Figure 2 This is a flowchart of a preferred embodiment of an artificial intelligence-based robot control method according to the present invention;

[0030] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of an artificial intelligence-based robot control device of the present invention;

[0031] Figure 4This is a schematic diagram of a preferred embodiment of a computer device according to the present invention;

[0032] Figure 5 This is another structural schematic diagram of a preferred embodiment of a computer device according to the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0034] The embodiments of the present invention will be described below with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0035] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Here, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0036] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0037] The AI-based robot control method provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The client accesses the server's network or business platform, and the server can obtain natural language task instructions. Based on a large language model, the natural language task instructions are decomposed into decomposed tasks, and success detection conditions for the decomposed tasks are generated. The decomposed tasks are converted into robot operation primitive instructions, and robot planning path instructions are generated based on the robot operation primitive instructions. The planned path instructions are executed in a simulation environment, generating simulation execution results and outputting robot operation trajectory information and environmental state change data. The simulation execution results are verified according to the success detection conditions of the decomposed tasks. If the verification is successful, the natural language task instructions, decomposed tasks, operation trajectory information, and success labels are stored in the database as multimodal information. The multimodal information is extracted from the database, and the initial diffusion strategy model is trained based on the multimodal information. After training, a robot decision model is generated. The robot is controlled based on the robot decision model. In this invention, an automated and large-scale data generation process significantly improves the efficiency and robustness of robot learning. This not only reduces reliance on human instruction and reward design but also utilizes LLM guidance and a "verification and retry" mechanism to collect high-quality data containing failure recovery experience. The robot "brain," refined through a diffusion strategy, not only boasts superior performance but also surpasses the original data collection process in complex, long-term tasks requiring common-sense reasoning. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0038] To address the above problems, embodiments of the present invention provide an artificial intelligence-based robot control method. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart illustrating a preferred embodiment of an artificial intelligence-based robot control method according to the present invention. Figure 2 As shown, it includes:

[0039] Step S100: Obtain natural language task instructions, decompose the natural language task instructions based on a large language model to obtain decomposed tasks, and generate successful detection conditions for the decomposed tasks.

[0040] This invention primarily acquires user-issued natural language task instructions and uses these instructions as input to a large-scale language model. The natural language task instructions are in the form of text strings (e.g., "Please put the red blocks on the table into the blue box"). This instruction is fed into the LLM module, which performs task understanding and planning to obtain the decomposed task. An LLM is a large pre-trained neural network model, typically a deep learning model with billions or even more parameters (such as the evaluated GPT or LLaMA series models). It is invoked through a specific API interface or runs directly on a server.

[0041] In the medical field, LLMs can analyze textual information such as patient symptom descriptions, medical history, and examination results, and combine this with a medical knowledge base to provide doctors with possible diagnostic suggestions, assisting them in making more accurate and faster diagnoses. For example, GoogleMed-PaLM achieved an overall accuracy of 86.5% in the USMLE test, surpassing 90% of human test takers, and performed particularly well in clinical diagnosis (such as diabetes and cardiovascular disease) and drug interaction analysis. Based on the patient's specific situation and the latest medical research findings, LLMs can recommend personalized treatment plans for doctors, including drug selection, dosage adjustment, and treatment cycles. LLMs can interpret and analyze medical imaging reports, extracting key information to help doctors better understand imaging results and improve diagnostic efficiency. For example, the Dia-LLaMA framework combines the LLaMA2-7B model and pre-trained ViT3D to process high-dimensional CT data and generate reports, demonstrating excellent performance on the CTRG-Chest-548K dataset.

[0042] In the financial sector, LLM can be used to enhance financial customer service: for example, Bloomberg GPT specializes in handling financial conversations and understanding technical terminology; 24 / 7 multilingual services support seamless communication for global clients; and personalized financial advisory services provide customized investment advice based on clients' risk preferences. In financial information processing, it can also enable real-time market analysis, handling massive amounts of financial news, social media, and financial report data; intelligent report generation, automatically generating investment analysis reports and credit rating summaries; and regulatory document parsing, quickly extracting key information from SEC documents and prospectuses.

[0043] Step S100 involves obtaining natural language task instructions, decomposing the instructions based on a large language model to obtain decomposed tasks, and generating successful detection conditions for the decomposed tasks, including:

[0044] Step S101: Obtain natural language task instructions and input the natural language task instructions into the large language model;

[0045] Step S102: Obtain the decomposed subtask sequence or plan tree output by the large language model;

[0046] Step S103: Obtain the success detection conditions for each subtask in the subtask sequence generated by the large language model or for each subplan in the plan tree.

[0047] The system acquires user-issued natural language task instructions and uses these instructions as input to a large language model. These instructions are in the form of text strings (e.g., "Please put the red block on the table into the blue box"). This instruction is fed into the LLM module, which performs task understanding and planning to obtain the decomposed tasks. The system outputs a sequence of decomposed subtasks or a plan tree, in the form of structured text or an internal list / tree data structure (e.g., a Python list ['Approach the red block', 'Grab the red block', 'Move onto the blue box', 'Place the red block']), and success detection conditions inferred for each subtask or the overall task, in the form of text strings containing executable code (e.g., the Python code string "def check_success(sim_state):return is_object_inside(sim_state,'red_block','blue_box')").

[0048] Step S200: Convert the decomposed task into robot operation primitives, and generate robot planning path instructions based on the robot operation primitives.

[0049] Decomposing a task into multiple subtasks or a plan tree is insufficient for the robot to execute. Further conversion of the decomposed task into robot operation primitives is necessary. Then, based on these primitives, the robot's planned path instructions are generated.

[0050] Robot primitives are the lowest-level, indivisible functional units that constitute the basic capabilities and behaviors of a robot; they are the "atomic operations" of a robot system. These primitives form the basic modules of complex robot behaviors, and can be combined to achieve advanced functions.

[0051] Primitives can be categorized as follows: Perception Primitives, such as visual processing: pixel acquisition, edge detection, feature extraction; auditory processing: sound wave reception, speech feature extraction; tactile processing: pressure sensing, texture recognition; and environmental perception: distance measurement, object recognition, and SLAM (simultaneous localization and mapping).

[0052] The class of motion primitives includes: basic movement: linear movement, arc movement, and stopping; joint control: position control, speed control, and torque control; grasping actions: pre-grasp posture and grasping force control; and path planning: A* algorithm and RRT (Rapid Exploration Random Tree).

[0053] The Decision Primitives class includes: State Judgment: Condition Detection, Threshold Comparison; Behavior Selection: If-Then Rules, Utility Function Evaluation; Task Decomposition: Goal Breakdown, Subtask Generation.

[0054] Interaction primitives include: voice interaction (speech recognition, speech synthesis); physical interaction (force control, touch response); and social interaction (facial expression generation, body language).

[0055] Step S200 involves converting the decomposed task into robot operation primitives, and generating a planned path instruction for robot execution based on the robot operation primitives, including:

[0056] Step S201: Select the corresponding robot operation primitive according to the content of each sub-task or each sub-plan;

[0057] Step S202: Select the corresponding robot operation primitive instruction according to the robot operation primitive;

[0058] Step S203: Execute the robot operation primitive instruction, call the corresponding underlying module of the robot, and generate the robot's planned path instruction.

[0059] These subtasks (in the form of text strings, such as "grab the red block") are sequentially fed into a decision module. This module selects the appropriate underlying robot operation primitive based on the subtask content and outputs the selected primitive and its parameters. These primitives exist as function call instructions within the program or as data structures containing instructions and parameters (e.g., a dictionary {'primitive': ... ).

[0060] `'grasp_object','target':'red_block'}`, which triggers the actual function call. These primitive instructions are executed (their input is a function call and the corresponding numerical or text parameters, e.g., `robot.execute_primitive({'primitive':'grasp_object', ... )`.

[0061] When 'target':'red_block'})), the system calls the corresponding underlying modules in sequence: First, if it is a grasping primitive, the grasping sampler determines one or more suitable grasping postures based on the target object (its input parameter is the object name or ID in the form of a text string) and the environmental information (structured geometric data) obtained from the simulator. Its output is a list of numerical data containing 6D poses (position and orientation). Next, the motion planner takes the current robot state (the input is the joint angles or end poses of numerical vectors) as the starting point, takes the grasping posture obtained in the previous step or other target postures (the input is 6D pose numerical data) as the target, and combines the environmental model (the input is scene geometric data) to plan a collision-free motion path. Its output is a time series composed of joint angles or end poses.

[0062] The decision-making module acts as a "translator" and "scheduler" connecting high-level task planning with the underlying robot's specific actions. Its core function is to transform the subtask descriptions (such as "pick up the cup") output by the Large Language Model (LLM) into executable robot operations through logical judgment and information conversion. It implements a process through program code (usually a Python script), including conditional statements to handle different tasks and object types. It calls the simulator's API to obtain environmental state information (such as object position and attributes), and may even call the LLM's API again to obtain more specific execution details (such as explicitly specifying which object to grasp). Finally, it outputs function call instructions and corresponding parameters for other underlying modules such as the grasp sampler and motion planner.

[0063] A grasping sampler is a dedicated software module or algorithm whose core task is to help a robot find ways to stably grasp a target object using its hands (grippers). It is based on geometric analysis of the target object and the gripper's 3D model (e.g., finding contact points that can form force or shape closures) or random sampling strategies (e.g., generating a large number of candidate grasping postures on the object's surface or in the surrounding space, and then filtering them by criteria such as stability and reachability, such as uniform sampling in the object's point cloud). It is implemented through these algorithms and, based on the input object and gripper model data and the object's current pose, finally outputs one or more six-degree-of-freedom pose data describing feasible grasping methods.

[0064] A motion planner is a core software module or algorithm whose main function is to calculate a path for the robot to safely move from its current state to a target state without colliding with the environment or itself. It achieves this by searching in a high-dimensional "configuration space" formed by all possible combinations of robot joint angles, typically using sampling-based planning algorithms (such as RRT - Fast Exploration Random Tree). These algorithms can efficiently explore space and find feasible paths, while also combining the robot's kinematic model and a collision detection module (often using simulator functions to determine whether a collision has occurred). Based on the input robot and environment model, initial and target states, it ultimately outputs a motion trajectory consisting of a series of intermediate points (joint angles or end-effector poses) for the robot to execute.

[0065] In the financial sector, specific primitives for financial robots include: data acquisition primitives: market data subscription (subscribe_market_data); sentiment analysis (sentiment_analysis); trade execution primitives: order generation (generate_order); risk check (risk_check); trade execution (execute_trade); and compliance primitives including: AML check (aml_check) and regulatory report generation (generate_report).

[0066] Step S300: Execute the planned path instruction in the simulation environment, generate simulation execution results, and output the robot's operation trajectory information and environmental state change data.

[0067] The planned path instructions are sent to the simulator for execution. The simulator performs these operations in the simulation environment and outputs the robot's detailed operation trajectory and changes in the environmental state (in the form of updates to the simulator's internal state variables).

[0068] Step S300 involves executing the planned path instruction in a simulation environment, generating simulation execution results, and outputting the robot's operation trajectory information and environmental state change data, including:

[0069] Step S301: Input the planned path instruction into the simulation environment;

[0070] Step S302: Execute the path planning command in the simulation environment to generate simulation execution results;

[0071] Step S303: Based on the simulation execution results, output the robot's operation trajectory information and output the environmental state change data.

[0072] The planned path instructions are sent to the simulator for execution. The simulator performs these operations in the simulation environment and outputs the robot's detailed operation trajectory, which exists in the form of a structured file or memory object containing time series data (e.g., a list, where each element is a dictionary representing the state of a time step: [{'time':0.1,'joints':[..],'gripper_pose':[..],'image_path':'img_001.jpg'},...]) as well as changes in the environmental state (in the form of updates to the simulator's internal state variables).

[0073] A simulator is a complex software application (such as MuJoCo) that creates a virtual 3D environment that allows robots to operate, test, and learn safely and efficiently in a computer, thus avoiding the use of expensive and fragile physical hardware. Its core consists of a physics engine (simulating physical laws such as gravity and collisions), a collision detection system, a rendering engine (generating camera visual images), and a robot model interface. It interacts with external programs through a set of application programming interfaces (APIs), receiving control commands, providing sensor data, and world state information. All of this depends on the input scene description file to define the environment and objects, and the internal data structure tracks the state in real time, ultimately outputting simulated sensor readings, state information, and an optional visualization interface.

[0074] Step S400: Verify the simulation execution result according to the success detection conditions of the decomposed task. If the verification is successful, store the natural language task instruction, decomposed task, operation trajectory information and success label as multimodal information in the database.

[0075] Based on the success detection conditions generated by the LLM, the execution result of the current task is verified. The input is the current environment state (in the form of structured data obtained from the simulator API, such as a dictionary containing object poses) and the detection code (in the form of the previously generated code string). The output is a "success" or "failure" judgment, which exists as a boolean value (True or False). If successful, the data containing the task description (text string), subtask description (text string), operation trajectory (time series data structure), and success label (boolean value) is stored in the database.

[0076] In step S400, the simulated execution result is verified according to the success detection conditions of the decomposed task. If the verification is successful, the natural language task instructions, decomposed tasks, operation trajectory information, and success labels are stored in the database as multimodal information, including:

[0077] Step S401: Verify the simulation execution result based on the success detection conditions of the decomposed task and the environmental state change data. If the verification is successful, proceed to step S402; if the verification fails, proceed to step S403.

[0078] Step S402: Store the natural language task instructions, decomposed tasks, operation trajectory information, and success labels as multimodal information in the database;

[0079] Step S403: Re-execute the natural language task instruction decomposition process until successful verification. Then, store the natural language task instruction, decomposed task, operation trajectory information, and success label as multimodal information in the database.

[0080] Based on the success detection conditions generated by the LLM previously, the execution result of the current task is verified. The input is the current environment state (in the form of structured data obtained from the simulator API, such as a dictionary containing object poses) and the detection code (in the form of the previously generated code string). The output is a "success" or "failure" judgment, which exists as a boolean value (True or False). If it fails, the decomposition process is tried again until it succeeds. The natural language task instruction decomposition process refers to decomposing the natural language task instructions again based on the large language model to obtain decomposed tasks and generating success detection conditions for the decomposed tasks. The decomposed tasks are converted into robot operation primitive instructions, and the robot's planned path instructions are generated based on the robot operation primitive instructions. The planned path instructions are executed in the simulation environment to generate simulation execution results and output the robot's operation trajectory information and environmental state change data. The simulation execution results are verified based on the success detection conditions of the decomposed tasks. The specific steps have been described in the above embodiments and will not be repeated here. If successful, the data containing the task description (text string), subtask description (text string), operation trajectory (time series data structure), and success label (boolean value) will be stored in the database as the final output of this stage. Its form is a structured data record (e.g., a JSON record stored in a file or database: {'task':'Place blocks into the box','subtask':'Grab blocks','trajectory':[{'time':...,'joints':...}],'success':True}).

[0081] Step S500: Extract multimodal information from the database, train the initial diffusion strategy model based on the multimodal information, and generate a robot decision model after training.

[0082] Based on successful experience data collected from the database, multimodal information is extracted. Using this multimodal information, a robot "brain"—the diffusion policy model—is trained to understand language commands, observe the environment, and autonomously complete tasks. The diffusion policy model (Diffuser) is a decision generation method based on diffusion models, primarily used in reinforcement learning (RL) tasks. It generates decisions by learning the probability distribution of trajectories, offering unique advantages and application scenarios. Diffusion Model Basics: Diffusion models are a class of generative models based on Markov chains for progressive denoising, initially applied in image generation, audio synthesis, and other fields. Its core idea is to progressively add noise to the data through forward diffusion, and then recover the data from the noise through reverse denoising. Diffusion Policy Model: In reinforcement learning, the diffusion policy model represents the robot's visual motion policy as a conditional denoising diffusion process to generate robot behavior. It generates decisions by learning the probability distribution of trajectories, rather than using traditional policy gradient methods.

[0083] The technical implementation is as follows: Trajectory data collection: Initial trajectories, consisting of a series of states and actions, are collected from expert data or reinforcement learning environments. Diffusion model training: The trajectory data is forward-divided, with noise added progressively. Then, a neural network is trained to denoise the trajectory, enabling it to recover the trajectory from the noise. Inverse denoising and decision generation: Given an initial state and a target state, inverse denoising is performed starting from the noisy trajectory to generate a complete trajectory, and actions are extracted from it for environmental interaction.

[0084] Step S500, which involves extracting multimodal information from the database, training the initial diffusion strategy model based on the multimodal information, and generating a robot decision model after training, includes:

[0085] Step S501: Construct the initial diffusion strategy model;

[0086] Step S502: Extract multimodal information based on the database, input the multimodal information into the encoding layer of the initial diffusion strategy model, and generate multimodal fusion information;

[0087] Step S503: Train the initial diffusion strategy model based on the multimodal fusion information, and generate a robot decision model after training is completed.

[0088] First, the successful experience data collected in Phase 1 is extracted from the database as the main input. This data includes visual observations (the images seen by the robot's camera), language commands (the corresponding task or sub-task descriptions), and the robot's own state information (such as joint angles and end effector posture, collectively referred to as proprioception). This multimodal information is fed into different coding layers of the diffusion model (Layer 1, Layer 2, ... Layer N in the diagram), where these coding layers process and fuse this information.

[0089] Since the diffusion model already has a training foundation, the second stage simply uses the data generated in the first stage as the dataset, and the rest is the basic training process, which will not be described in detail here.

[0090] Step S503, which involves training the initial diffusion strategy model based on the multimodal fusion information and generating a robot decision model after training, includes:

[0091] Step S531: Obtain multimodal fusion information, extract the action sequence and condition information from the multimodal fusion information; add real noise to the action sequence, input the action sequence with added real noise and the corresponding condition information into the initial diffusion strategy model, and obtain the prediction noise output by the initial diffusion strategy model;

[0092] Step S532: Update the parameters of the initial diffusion strategy model based on the difference between the predicted noise and the actual noise;

[0093] Step S533: Obtain the predicted noise output by the initial diffusion strategy model after parameter update, calculate the difference between the predicted noise and the real noise, until the difference between the predicted noise and the real noise is less than a preset threshold, the diffusion strategy model training is completed, and the robot decision model is generated.

[0094] During training, the core task of the diffusion model is to learn how to predict the original added noise from a noisy action sequence, based on current conditional information (visual, linguistic, state). Therefore, at each step of training, the model receives a noisy action sequence and corresponding conditional information as input, and outputs a prediction of the added noise. This "predicted noise" is compared with the actual added "real noise," and the difference between them is calculated (e.g., using mean squared error, MSE). This difference is the loss. Then, through backpropagation, the parameters within the diffusion model are updated based on this loss; this is the output of the internal loop at this stage (the gradient of the parameter update). This process is repeated until the model can accurately predict the noise, meaning it has learned to generate correct, smooth action sequences under given conditions. Finally, the trained diffusion policy model is the output of this stage, and it can be deployed on a robot to perform practical tasks.

[0095] Step S600: Control the robot based on the robot decision model.

[0096] After generating the robot decision model, precise robot control can be achieved based on the language input by the user.

[0097] This invention eliminates the need for manual teaching and reward function design, automatically generating data and learning strategies solely based on task description. The automated and scalable data generation process significantly improves the efficiency and robustness of robot learning, reducing reliance on manual teaching and reward design. Furthermore, it utilizes LLM guidance and a "verification and retry" mechanism to collect high-quality data containing failure recovery experience. Ultimately, the robot "brain" refined through a diffusion strategy not only boasts superior performance, surpassing the original data collection process in complex, long-term tasks requiring common-sense reasoning, but also inherits strong robust recovery capabilities, enabling it to respond to natural language commands to execute multiple tasks and demonstrating excellent potential for transfer from simulation to reality.

[0098] It should be noted that there is no necessary order between the above steps. Those skilled in the art will understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0099] Another embodiment of the present invention provides an artificial intelligence-based robot control device, which corresponds one-to-one with the artificial intelligence-based robot control method described in the above embodiments. For example... Figure 3 As shown, device 1 includes:

[0100] The task decomposition module 100 is used to acquire natural language task instructions, decompose the natural language task instructions based on a large language model to obtain decomposed tasks, and generate successful detection conditions for the decomposed tasks.

[0101] The task conversion module 200 is used to convert the decomposed task into robot operation primitive instructions, and generate robot planning path instructions based on the robot operation primitive instructions.

[0102] The path planning simulation module 300 is used to execute the path planning instructions in a simulation environment, generate simulation execution results, and output the robot's operation trajectory information and environmental state change data.

[0103] The verification and storage module 400 is used to verify the simulation execution result according to the success detection conditions of the decomposed task. If the verification is successful, the natural language task instruction, decomposed task, operation trajectory information and success label are stored in the database as multimodal information.

[0104] The diffusion strategy training module 500 is used to extract multimodal information based on the database, train the initial diffusion strategy model based on the multimodal information, and generate a robot decision model after training is completed.

[0105] The control module 600 is used to control the robot based on the robot decision model.

[0106] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0107] In one embodiment, the task decomposition module 100 is specifically used for:

[0108] Obtain natural language task instructions and input the natural language task instructions into the large language model;

[0109] Obtain the decomposed subtask sequence or plan tree output by the large language model;

[0110] Obtain the success detection conditions for each subtask in the subtask sequence generated by the large language model or for each subplan in the plan tree.

[0111] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0112] In one embodiment, the task conversion module 200 is specifically used for:

[0113] Select the corresponding robot operation primitive based on the content of each subtask or each sub-plan;

[0114] Select the corresponding robot operation primitive instruction according to the robot operation primitive;

[0115] The robot operation primitive instructions are executed to call the corresponding underlying modules of the robot and generate the robot's planned path instructions.

[0116] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0117] In one embodiment, the path planning simulation module 300 is specifically used for:

[0118] Input the planned path command into the simulation environment;

[0119] Execute the path planning instructions in the simulation environment to generate simulation execution results;

[0120] Based on the simulation results, the robot's operation trajectory information is output, and the environmental state change data is also output.

[0121] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0122] In one embodiment, the verification and storage module 400 is specifically used for:

[0123] The simulation execution results are verified based on the successful detection conditions of the decomposed task and the environmental state change data;

[0124] If the verification is successful, the natural language task instructions, decomposed tasks, operation trajectory information and success labels will be stored in the database as multimodal information;

[0125] If verification fails, the natural language task instruction decomposition process is re-executed until verification is successful. Then, the natural language task instruction, decomposed task, operation trajectory information, and success label are stored in the database as multimodal information.

[0126] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0127] In one embodiment, the diffusion strategy training module 500 is specifically used for:

[0128] Construct an initial diffusion strategy model;

[0129] Multimodal information is extracted from the database and input into the encoding layer of the initial diffusion strategy model to generate multimodal fusion information.

[0130] The initial diffusion strategy model is trained based on the multimodal fusion information, and a robot decision model is generated after training is completed.

[0131] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0132] In one embodiment, the diffusion strategy training module 500 is further configured to:

[0133] Acquire multimodal fusion information, extract action sequences and condition information from the multimodal fusion information; add real noise to the action sequences, input the action sequences with added real noise and the corresponding condition information into the initial diffusion strategy model, and obtain the prediction noise output by the initial diffusion strategy model;

[0134] The parameters of the initial diffusion strategy model are updated based on the difference between the predicted noise and the actual noise.

[0135] The predicted noise output by the initial diffusion strategy model after parameter update is obtained, and the difference between the predicted noise and the real noise is calculated. The diffusion strategy model training is completed when the difference between the predicted noise and the real noise is less than a preset threshold, and the robot decision model is generated.

[0136] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0137] This invention provides an artificial intelligence-based robot control device that significantly improves the efficiency and robustness of robot learning through an automated and large-scale data generation process. It not only reduces the reliance on human instruction and reward design, but also collects high-quality data containing failure recovery experience by utilizing LLM guidance and a "verification and retry" mechanism. Finally, the robot "brain" refined through a diffusion strategy not only has excellent performance, surpassing the original data collection process in complex, long-term tasks that require common sense reasoning, but more importantly, it inherits strong robust recovery capabilities, can respond to natural language commands to execute multiple tasks, and demonstrates good potential for transfer from simulation to reality.

[0138] Another embodiment of the present invention provides a computer device, which may be a server, and its internal structure diagram may be as follows. Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side artificial intelligence-based robot control method.

[0139] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of an artificial intelligence-based robot control method.

[0140] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0141] Obtain natural language task instructions, decompose the natural language task instructions based on a large language model to obtain decomposed tasks, and generate successful detection conditions for the decomposed tasks.

[0142] The decomposed task is converted into robot operation primitives, and the robot's planned path instructions are generated based on the robot operation primitives.

[0143] The planned path instructions are executed in a simulation environment to generate simulation execution results and output the robot's operation trajectory information and environmental state change data.

[0144] The simulation execution results are verified according to the success detection conditions of the decomposed task. If the verification is successful, the natural language task instructions, decomposed tasks, operation trajectory information and success labels are stored in the database as multimodal information.

[0145] Multimodal information is extracted from the database, and the initial diffusion strategy model is trained based on the multimodal information. After training, a robot decision model is generated.

[0146] The robot is controlled based on the robot decision-making model.

[0147] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0148] Obtain natural language task instructions, decompose the natural language task instructions based on a large language model to obtain decomposed tasks, and generate successful detection conditions for the decomposed tasks.

[0149] The decomposed task is converted into robot operation primitives, and the robot's planned path instructions are generated based on the robot operation primitives.

[0150] The planned path instructions are executed in a simulation environment to generate simulation execution results and output the robot's operation trajectory information and environmental state change data.

[0151] The simulation execution results are verified according to the success detection conditions of the decomposed task. If the verification is successful, the natural language task instructions, decomposed tasks, operation trajectory information and success labels are stored in the database as multimodal information.

[0152] Multimodal information is extracted from the database, and the initial diffusion strategy model is trained based on the multimodal information. After training, a robot decision model is generated.

[0153] The robot is controlled based on the robot decision-making model.

[0154] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0156] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can exist in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0158] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

[0159] Among other things, conditional language such as “can,” “may,” “may,” or “may,” unless otherwise specifically stated or otherwise understood as in the context in which they are used, is generally intended to convey that a particular implementation may include (but not others) certain features, elements, and / or operations. Therefore, such conditional language is also generally intended to imply that features, elements, and / or operations are necessary for one or more implementations in any way, or that one or more implementations must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular implementation.

[0160] The contents already described herein in this specification and accompanying drawings include examples of methods and apparatuses capable of providing artificial intelligence-based robot control. It is certainly not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of this disclosure, but it will be appreciated that many other combinations and substitutions of the disclosed features are possible. Therefore, it will be apparent that various modifications can be made to this disclosure without departing from the scope or spirit of this disclosure. Furthermore, or in alternatives, other embodiments of this disclosure may become apparent from consideration of this specification and accompanying drawings and from practice of this disclosure as presented herein. It is intended that the examples presented in this specification and accompanying drawings be considered illustrative rather than restrictive in all respects. Although specific terminology is used herein, it is used in a general and descriptive sense and is not intended for limiting purposes.

Claims

1. A robot control method based on artificial intelligence, characterized in that... The method comprises: acquiring a natural language task instruction, decomposing the natural language task instruction based on a large language model to obtain a decomposition task, and generating a success detection condition of the decomposition task; converting the decomposition task into a robot operation primitive instruction, and generating a planning path instruction of the robot according to the robot operation primitive instruction; executing the planning path instruction in a simulation environment, generating a simulation execution result, and outputting operation trajectory information and environment state change data of the robot; verifying the simulation execution result according to the success detection condition of the decomposition task, and if the verification is successful, storing the natural language task instruction, the decomposition task, the operation trajectory information and a success label as multi-modal information in a database; extracting multi-modal information based on the database, training an initial diffusion strategy model based on the multi-modal information, and generating a robot decision model after the training is completed; controlling the robot based on the robot decision model; the conversion of the decomposition task into the robot operation primitive instruction, and the generation of the planning path instruction of the robot according to the robot operation primitive instruction, comprising: selecting a corresponding robot operation primitive according to the content of each subtask or each subplan; selecting a corresponding robot operation primitive instruction according to the robot operation primitive; executing the robot operation primitive instruction, calling a corresponding bottom module of the robot, and generating a planning path instruction of the robot; feeding the subtasks or subplans into a decision module in turn, and the decision module selecting a bottom robot operation primitive according to the content of the subtasks or subplans and outputting the selected primitive and parameters; wherein the existence form is a function call instruction inside a program or a data structure containing instructions and parameters; the primitive and the parameters jointly constitute a robot operation primitive instruction; when the primitive instruction is executed, the system calls the corresponding bottom module in a preset order; if it is a grasping primitive, one or more grasping poses are determined by a grasping sampler according to a target object and environment information obtained by a simulator; a motion planner generates a time sequence composed of joint angles or end poses, i.e., a planning path instruction, based on the current state of the robot as a starting point and the grasping pose as a target, and in combination with a motion path planned by an environment model. 2.The artificial intelligence-based robot control method of claim 1, wherein, The acquisition of the natural language task instruction, the decomposition of the natural language task instruction based on the large language model to obtain the decomposition task, and the generation of the success detection condition of the decomposition task, comprising: acquiring a natural language task instruction, and inputting the natural language task instruction into the large language model; acquiring a decomposed subtask sequence or plan tree output by the large language model; acquiring a success detection condition of each subtask in the subtask sequence or each subplan in the plan tree generated by the large language model. 3.The artificial intelligence-based robot control method of claim 1, wherein, The execution of the planning path instruction in the simulation environment, the generation of the simulation execution result, and the output of the operation trajectory information and the environment state change data of the robot, comprising: inputting the planning path instruction into the simulation environment; executing the planning path instruction in the simulation environment to generate a simulation execution result; According to the simulation execution result, the operation trajectory information of the robot is output, and the environment state change data is output. 4.The artificial intelligence-based robot control method of claim 1, wherein, The simulation execution result is verified according to the success detection condition of the decomposition task, and if the verification is successful, the natural language task instruction, the decomposition task, the operation trajectory information and the success label are stored in the database as multi-modal information, including: The simulation execution result is verified according to the success detection condition of the decomposition task and the environment state change data; If the verification is successful, the natural language task instruction, the decomposition task, the operation trajectory information and the success label are stored in the database as multi-modal information; If the verification fails, the decomposition process of the natural language task instruction is re-executed until the verification is successful, and the natural language task instruction, the decomposition task, the operation trajectory information and the success label are stored in the database as multi-modal information. 5.The artificial intelligence-based robot control method of claim 1, wherein, The multi-modal information is extracted based on the database, the initial diffusion strategy model is trained based on the multi-modal information, and the robot decision model is generated after the training is completed, including: An initial diffusion strategy model is constructed; The multi-modal information is extracted based on the database, and the multi-modal information is input into the encoding layer of the initial diffusion strategy model to generate multi-modal fusion information; The initial diffusion strategy model is trained based on the multi-modal fusion information, and the robot decision model is generated after the training is completed. 6.The artificial intelligence-based robot control method of claim 5, wherein, The multi-modal fusion information is obtained, the action sequence and the condition information in the multi-modal fusion information are extracted, the action sequence is added with a real noise, the action sequence with the added real noise and the corresponding condition information are input into the initial diffusion strategy model, and the predicted noise output by the initial diffusion strategy model is obtained. According to the difference between the predicted noise and the real noise, the parameters of the initial diffusion strategy model are updated; The predicted noise output by the initial diffusion strategy model after the parameter update is obtained, the difference between the predicted noise and the real noise is calculated, and until the difference between the predicted noise and the real noise is less than a preset threshold, the diffusion strategy model training is completed, and the robot decision model is generated. The device comprises:

7. An artificial intelligence-based robot control device characterized by comprising: A task decomposition module is configured to obtain a natural language task instruction, decompose the natural language task instruction based on a large language model to obtain a decomposition task, and generate a success detection condition of the decomposition task. A task conversion module is configured to convert the decomposition task into a robot operation primitive instruction, and generate a planning path instruction of the robot according to the robot operation primitive instruction. A planning path simulation module is configured to execute the planning path instruction in a simulation environment, generate a simulation execution result, and output operation trajectory information of the robot and environment state change data. A verification and storage module is configured to verify the simulation execution result according to the success detection condition of the decomposition task, and if the verification is successful, store the natural language task instruction, the decomposition task, the operation trajectory information and the success label in the database as multi-modal information. ​ The diffusion strategy training module is configured to extract multi-modal information based on the database, train an initial diffusion strategy model based on the multi-modal information, and generate a robot decision model after the training is completed. The control module is configured to control the robot based on the robot decision model. The robot operation primitive instruction is converted from the decomposition task, and a planning path instruction of the robot is generated according to the robot operation primitive instruction, including: According to the content of each subtask or each subplan, a corresponding robot operation primitive is selected; According to the robot operation primitive, a corresponding robot operation primitive instruction is selected; The robot operation primitive instruction is executed, a corresponding bottom module of the robot is called, and a planning path instruction of the robot is generated; The subtask or subplan is sequentially sent to a decision module, the decision module selects a bottom robot operation primitive according to the content of the subtask or subplan, and outputs the selected primitive and parameters; The primitive and the parameters jointly constitute a robot operation primitive instruction. When the primitive instruction is executed, the system calls a corresponding bottom module in a preset order; If it is a grasping primitive, one or more grasping poses are determined by a grasping sampler according to a target object and environment information obtained by a simulator; A motion planner generates a time sequence composed of joint angles or end poses, i.e., a planning path instruction, based on the current state of the robot as a starting point and the grasping pose as a target, and in combination with an environment model planning motion path. The computer device includes at least one processor; and 8. A computer device, comprising: The memory is in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the steps of the artificial intelligence-based robot control method of any one of claims 1-6. The computer readable storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to enable the one or more processors to execute the steps of the artificial intelligence-based robot control method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, ​

Images