Robot control method and device based on artificial intelligence, equipment and medium

Abstract

The invention relates to an artificial intelligence technology, can be applied to business system platforms of medical health, financial science and technology and the like, and discloses a robot control method, device and equipment based on artificial intelligence and a medium. Generating a planning path instruction of the robot according to the decomposition task; executing the path planning instruction in the simulation environment, and outputting operation track information and environment state change data of the robot; the simulation execution result is verified, and if verification succeeds, the natural language task instruction, the decomposition task, the operation track information and the success label serve as multi-modal information to be stored; training the initial diffusion strategy model based on the multi-modal information to generate a robot decision model; and controlling the robot based on the robot decision model. Through the automatic and large-scale data generation process, the robot learning efficiency and robustness are remarkably improved, and a natural language instruction can be responded to execute multiple tasks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based robot control method, device, equipment and medium. Background Art

[0002] With the development of technology, the application fields of robots are becoming increasingly broad, covering the medical and financial fields, bringing revolutionary changes to the industry.

[0003] In the medical field, surgical robots, rehabilitation robots, nursing robots, and medical service robots have emerged. Surgical robots combine AI, robotics, and medical technologies to assist or completely replace doctors in surgical operations. Using high-precision robotic arms and advanced vision systems, surgical robots can precisely locate and remove lesions, reducing surgical risks and recovery time, thereby improving surgical success rates and patient safety. Rehabilitation robots combine AI, robotics, and rehabilitation medicine theory to provide patients with personalized rehabilitation training programs. By simulating the movement patterns and mechanical properties of the human body, rehabilitation robots can precisely train and stimulate patients' joints, muscles, and other areas, promoting the recovery of motor function. Nursing robots combine AI, robotics, and nursing medicine theory to provide patients with personalized daily care services, such as body positioning, medication management, and health monitoring. Medical service robots are primarily used for auxiliary tasks within hospitals, such as patient guidance, medication delivery, and disinfection, improving hospital operational efficiency and service quality.

[0004] In the financial sector, chatbots, leveraging natural language processing technology, can provide 24 / 7 customer service, answering inquiries and handling diverse needs such as account inquiries, transfers, and financial product recommendations. This intelligent customer service approach not only improves customer satisfaction but also significantly reduces labor costs for financial institutions. Based on a customer's history and preferences, robots can provide personalized service recommendations, such as customized financial product recommendations and investment advice, thereby strengthening customer retention. Fraud detection and risk management robots can also leverage AI to analyze trillions of data points, assessing transaction risks in seconds, improving fraud detection rates, and reducing false positives.

[0005] Existing robotic skill acquisition methods primarily rely on large-scale data collection (such as manual teaching and trial-and-error exploration) and effective policy learning from this large-scale data. However, these methods have significant limitations: manual teaching and labeling rely on significant manpower and are difficult to scale; autonomous exploration or the use of non-robotic data often lacks essential information, is of low quality, or requires complex cross-modal transfer; directly using large language models (LLMs) as underlying control strategies often lacks the accuracy and robustness of the physical world; and traditional behavior cloning methods rely heavily on difficult-to-obtain high-quality expert data. Consequently, existing robotic datasets lack essential information and are of poor quality. Low-level control relies solely on large speech models, resulting in poor accuracy and robustness. Summary of the Invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a robot control method, device, equipment and medium based on artificial intelligence, aiming to solve the problems in the prior art that the robot data set sample collection lacks necessary information, the data set quality is poor, and the control of the underlying layer only relies on large-scale voice models, resulting in poor accuracy and robustness.

[0007] The technical solutions of the present invention are as follows:

[0008] A first embodiment of the present invention provides a robot control method based on artificial intelligence, the method comprising:

[0009] Obtaining a natural language task instruction, decomposing the natural language task instruction based on a large language model to obtain a decomposed task, and generating a success detection condition for the decomposed task;

[0010] Converting the decomposed tasks into robot operation primitive instructions, and generating robot planning path instructions according to the robot operation primitive instructions;

[0011] Executing the planned path instructions in a simulation environment, generating simulation execution results, and outputting the robot's operation trajectory information and environmental state change data;

[0012] Verifying the simulation execution result according to the success detection condition of the decomposed task, and if the verification is successful, storing the natural language task instruction, decomposed task, operation trajectory information and success label as multimodal information in a database;

[0013] Extracting multimodal information based on the database, training an initial diffusion strategy model based on the multimodal information, and generating a robot decision model after the training is completed;

[0014] The robot is controlled based on the robot decision model.

[0015] Another embodiment of the present invention provides a robot control device based on artificial intelligence, the device comprising:

[0016] A task decomposition module is used to obtain natural language task instructions, decompose the natural language task instructions based on a large language model to obtain decomposed tasks, and generate success detection conditions for the decomposed tasks;

[0017] A task conversion module is used to convert the decomposed task into a robot operation primitive instruction, and generate a robot planning path instruction based on the robot operation primitive instruction;

[0018] A planning path simulation module is used to execute the planning path instructions in a simulation environment, generate simulation execution results, and output the robot's operation trajectory information and environment state change data;

[0019] A verification and storage module is used to verify the simulation execution result according to the success detection condition of the decomposed task. If the verification is successful, the natural language task instruction, decomposed task, operation trajectory information and success label are stored as multimodal information in a database;

[0020] a diffusion strategy training module, configured to extract multimodal information based on the database, train an initial diffusion strategy model based on the multimodal information, and generate a robot decision model after the training is completed;

[0021] A control module is used to control the robot based on the robot decision model.

[0022] Another embodiment of the present invention provides a computer device, the computer device comprising at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the above-mentioned artificial intelligence-based robot control method.

[0025] Another embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can perform the steps of the above-mentioned artificial intelligence-based robot control method.

[0026] Beneficial effects: The artificial intelligence-based robot control method, device, equipment and medium of the embodiments of the present invention can be achieved through the artificial intelligence-based robot control method, device, equipment and medium, including: obtaining natural language task instructions, decomposing the natural language task instructions based on a large language model to obtain decomposed tasks, and generating success detection conditions for the decomposed tasks; converting the decomposed tasks into robot operation primitive instructions, and generating robot planning path instructions based on the robot operation primitive instructions; executing the planning path instructions in a simulation environment to generate simulation execution results, and outputting the robot's operation trajectory information and environmental state change data; verifying the simulation execution results according to the success detection conditions of the decomposed tasks, and if the verification is successful, storing the natural language task instructions, decomposed tasks, operation trajectory information and success labels as multimodal information in a database; extracting multimodal information based on the database, training the initial diffusion strategy model based on the multimodal information, and generating a robot decision model after the training is completed; and controlling the robot based on the robot decision model. The present invention significantly improves the efficiency and robustness of robot learning through an automated and large-scale data generation process. It not only reduces the dependence on human instruction and reward design, but also uses LLM guidance and the "verification and retry" mechanism to collect high-quality data including failure recovery experience; the robot "brain" refined by the diffusion strategy ultimately has excellent performance, surpassing the original data collection process in complex, long-term, and common-sense reasoning tasks. More importantly, it inherits strong robust recovery capabilities, can respond to natural language instructions to perform multiple tasks, and demonstrates good simulation-to-reality migration potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic diagram of an application environment of an embodiment of an artificial intelligence-based robot control method of the present invention;

[0029] Figure 2 This is a flow chart of a preferred embodiment of a robot control method based on artificial intelligence of the present invention;

[0030] Figure 3 This is a functional module diagram of a preferred embodiment of an artificial intelligence-based robot control device of the present invention;

[0031] Figure 4A schematic structural diagram of a preferred embodiment of a computer device of the present invention;

[0032] Figure 5 This is another structural diagram of a preferred embodiment of a computer device of the present invention. DETAILED DESCRIPTION

[0033] To make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0034] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0035] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. Here, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0036] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to achieve the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural speech processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0037] The robot control method based on artificial intelligence provided by the embodiment of the present invention can be applied in Figure 1In an application environment, the client communicates with the server through a network. The client accesses the network or business platform of the server, and the server can obtain natural language task instructions, decompose the natural language task instructions based on a large language model to obtain decomposed tasks, and generate success detection conditions for the decomposed tasks; convert the decomposed tasks into robot operation primitive instructions, and generate robot planning path instructions based on the robot operation primitive instructions; execute the planning path instructions in a simulation environment, generate simulation execution results, and output the robot's operation trajectory information and environment state change data; verify the simulation execution results according to the success detection conditions of the decomposed tasks, and if the verification is successful, store the natural language task instructions, decomposed tasks, operation trajectory information and success labels as multimodal information in a database; extract multimodal information based on the database, train the initial diffusion strategy model based on the multimodal information, and generate a robot decision model after the training is completed; control the robot based on the robot decision model. In the present invention, the efficiency and robustness of robot learning are significantly improved through the automated and large-scale data generation process, which not only reduces the dependence on human instruction and reward design, but also uses LLM guidance and the "verification and retry" mechanism to collect high-quality data containing failure recovery experience. The robot "brain" refined by the diffusion strategy not only has excellent performance, but also surpasses the original data collection process in complex, long-time tasks that require common sense reasoning. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablets and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0038] In order to solve the above problems, the present invention provides a robot control method based on artificial intelligence. Figure 2 , Figure 2 This is a flow chart of a preferred embodiment of a robot control method based on artificial intelligence of the present invention. Figure 2 As shown, it includes:

[0039] Step S100: Obtain a natural language task instruction, decompose the natural language task instruction based on a large language model to obtain a decomposed task, and generate a success detection condition for the decomposed task.

[0040] The embodiment of the present invention mainly obtains the natural language task instructions issued by the user, and uses the natural language task instructions issued by the user as the input of a large language model. The natural language task instruction exists in the form of a text string (for example, "Please put the red building blocks on the table into the blue box"). This instruction is sent to the LLM module, and the LLM performs task understanding and planning to obtain the decomposed task. .LLM is a large pre-trained neural network model. It is usually a deep learning model with billions or even more parameters (such as the evaluated GPT series or LLaMA series models). It is called through a specific API interface or runs directly on the server.

[0041] In the healthcare field, LLM can analyze textual information such as patient symptom descriptions, medical history, and test results. Combined with medical knowledge bases, it provides doctors with possible diagnostic recommendations, helping them make diagnoses more accurately and quickly. For example, GoogleMed-PaLM achieved an overall accuracy of 86.5% on the USMLE exam, surpassing 90% of human test takers. It performed particularly well in clinical diagnoses (such as diabetes and cardiovascular disease) and drug interaction analysis. Based on the patient's specific condition and the latest medical research findings, LLM can recommend personalized treatment plans, including drug selection, dosage adjustments, and treatment cycles. LLM can interpret and analyze medical imaging reports, extracting key information to help doctors better understand imaging results and improve diagnostic efficiency. For example, the Dia-LLaMA framework combines the LLaMA2-7B model with pre-trained ViT3D to process high-dimensional CT data and generate reports, demonstrating excellent performance on the CTRG-Chest-548K dataset.

[0042] In the financial sector, LLMs can be used to upgrade financial customer service: for example, Bloomberg GPT specializes in handling financial conversations and understanding professional terminology; 24 / 7 multilingual service supports seamless communication for global clients; and personalized financial advisors provide customized investment advice based on client risk preferences. In terms of financial information processing, LLMs can also be used for real-time market analysis, processing massive amounts of financial news, social media, and financial report data; intelligent report generation, automatically writing investment analysis reports and credit rating summaries; and regulatory document parsing, rapidly extracting key information from SEC filings and prospectuses.

[0043] Step S100, i.e., obtaining a natural language task instruction, decomposing the natural language task instruction based on a large language model to obtain a decomposed task, and generating a success detection condition for the decomposed task, includes:

[0044] Step S101: obtaining a natural language task instruction, and inputting the natural language task instruction into the large language model;

[0045] Step S102: obtaining a decomposed subtask sequence or plan tree output by the large language model;

[0046] Step S103: Obtain the success detection conditions of each subtask in the subtask sequence or each subplan in the plan tree generated by the large language model.

[0047] The system obtains natural language task instructions from users and uses them as input to the large language model. These instructions are presented as text strings (e.g., "Please place the red block on the table into the blue box"). These instructions are fed into the LLM module, which performs task understanding and planning, resulting in a decomposed task. The system then outputs a decomposed subtask sequence or plan tree, presented as structured text or a list / tree data structure within a program (e.g., a Python list ['approach the red block', 'grab the red block', 'move above the blue box', 'place the red block']), as well as inferred success check conditions for each subtask or the overall task, presented as text strings containing executable code (e.g., a Python code string "def check_success(sim_state):return is_object_inside(sim_state,'red_block','blue_box')").

[0048] Step S200: convert the decomposed task into a robot operation primitive instruction, and generate a robot planning path instruction according to the robot operation primitive instruction.

[0049] Decomposing a task into multiple subtasks or a plan tree is difficult for the robot to execute. The decomposed tasks need to be further converted into robot operation primitives. Based on these robot operation primitives, the robot's planned path instructions are generated.

[0050] Robotic primitives are the lowest-level, indivisible functional units that constitute a robot's basic capabilities and behaviors. They are the "atomic operations" of a robotic system. These primitives form the building blocks of a robot's complex behaviors, and through combination, they enable advanced functionality.

[0051] Primitive categories include: Perception Primitives, such as visual processing: pixel acquisition, edge detection, feature extraction; auditory processing: sound wave reception, speech feature extraction; tactile processing: pressure sensing, texture recognition; environmental perception: distance measurement, object recognition, SLAM (simultaneous localization and mapping).

[0052] The Motion Primitives class includes: basic movement: linear motion, arc motion, and stop; joint control: position control, speed control, and torque control; grasping action: pre-grasp posture and grasping force control; path planning: A* algorithm and RRT (rapid exploration random tree).

[0053] The Decision Primitives class includes: state judgment: condition detection, threshold comparison; behavior selection: if-then rules, utility function evaluation; task decomposition: goal disassembly, subtask generation.

[0054] Interaction primitives include: voice interaction: speech recognition, speech synthesis; physical interaction: force control, contact response; social interaction: expression generation, body language.

[0055] Step S200, that is, converting the decomposed task into a robot operation primitive instruction, and generating a robot-executed planning path instruction according to the robot operation primitive instruction, includes:

[0056] Step S201: selecting a corresponding robot operation primitive according to the content of each subtask or each subplan;

[0057] Step S202: selecting a corresponding robot operation primitive instruction according to the robot operation primitive;

[0058] Step S203: execute the robot operation primitive instruction, call the corresponding underlying module of the robot, and generate the robot's planned path instruction.

[0059] These subtasks (in the form of text strings, such as "grab the red block") are sequentially fed into a decision module, which selects the appropriate underlying robot operation primitive based on the subtask content and outputs the selected primitive and its parameters, which exist in the form of function call instructions within the program or a data structure containing instructions and parameters (for example, a dictionary {'primitive':

[0060] 'grasp_object','target':'red_block'}, which will trigger the actual function call). Execute these primitive instructions (whose input form is a function call and the corresponding numerical or text parameters, such as robot.execute_primitive({'primitive':'grasp_object',

[0061] 'target':'red_block'})), the system calls the corresponding underlying modules in sequence: first, if it is a grasping primitive, the grasping sampler determines one or more suitable grasping postures based on the target object (its input parameter is the object name or ID in the form of a text string) and the environmental information (structured geometric data) obtained from the simulator, and its output is a list of numerical data containing 6D pose (position and orientation); then, the motion planner takes the current robot state (the input is the joint angle or end pose of a numerical vector) as the starting point, and the grasping posture or other target posture obtained in the previous step (the input is 6D pose numerical data) as the target, and plans a collision-free motion path in combination with the environmental model (the input is the scene geometric data). Its output is a time series consisting of joint angles or end poses.

[0062] The decision module serves as the "translator" and "dispatcher" connecting high-level task planning with the underlying robot's specific actions. Its core function is to translate subtask descriptions (such as "pick up a cup") output by the Large Language Model (LLM) into executable robot actions through logical reasoning and information conversion. It implements a set of processes through program code (typically a Python script), including conditional statements to handle different tasks and object types. It calls the simulator's API to obtain environmental state information (such as object location and attributes), and may even call the LLM's API again to obtain more specific execution details (such as specifying which object to grasp). Ultimately, it outputs function call instructions and corresponding parameters for other underlying modules, such as the grasp sampler and motion planner.

[0063] The grasp sampler is a dedicated software module or algorithm whose core task is to help the robot figure out how to stably grasp the target object with its hand (gripper); it is based on geometric analysis of the three-dimensional model of the target object and the gripper (for example, finding contact points that can form force closure or shape closure) or random sampling strategies (for example, generating a large number of candidate grasping postures on the surface of the object or in the surrounding space, and then screening them according to criteria such as stability and reachability, such as uniform sampling in the object point cloud). Through these algorithms, and based on the input object and gripper model data and the current posture of the object, it finally outputs one or more six-degree-of-freedom posture data describing feasible grasping methods.

[0064] The motion planner is a core software module or algorithm whose main function is to calculate a path for the robot to move safely from its current state to its target state without colliding with the environment or itself. It does this by searching in a high-dimensional "configuration space" consisting of all possible combinations of the robot's joint angles. Sampling-based planning algorithms (such as RRT - Rapidly Exploring Random Trees) are usually used. These algorithms can efficiently explore space and find feasible paths. At the same time, they must be combined with the robot's kinematic model and collision detection module (often using simulator functions to determine whether postures collide). Based on the input robot and environment models, the starting and target states, it ultimately outputs a motion trajectory consisting of a series of intermediate points (joint angles or end postures) for the robot to execute.

[0065] In the financial field, special primitives for financial robots include: data collection primitives: market data subscription (subscribe_market_data); news sentiment analysis (sentiment_analysis); transaction execution primitives: order generation (generate_order); risk check (risk_check); transaction execution (execute_trade); compliance primitives include: AML check (aml_check) and regulatory report generation (generate_report).

[0066] Step S300: Execute the planned path instruction in a simulation environment, generate simulation execution results, and output the robot's operation trajectory information and environment state change data.

[0067] The planned path instructions are sent to the simulator for execution. The simulator performs these operations in the simulated environment and outputs the robot's detailed operation trajectory and changes in the environment state (in the form of updates to the simulator's internal state variables).

[0068] Among them, step S300, that is, executing the planned path instruction in the simulation environment, generating a simulation execution result, and outputting the robot's operation trajectory information and environment state change data, includes:

[0069] Step S301: inputting the planned path instruction into the simulation environment;

[0070] Step S302: executing the planned path instruction in the simulation environment to generate a simulation execution result;

[0071] Step S303: output the robot's operation trajectory information and the environmental state change data according to the simulation execution result.

[0072] The planned path instructions are sent to the simulator for execution. The simulator performs these operations in the simulation environment and outputs the detailed operation trajectory of the robot, which exists in the form of a structured file or memory object containing time series data (for example, a list where each element is a dictionary representing the state of a time step: [{'time':0.1,'joints':[..],'gripper_pose':[..],'image_path':'img_001.jpg'},...]) and changes in the environment state (in the form of updates to the internal state variables of the simulator).

[0073] A simulator is a complex software application (such as MuJoCo) that creates a virtual three-dimensional environment that allows robots to operate, test, and learn safely and efficiently in a computer, thus avoiding the use of expensive and fragile physical hardware; its underlying core consists of a physics engine (simulating physical laws such as gravity and collisions), a collision detection system, a rendering engine (generating camera visual images), and a robot model interface. It interacts with external programs through a set of application programming interfaces (APIs), receives control instructions, provides sensor data, and world state information. All of this relies on input scene description files to define the environment and objects, and the internal data structure tracks the state in real time, and finally outputs simulated sensor readings, state information, and an optional visualization interface.

[0074] Step S400: Verify the simulation execution result according to the success detection condition of the decomposed task. If the verification is successful, store the natural language task instruction, decomposed task, operation trajectory information and success label as multimodal information in the database.

[0075] Verify the execution results of the current task based on the success detection conditions previously generated by the LLM. The input is the current environment state (in the form of structured data obtained from the simulator API, such as a dictionary containing object poses) and the detection code (in the form of a previously generated code string). The output is a "success" or "failure" judgment, which exists in the form of a Boolean value (True or False). If successful, the data with the task description (text string), subtask description (text string), operation trajectory (time series data structure), and success label (Boolean value) is stored in the database.

[0076] Among them, in step S400, the simulation execution result is verified according to the success detection condition of the decomposed task. If the verification is successful, the natural language task instruction, decomposed task, operation trajectory information and success label are stored in the database as multimodal information, including:

[0077] Step S401: verify the simulation execution result according to the success detection condition of the decomposed task and the environmental state change data. If the verification is successful, execute step S402; if the verification fails, execute step S403;

[0078] Step S402: storing the natural language task instructions, decomposed tasks, operation trajectory information, and success labels as multimodal information in a database;

[0079] Step S403: re-execute the natural language task instruction decomposition process until the verification is successful, and store the natural language task instruction, decomposed task, operation trajectory information and success label as multimodal information in the database.

[0080] According to the successful detection conditions generated by the previous LLM, the execution result of the current task is verified. The input is the current environment state (in the form of structured data obtained from the simulator API, such as a dictionary containing the object posture) and the detection code (in the form of a previously generated code string). The output is a judgment of "success" or "failure", which exists in the form of a Boolean value (True or False). If it fails, try to execute the decomposition process again until it succeeds; wherein the natural language task instruction decomposition process refers to decomposing the natural language task instruction again based on the large language model to obtain the decomposed task and generate the successful detection conditions of the decomposed task; converting the decomposed task into a robot operation primitive instruction, and generating the robot's planned path instruction according to the robot operation primitive instruction; executing the planned path instruction in the simulation environment, generating a simulation execution result, and outputting the robot's operation trajectory information and environment state change data; verifying the simulation execution result according to the successful detection conditions of the decomposed task. The specific steps have been described in the above embodiment and will not be repeated here. If successful, the data with the task description (text string), subtask description (text string), operation trajectory (time series data structure) and success label (Boolean value) are stored in the database as the final output of this stage, which exists in the form of a structured data record (for example, a JSON record stored in a file or database: {'task':'Put the block into the box', 'subtask':'Grab the block', 'trajectory':[{'time':..., 'joints':...}], 'success':True}).

[0081] Step S500: extracting multimodal information based on the database, training an initial diffusion strategy model based on the multimodal information, and generating a robot decision model after the training is completed.

[0082] Based on successful experience data collected from a database, multimodal information is extracted. Based on this multimodal information, a robot "brain"—the diffusion strategy model—is trained to understand language commands, observe the environment, and autonomously complete tasks. The diffusion strategy model (Diffuser) is a decision-making method based on diffusion models, primarily used for reinforcement learning (RL) tasks. It generates decisions by learning the probability distribution of trajectories, and has unique advantages and application scenarios. Diffusion model basics: Diffusion models are a type of generative model based on Markov chain step-by-step denoising, initially applied to fields such as image generation and audio synthesis. Their core idea is to gradually add noise to the data through forward diffusion and then recover the data from the noise through reverse denoising. Diffusion strategy model: In reinforcement learning, the diffusion strategy model represents the robot's visual-motor strategy as a conditional denoising diffusion process to generate robot behavior. It generates decisions by learning the probability distribution of trajectories, rather than using traditional policy gradient methods.

[0083] The technology is implemented as follows: Trajectory data collection: Initial trajectories are collected from expert data or reinforcement learning environments. These trajectories consist of a series of states and actions. Diffusion model training: Forward diffusion is performed on the trajectory data, gradually adding noise. A neural network is then trained to denoise the trajectories and recover them from the noise. Inverse denoising and decision generation: Given an initial state and a target state, reverse denoising is performed from the noisy trajectory to generate a complete trajectory, from which actions are extracted for interaction with the environment.

[0084] Step S500, i.e., extracting multimodal information based on the database, training an initial diffusion strategy model based on the multimodal information, and generating a robot decision model after the training is completed, includes:

[0085] Step S501: construct an initial diffusion strategy model;

[0086] Step S502: extracting multimodal information based on the database, inputting the multimodal information into the encoding layer of the initial diffusion strategy model, and generating multimodal fusion information;

[0087] Step S503: training the initial diffusion strategy model based on the multimodal fusion information, and generating a robot decision model after the training is completed.

[0088] First, the successful experience data collected in Phase 1 is extracted from the database as the primary input. This data includes visual observations (what the robot's camera sees), language instructions (describing the corresponding task or subtask), and the robot's own state information (such as joint angles and end-effector posture, collectively known as proprioception). This multimodal information is fed into the different encoding layers of the diffusion model (Layer 1, Layer 2, ..., Layer N in the figure), which process and fuse the information.

[0089] Since the diffusion model already has a foundation in training, the second stage only uses the data generated in the first stage as the dataset. The rest is the basic training process and will not be described in detail.

[0090] Step S503, i.e., training the initial diffusion strategy model based on the multimodal fusion information and generating a robot decision model after the training is completed, includes:

[0091] Step S531: Acquire multimodal fusion information, extract action sequences and condition information from the multimodal fusion information; add a real noise to the action sequence, input the action sequence with the real noise added and the corresponding condition information into an initial diffusion strategy model, and obtain predicted noise output by the initial diffusion strategy model;

[0092] Step S532: updating the parameters of the initial diffusion strategy model according to the difference between the predicted noise and the actual noise;

[0093] Step S533: Obtain the predicted noise output by the initial diffusion strategy model after parameter update, calculate the difference between the predicted noise and the actual noise, until the difference between the predicted noise and the actual noise is less than a preset threshold, the diffusion strategy model training is completed, and the robot decision model is generated.

[0094] During training, the core task of the diffusion model is to learn how to predict the original noise from a noisy action sequence based on the current conditional information (visual, language, and state). Therefore, at each training step, the model receives a noisy action sequence and the corresponding conditional information as input and outputs a prediction of the added noise. This "predicted noise" is compared with the actual "true noise" and the difference between them is calculated (for example, using the mean squared error (MSE)). This difference is the loss. The backpropagation algorithm then uses this loss to update the internal parameters of the diffusion model. This is the output of the internal loop at this stage (the gradient of the parameter update). This process is repeated until the model can accurately predict the noise, meaning it has learned to generate correct, smooth action sequences under the given conditions. Ultimately, the trained diffusion policy model is the output of this stage and can be deployed on a robot to perform real-world tasks.

[0095] Step S600: Control the robot based on the robot decision model.

[0096] After the robot decision model is generated, precise robot control can be achieved based on the language input by the user.

[0097] The embodiment of the present invention does not require manual teaching and reward function design, and automatically completes data generation and strategy learning based on task description alone. The automated and large-scale data generation process significantly improves the efficiency and robustness of robot learning, not only reducing the dependence on human teaching and reward design, but also utilizing LLM guidance and the "verification and retry" mechanism to collect high-quality data containing failure recovery experience. Ultimately, the robot "brain" refined through the diffusion strategy not only has excellent performance, surpassing the original data collection process in complex, long-term, and common-sense reasoning tasks, but more importantly, it inherits strong robust recovery capabilities, can respond to natural language instructions to perform multiple tasks, and demonstrates good simulation-to-reality migration potential.

[0098] It should be noted that there is not necessarily a certain order between the above steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or may be executed interchangeably, etc.

[0099] Another embodiment of the present invention provides a robot control device based on artificial intelligence, which corresponds one-to-one to the robot control method based on artificial intelligence in the above embodiment. Figure 3 As shown, the device 1 includes:

[0100] The task decomposition module 100 is used to obtain natural language task instructions, decompose the natural language task instructions based on a large language model to obtain decomposed tasks, and generate success detection conditions for the decomposed tasks;

[0101] A task conversion module 200 is used to convert the decomposed task into a robot operation primitive instruction, and generate a robot planning path instruction according to the robot operation primitive instruction;

[0102] The planning path simulation module 300 is used to execute the planning path instruction in a simulation environment, generate simulation execution results, and output the robot's operation trajectory information and environment state change data;

[0103] The verification and storage module 400 is used to verify the simulation execution result according to the success detection condition of the decomposed task. If the verification is successful, the natural language task instruction, decomposed task, operation trajectory information and success label are stored as multimodal information in the database;

[0104] A diffusion strategy training module 500 is configured to extract multimodal information based on the database, train an initial diffusion strategy model based on the multimodal information, and generate a robot decision model after the training is completed;

[0105] The control module 600 is configured to control the robot based on the robot decision model.

[0106] The specific implementation method is shown in the method embodiment, which will not be repeated here.

[0107] In one embodiment, the task decomposition module 100 is specifically configured to:

[0108] Obtaining a natural language task instruction, and inputting the natural language task instruction into the large language model;

[0109] Obtaining a decomposed subtask sequence or plan tree output by the large language model;

[0110] Obtain a success detection condition for each subtask in the subtask sequence or each subplan in the plan tree generated by the large language model.

[0111] The specific implementation method is shown in the method embodiment, which will not be repeated here.

[0112] In one embodiment, the task conversion module 200 is specifically configured to:

[0113] Selecting a corresponding robot operation primitive according to the content of each subtask or each subplan;

[0114] Selecting a corresponding robot operation primitive instruction according to the robot operation primitive;

[0115] Execute the robot operation primitive instructions, call the corresponding underlying module of the robot, and generate the robot's planning path instructions.

[0116] The specific implementation method is shown in the method embodiment, which will not be repeated here.

[0117] In one embodiment, the planning path simulation module 300 is specifically configured to:

[0118] inputting the planned path instruction into the simulation environment;

[0119] Executing the planned path instruction in the simulation environment to generate a simulation execution result;

[0120] According to the simulation execution result, the operation trajectory information of the robot is output, and the environmental state change data is output.

[0121] The specific implementation method is shown in the method embodiment, which will not be repeated here.

[0122] In one embodiment, the verification and storage module 400 is specifically configured to:

[0123] Verifying the simulation execution result according to the success detection conditions of the decomposed tasks and the environmental state change data;

[0124] If the verification is successful, the natural language task instructions, decomposed tasks, operation trajectory information and success labels are stored in the database as multimodal information;

[0125] If the verification fails, the natural language task instruction decomposition process is re-executed until the verification is successful. The natural language task instruction, decomposed task, operation trajectory information and success label are stored in the database as multimodal information.

[0126] The specific implementation method is shown in the method embodiment, which will not be repeated here.

[0127] In one embodiment, the diffusion strategy training module 500 is specifically configured to:

[0128] Construct an initial diffusion strategy model;

[0129] Extracting multimodal information based on the database, inputting the multimodal information into the encoding layer of the initial diffusion strategy model to generate multimodal fusion information;

[0130] The initial diffusion strategy model is trained based on the multimodal fusion information, and a robot decision model is generated after the training is completed.

[0131] The specific implementation method is shown in the method embodiment, which will not be repeated here.

[0132] In one embodiment, the diffusion strategy training module 500 is further configured to:

[0133] Acquire multimodal fusion information, extract action sequences and condition information from the multimodal fusion information; add a real noise to the action sequence, input the action sequence after adding the real noise and the corresponding condition information into an initial diffusion strategy model, and obtain predicted noise output by the initial diffusion strategy model;

[0134] updating the parameters of the initial diffusion strategy model according to the difference between the predicted noise and the actual noise;

[0135] The predicted noise output by the initial diffusion strategy model after parameter update is obtained, and the difference between the predicted noise and the actual noise is calculated until the difference between the predicted noise and the actual noise is less than the preset threshold. The diffusion strategy model training is completed and the robot decision model is generated.

[0136] The specific implementation method is shown in the method embodiment, which will not be repeated here.

[0137] The present invention provides an artificial intelligence-based robot control device, which significantly improves the efficiency and robustness of robot learning through an automated and large-scale data generation process. It not only reduces the dependence on human instruction and reward design, but also uses LLM guidance and a "verify and retry" mechanism to collect high-quality data including failure recovery experience; the robot "brain" finally refined by the diffusion strategy not only has excellent performance, surpassing the original data collection process in complex, long-time tasks that require common sense reasoning, but more importantly, it inherits strong robust recovery capabilities, can respond to natural language instructions to perform multiple tasks, and shows good simulation-to-reality migration potential.

[0138] Another embodiment of the present invention provides a computer device, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a robot control method based on artificial intelligence.

[0139] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a robot control method based on artificial intelligence.

[0140] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0141] Obtaining a natural language task instruction, decomposing the natural language task instruction based on a large language model to obtain a decomposed task, and generating a success detection condition for the decomposed task;

[0142] Converting the decomposed tasks into robot operation primitive instructions, and generating robot planning path instructions according to the robot operation primitive instructions;

[0143] Executing the planned path instructions in a simulation environment, generating simulation execution results, and outputting the robot's operation trajectory information and environmental state change data;

[0144] Verifying the simulation execution result according to the success detection condition of the decomposed task, and if the verification is successful, storing the natural language task instruction, decomposed task, operation trajectory information and success label as multimodal information in a database;

[0145] Extracting multimodal information based on the database, training an initial diffusion strategy model based on the multimodal information, and generating a robot decision model after the training is completed;

[0146] The robot is controlled based on the robot decision model.

[0147] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0148] Obtaining a natural language task instruction, decomposing the natural language task instruction based on a large language model to obtain a decomposed task, and generating a success detection condition for the decomposed task;

[0149] Converting the decomposed tasks into robot operation primitive instructions, and generating robot planning path instructions according to the robot operation primitive instructions;

[0150] Executing the planned path instructions in a simulation environment, generating simulation execution results, and outputting the robot's operation trajectory information and environmental state change data;

[0151] Verifying the simulation execution result according to the success detection condition of the decomposed task, and if the verification is successful, storing the natural language task instruction, decomposed task, operation trajectory information and success label as multimodal information in a database;

[0152] Extracting multimodal information based on the database, training an initial diffusion strategy model based on the multimodal information, and generating a robot decision model after the training is completed;

[0153] The robot is controlled based on the robot decision model.

[0154] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0155] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0156] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the objectives of the present embodiments as needed.

[0157] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, or of course, can also be implemented by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the relevant technology can be embodied in the form of a software product. This computer software product can be present in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0158] It should be noted that if software tools or components other than those of our company appear in the embodiments of this application, they are only used for illustration and do not represent actual use.

[0159] Conditional language such as "can," "may," or "might," among others, unless specifically stated otherwise or otherwise understood within the context as used, is generally intended to convey that particular embodiments can include, while other embodiments do not, particular features, elements, and / or operations. Thus, such conditional language is also generally intended to imply that features, elements, and / or operations are anyway required for one or more embodiments or that one or more embodiments must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or to be performed in any particular embodiment.

[0160] What has been described herein in this specification and the accompanying drawings includes examples that can provide methods and devices for controlling robots based on artificial intelligence. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it is recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is apparent that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure. In addition, or in the alternative, other embodiments of the present disclosure may be apparent from consideration of this specification and the accompanying drawings and from the practice of the present disclosure as presented herein. It is intended that the examples set forth in this specification and the accompanying drawings be considered in all respects to be illustrative and not restrictive. Although specific terms are employed herein, they are used in a general and descriptive sense and are not used for limiting purposes.

Claims

1. A robot control method based on artificial intelligence, characterized in that , the method comprises: Obtaining a natural language task instruction, decomposing the natural language task instruction based on a large language model to obtain a decomposed task, and generating a success detection condition for the decomposed task; Converting the decomposed tasks into robot operation primitive instructions, and generating robot planning path instructions according to the robot operation primitive instructions; Executing the planned path instructions in a simulation environment, generating simulation execution results, and outputting the robot's operation trajectory information and environmental state change data; Verifying the simulation execution result according to the success detection condition of the decomposed task, and if the verification is successful, storing the natural language task instruction, decomposed task, operation trajectory information and success label as multimodal information in a database; Extracting multimodal information based on the database, training an initial diffusion strategy model based on the multimodal information, and generating a robot decision model after the training is completed; The robot is controlled based on the robot decision model.

2. The robot control method based on artificial intelligence according to claim 1, characterized in that: The obtaining of natural language task instructions, decomposing the natural language task instructions based on a large language model to obtain decomposed tasks, and generating success detection conditions for the decomposed tasks include: Obtaining a natural language task instruction, and inputting the natural language task instruction into the large language model; Obtaining a decomposed subtask sequence or plan tree output by the large language model; Obtain a success detection condition for each subtask in the subtask sequence or each subplan in the plan tree generated by the large language model.

3. The robot control method based on artificial intelligence according to claim 2, characterized in that: The step of converting the decomposed tasks into robot operation primitive instructions and generating a planning path instruction for the robot according to the robot operation primitive instructions includes: Select the corresponding robot operation primitive according to the content of each subtask or each subplan; Selecting a corresponding robot operation primitive instruction according to the robot operation primitive; Execute the robot operation primitive instructions, call the corresponding underlying module of the robot, and generate the robot's planning path instructions.

4. The robot control method based on artificial intelligence according to claim 1, characterized in that: The step of executing the planned path instruction in the simulation environment, generating simulation execution results, and outputting the robot's operation trajectory information and environmental state change data includes: inputting the planned path instruction into the simulation environment; Executing the planned path instruction in the simulation environment to generate a simulation execution result; According to the simulation execution result, the operation trajectory information of the robot is output, and the environmental state change data is output.

5. The robot control method based on artificial intelligence according to claim 1, characterized in that: The simulation execution result is verified according to the success detection condition of the decomposed task. If the verification is successful, the natural language task instruction, decomposed task, operation trajectory information and success label are stored in the database as multimodal information, including: Verifying the simulation execution result according to the success detection conditions of the decomposed tasks and the environmental state change data; If the verification is successful, the natural language task instructions, decomposed tasks, operation trajectory information and success labels are stored in the database as multimodal information; If the verification fails, the decomposition process of the natural language task instructions is re-executed until the verification is successful. The natural language task instructions, decomposed tasks, operation trajectory information and success labels are stored in the database as multimodal information.

6. The robot control method based on artificial intelligence according to claim 1, characterized in that: The extracting multimodal information based on the database, training the initial diffusion strategy model based on the multimodal information, and generating a robot decision model after the training is completed include: Construct an initial diffusion strategy model; Extracting multimodal information based on the database, inputting the multimodal information into the encoding layer of the initial diffusion strategy model to generate multimodal fusion information; The initial diffusion strategy model is trained based on the multimodal fusion information, and a robot decision model is generated after the training is completed.

7. The robot control method based on artificial intelligence according to claim 6, characterized in that: The initial diffusion strategy model is trained based on the multimodal fusion information, and a robot decision model is generated after the training is completed, including: Acquire multimodal fusion information, extract action sequences and condition information from the multimodal fusion information; add a real noise to the action sequence, input the action sequence sequence with the real noise added and the corresponding condition information into an initial diffusion strategy model, and obtain predicted noise output by the initial diffusion strategy model; updating the parameters of the initial diffusion strategy model according to the difference between the predicted noise and the actual noise; The predicted noise output by the initial diffusion strategy model after parameter update is obtained, and the difference between the predicted noise and the actual noise is calculated until the difference between the predicted noise and the actual noise is less than the preset threshold. The diffusion strategy model training is completed and the robot decision model is generated.

8. A robot control device based on artificial intelligence, characterized in that: The device comprises: A task decomposition module is used to obtain natural language task instructions, decompose the natural language task instructions based on a large language model to obtain decomposed tasks, and generate success detection conditions for the decomposed tasks; A task conversion module is used to convert the decomposed task into a robot operation primitive instruction, and generate a robot planning path instruction based on the robot operation primitive instruction; A planning path simulation module is used to execute the planning path instructions in a simulation environment, generate simulation execution results, and output the robot's operation trajectory information and environment state change data; A verification and storage module is used to verify the simulation execution result according to the success detection condition of the decomposed task. If the verification is successful, the natural language task instruction, decomposed task, operation trajectory information and success label are stored as multimodal information in a database; a diffusion strategy training module, configured to extract multimodal information based on the database, train an initial diffusion strategy model based on the multimodal information, and generate a robot decision model after the training is completed; A control module is used to control the robot based on the robot decision model.

9. A computer device, characterized in that: The computer device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the artificial intelligence-based robot control method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to perform the steps of the artificial intelligence-based robot control method described in any one of claims 1 to 7.

Images