Robot control and training method and device based on adaptive clustering agent

Abstract

The embodiment of the invention discloses a self-adaptive control method, training method and device for a clustered agent, and relates to the technical field of artificial intelligence and computer scientificity, and the method comprises the steps: constructing a joint agent for each joint of a robot, configuring a connection relation, constructing a central agent to process input information, and constructing a joint intelligent agent for each joint of the robot; a parameter sharing and synchronous updating mechanism is set for each joint intelligent body and the central intelligent body, and the central intelligent body is used for outputting a signal to control each joint intelligent body to obtain real motion data and evaluation data; and according to the real motion data and the evaluation data, training and optimizing each joint intelligent body and central intelligent body through imitation learning and reinforcement learning, and then performing robot control. According to the invention, the intelligent agent is constructed for each joint of the robot, the connection relation is configured, and the central intelligent agent for processing multi-element input information is constructed, so that the problems that the prior art cannot realize multi-task and multi-scene robot control and is poor in generalization ability and flexibility are solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer science technology, and in particular to an adaptive control method, training method and device for a swarm intelligent body. Background Art

[0002] With the development of multimodal large intelligent agents, current robots have acquired universal language and image understanding capabilities. However, there are still many challenges in robot control. These challenges mainly include the problem of heterogeneous robot sensors and structures. Different robots require different control strategies and different control variable dimensions, making it difficult to handle highly diverse and complex mechanical control in one intelligent agent.

[0003] In addition, the signals at the cognitive level and the execution level are far apart, making it difficult to consider them in the same intelligent agent. When using layered agents for control, it is difficult to define the division of tasks and communication methods for agents at different levels. Artificial definition can easily lead to loss of effect and error transmission.

[0004] At the same time, current robot control methods are difficult to use for multiple tasks and multiple scenarios. A large amount of data must be collected in advance for each task and scenario, and they do not have the ability to generalize out-of-distribution.

[0005] Therefore, there is a need for a robot control method that can realize complex, nonlinear, multi-joint robot control, has language, text, image interaction, and multi-scene and multi-task generalization capabilities, and can adapt to heterogeneous robots and different control requirements. Summary of the invention

[0006] The embodiments of the present invention provide a robot control method based on an adaptive cluster intelligent agent to solve the problem that the prior art cannot realize multi-task and multi-scenario robot control and has poor generalization ability and flexibility. The technical solution is as follows:

[0007] According to one aspect of the present invention, a robot control method based on adaptive cluster intelligent agents includes: constructing joint intelligent agents for each joint of the robot and configuring the connection relationship between each joint intelligent agent; the joint intelligent agents include arm intelligent agents and waist intelligent agents; constructing a central intelligent agent for the robot, and processing input information through the central intelligent agent; the input information includes image signals, text signals and voice signals; setting a parameter sharing and synchronous update mechanism for the joint intelligent agents and the central intelligent agent; the central intelligent agent is used to output signals to control the joint intelligent agents; acquiring real motion data and evaluation data, and controlling the robot after training and optimizing the joint intelligent agents and the central intelligent agent through imitation learning and reinforcement learning according to the real motion data and evaluation data.

[0008] In one embodiment, the arm agent is connected to the finger root agent and the finger joint agent, and the waist agent is connected to the hip joint agent, the knee joint agent and the ankle joint agent.

[0009] In one embodiment, the formula for information transmission between the interconnected intelligent agents is as follows:

[0010] y t , M t =F i (x t ;M t-1 ,θ i );

[0011] Among them, θ i represents the parameters of the ith agent, M t Represents the memory tensor at time t, y t represents the agent output at time t, F i represents the structure of the ith agent, x t Represents the input at time t.

[0012] In one of the embodiments, a central intelligent agent is constructed for the robot, and processing input information by the central intelligent agent is achieved through the following steps: if the input information is a text signal, signal processing is performed on the text signal to obtain a symbol sequence, and the symbol sequence is converted into a vector sequence through the embedded coding layer of the central intelligent agent; if the input information is an image signal, the image signal is encoded into a vector sequence through a convolutional neural network; if the input information is a speech signal, the speech signal is processed into a vector sequence through a speech processing algorithm and a sequence coding network.

[0013] In one embodiment, obtaining real motion data and evaluation data, and training and optimizing the joint agents and the central agent through imitation learning and reinforcement learning according to the real motion data and evaluation data are achieved through the following steps: obtaining real motion data and evaluation data of each type of robot performing tasks; the types include wheeled, wheeled-footed, quadrupedal, and bipedal; the tasks include gripping with grippers, grasping with dexterous hands, and navigation and object searching; training the joint agents and the central agent according to the real motion data through an imitation learning algorithm; training the joint agents and the central agent according to the real motion data and evaluation data through a reinforcement learning algorithm.

[0014] According to one aspect of the present invention, a robot training method based on adaptive swarm intelligent agents comprises: collecting training data; the training data comprises expert demonstration data and automation strategy data; the expert demonstration data is data generated by an expert demonstrating the execution of a task; the automation strategy data is data generated by the execution of an intelligent agent; during the training process, an expert evaluation mechanism is introduced to automatically evaluate the intelligent agent, and the parameters of each intelligent agent are optimized by reducing the error between the expert demonstration action and the intelligent agent output, and each intelligent agent is trained by imitation learning; during the training process, an automation evaluation mechanism is introduced to score the input and output recorded in each intelligent agent, and each intelligent agent is optimized using reinforcement learning according to the scoring results, and each intelligent agent is trained by reinforcement learning.

[0015] In one embodiment, the optimization of the parameters of each agent by reducing the error between the expert demonstration action and the agent output is achieved by the following steps: inputting the input information recorded by the expert demonstration data into the agent for operation, obtaining output predictions based on the operation data of each agent at each moment, and calculating the loss value for training based on the output predictions and the output information recorded in the expert demonstration;

[0016] The specific formula for calculating the loss value is as follows:

[0017]

[0018] Among them, L IL represents the loss value of imitation learning, k represents the input index of the expert demonstration data, t represents the time, represents the output prediction of the agent, Represents the output data of the expert demonstration.

[0019] In one embodiment, the optimization of each agent using reinforcement learning according to the scoring results is achieved by the following steps: inputting the input and output recorded in each agent into an automated evaluation system to obtain an evaluation value, and inputting the input recorded in each agent into the agent to obtain an output prediction; calculating a loss value according to the output prediction, the recorded output and the evaluation value for training;

[0020] The specific formula for calculating the loss value is as follows:

[0021]

[0022] Among them, L RL represents the loss value of reinforcement learning, and γ is the attenuation coefficient, which takes values ​​between 0.95 and 0.99.

[0023] In one embodiment, the method further includes: obtaining a central training set, initializing a node neural network in the central intelligent agent, and setting shared properties of the node neural network parameters; the central training set includes a large amount of corpus, images, videos and multimodal alignment data; when optimizing the loss value, the gradient is back-propagated, and the gradient is transferred by communication between nodes so that all agent parameters are optimized; the central intelligent agent is hot-started using a multimodal pre-training model, the central intelligent agent is pre-trained using the central training set, and the central intelligent agent parameters are optimized according to the input and output of the central intelligent agent at each moment.

[0024] According to one aspect of the present invention, a robot control device comprises: a joint intelligent agent construction module, which is used to construct joint intelligent agents for each joint of the robot and configure the connection relationship between each joint intelligent agent; the joint intelligent agent comprises an arm intelligent agent and a waist intelligent agent; a central intelligent agent construction module, which is used to construct a central intelligent agent for the robot and process input information through the central intelligent agent; the input information comprises image signals, text signals and voice signals; a parameter mechanism setting module, which is used to set a parameter sharing and synchronous update mechanism for each joint intelligent agent and the central intelligent agent; the central intelligent agent is used to output signals to control each joint intelligent agent; a robot training module, which is used to obtain real motion data and evaluation data, and control the robot after training and optimizing each joint intelligent agent and the central intelligent agent through imitation learning and reinforcement learning according to the real motion data and evaluation data.

[0025] The beneficial effects brought by the technical solution provided by the present invention are:

[0026] In the above technical scheme, the present invention first constructs a joint intelligent agent for each joint of the robot and configures the connection relationship between each joint intelligent agent, constructs a central intelligent agent for the robot, processes input information through the central intelligent agent, the input information includes image signals, text signals and voice signals, sets a parameter sharing and synchronous update mechanism for each joint intelligent agent and the central intelligent agent, the central intelligent agent is used to output signals to control the joint intelligent agents, obtain real motion data and evaluation data, train and optimize each joint intelligent agent and the central intelligent agent through imitation learning and reinforcement learning according to the real motion data and evaluation data, and then control the robot, and the robot control method of the adaptive cluster intelligent agent is used. The method significantly improves the flexibility and intelligence of the robot's behavior. By building intelligent agents for each joint of the robot and configuring connection relationships, as well as building a central intelligent agent to process multiple input information, the robot's environmental adaptability and decision-making efficiency are enhanced. Parameter sharing and synchronous update mechanisms ensure the collaborative work between intelligent agents. Combining imitation learning with reinforcement learning, the robot can self-optimize based on real motion data and evaluation data, improve control accuracy and response speed, and provide an efficient and intelligent solution for robot applications in complex environments, thereby effectively solving the problems that existing technologies cannot achieve multi-task and multi-scenario robot control, and have poor generalization ability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0028] Figure 1 is a flow chart of a robot control method based on an adaptive swarm agent according to an exemplary embodiment;

[0029] Figure 2 is a topological diagram of input and output between various intelligent agents shown in an exemplary embodiment;

[0030] Figure 3 is a block diagram of a robot control device according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0032] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0033] To this end, the present invention provides a robot control method based on an adaptive swarm intelligent agent. By constructing a fine joint intelligent agent and a central intelligent agent and configuring their connection relationship, accurate and flexible robot motion control is achieved, which can effectively solve the problem that the prior art cannot achieve multi-task and multi-scenario robot control, and has poor generalization ability and flexibility. The robot control method based on an adaptive swarm intelligent agent is suitable for a robot control device, which can be an electronic device. The robot control method based on an adaptive swarm intelligent agent in the embodiment of the present invention can be applied to a variety of scenarios, such as robot control.

[0034] See also Figure 1 An embodiment of the present invention provides a robot control method based on an adaptive cluster intelligent agent, which is applicable to electronic devices.

[0035] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0036] like Figure 1 As shown, the method may include the following steps:

[0037] Step 110, constructing joint intelligent bodies for each joint of the robot and configuring the connection relationship between each joint intelligent body.

[0038] Among them, the joint agent includes the arm agent and the waist agent.

[0039] In one possible implementation, the arm agent is connected to the finger root agent and the finger joint agent, and the waist agent is connected to the hip joint agent, the knee joint agent and the ankle joint agent.

[0040] In one possible implementation, the formula for information transfer between connected agents is as follows:

[0041] y t , M t =F i (x t ;M t-1 ,θ i );

[0042] Among them, θ i represents the parameters of the ith agent, M t Represents the memory tensor at time t, y t represents the agent output at time t, F i represents the structure of the ith agent, x t Represents the input at time t.

[0043] Step 130, construct a central intelligent agent for the robot, and process input information through the central intelligent agent.

[0044] The input information includes image signals, text signals and voice signals.

[0045] In one possible implementation, if the input information is a text signal, the text signal is processed to obtain a symbol sequence, and the symbol sequence is converted into a vector sequence through the embedded coding layer of the central intelligent agent; if the input information is an image signal, the image signal is encoded into a vector sequence through a convolutional neural network; if the input information is a speech signal, the speech signal is processed into a vector sequence through a speech processing algorithm and a sequence coding network.

[0046] Step 150, setting up a parameter sharing and synchronous updating mechanism for each joint agent and the central agent.

[0047] Among them, the central intelligent agent is used to output signals to control each joint intelligent agent.

[0048] Step 170, obtain real motion data and evaluation data, train and optimize each joint agent and central agent through imitation learning and reinforcement learning based on the real motion data and evaluation data, and then control the robot.

[0049] In one possible implementation, the real motion data and evaluation data of each type of robot performing tasks are obtained, and each joint agent and central agent are trained according to the real motion data through an imitation learning algorithm, and each joint agent and central agent are trained according to the real motion data and evaluation data through a reinforcement learning algorithm.

[0050] Among them, the models include wheeled, wheeled-footed, quadrupedal, bipedal, etc., and the tasks include gripping with claws, grasping with dexterous hands, and navigation and searching, etc., which are not limited here.

[0051] In one possible implementation, an embodiment of the present invention provides a robot training method based on the above-mentioned adaptive swarm intelligent agent, which collects training data, introduces an expert evaluation mechanism during the training process to automatically evaluate the intelligent agent, and optimizes the parameters of each intelligent agent by reducing the error between the expert demonstration action and the intelligent agent output, and performs imitation learning training on each intelligent agent; during the training process, an automatic evaluation mechanism is introduced to score the input and output recorded in each intelligent agent, and each intelligent agent is optimized using reinforcement learning based on the scoring results, and reinforcement learning training is performed on each intelligent agent.

[0052] Among them, the training data includes expert demonstration data and automation strategy data. The expert demonstration data is the data generated by experts demonstrating the execution of tasks, and the automation strategy data is the data generated by the execution of intelligent agents.

[0053] In one possible implementation, the input information recorded in the expert demonstration data is input into the intelligent agent for operation, the output prediction is obtained based on the operation data of each intelligent agent at each moment, and the loss value is calculated based on the output prediction and the output information recorded in the expert demonstration for training.

[0054] The specific formula for calculating the loss value is as follows:

[0055]

[0056] Among them, L IL represents the loss value of imitation learning, k represents the input index of the expert demonstration data, t represents the time, represents the output prediction of the agent, Represents the output data of the expert demonstration.

[0057] In one possible implementation, the input and output recorded in each agent are input into an automated evaluation system to obtain an evaluation value, the input recorded in each agent is input into the agent to obtain an output prediction, and the loss value is calculated based on the output prediction, the recorded output, and the evaluation value for training.

[0058] The specific formula for calculating the loss value is as follows:

[0059]

[0060] Among them, L RL represents the loss value of reinforcement learning, and γ is the attenuation coefficient, which takes values ​​between 0.95 and 0.99.

[0061] In one possible implementation, a central training set is obtained, the node neural network in the central intelligent agent is initialized, and the shared properties of the node neural network parameters are set. When the loss value is optimized, the gradient is back-propagated, and the gradient is transferred by communication between nodes so that all the agent parameters are optimized. The central intelligent agent is hot-started using a multimodal pre-training model, and the central intelligent agent is pre-trained using the central training set. The central intelligent agent parameters are optimized according to the input and output of the central intelligent agent at each moment.

[0062] Among them, the core training set includes a large amount of corpus, images, videos and multimodal alignment data.

[0063] Through the above process, the embodiment of the present invention first constructs a joint intelligent agent for each joint of the robot and configures the connection relationship between each joint intelligent agent, constructs a central intelligent agent for the robot, processes input information through the central intelligent agent, the input information includes image signals, text signals and voice signals, sets a parameter sharing and synchronous update mechanism for each joint intelligent agent and the central intelligent agent, the central intelligent agent is used to output signals to control the joint intelligent agents, obtains real motion data and evaluation data, trains and optimizes each joint intelligent agent and the central intelligent agent through imitation learning and reinforcement learning according to the real motion data and evaluation data, and then controls the robot, and controls the robot through adaptive cluster intelligent agents The method significantly improves the flexibility and intelligence of the robot's behavior. By building intelligent agents for each joint of the robot and configuring connection relationships, as well as building a central intelligent agent to process multiple input information, the robot's environmental adaptability and decision-making efficiency are enhanced. Parameter sharing and synchronous update mechanisms ensure the collaborative work between intelligent agents. Combining imitation learning with reinforcement learning, the robot can self-optimize based on real motion data and evaluation data, improve control accuracy and response speed, and provide an efficient and intelligent solution for robot applications in complex environments, thereby effectively solving the problems that existing technologies cannot achieve multi-task and multi-scenario robot control and have poor generalization ability and flexibility.

[0064] In an exemplary embodiment, the present invention provides a robot control method based on an adaptive swarm agent for robot control.

[0065] The central agent is bound to the brain or eyes of the robot, and mechanical arm agents and mechanical leg root agents are established for the robot's hands and legs respectively. To support hand movements, a finger root agent and a finger joint agent are set up for each finger (assuming that each finger has two joints). To support leg movements, a hip joint agent, a knee joint agent, and an ankle joint agent are set up in turn.

[0066] Furthermore, in this modular design, there is gradient information transmission between each agent, and the specific formula is as follows:

[0067] y t , M t =F i (x t ;M t-1 ,θ i );

[0068] Among them, θ i represents the parameters of the ith agent, M t Represents the memory tensor at time t, y t represents the agent output at time t, F i represents the structure of the ith agent, x t Represents the input at time t.

[0069] Except for the central agent, all other joint control agents use the same agent structure F i and the same parameter θ i .

[0070] It is worth noting that the above-mentioned agents may include RNN, LSTM, Neural Turing Machine, Transformer, Mamba and other structures, and the specific agent structure corresponding to the agent design is not limited here.

[0071] Taking RNN / LSTM / Mamba and other structures as examples, x t represents a vector sequence, each vector represents a symbol; y t is the output, shape and x t Should be completely consistent; M t-1 Corresponding to the hidden state, it will be updated every time it receives a piece of input.

[0072] Taking Transformer as an example, M t-1 The corresponding Key-Value Cache will be expanded on this basis every time it is inferred.

[0073] Among them, the memory tensor M is initialized at each deployment t , the memory tensor of each agent is updated during reasoning, which enables the agent to adapt to different joints.

[0074] Specifically, the input signal mainly includes the original sensor or the sensor signal after preliminary processing and other related intelligent agent outputs. Including but not limited to: text signals, which are converted into a series of symbols through preliminary processing and converted into vector sequences through the embedding coding layer; image signals, which are encoded into vector matrices and sequences through convolutional neural networks, etc.; speech signals, which are processed into vector sequences through speech processing methods or sequence coding networks; the vector sequence output by the related intelligent agent is added with the prefix vector of the corresponding intelligent agent identifier. If multiple related intelligent agent outputs are received, these vector sequences are connected end to end to form a new vector sequence as output.

[0075] Specifically, the output signal includes the communication information of other related agents and the terminal output signal. If the output contains a specific prefix (the identification prefix of the related agent), the following information is considered to be the signal output to the related agent until an end symbol is encountered, which indicates that the signal output to the agent has ended; the terminal output signal includes voice, text, and actuator control signals, etc. The central agent is responsible for the output of voice and text, and other node networks output actuator control signals to control the movement of the robot.

[0076] like Figure 2 As shown in the figure, the topological structure of input and output between each intelligent agent is shown. The topological structure diagram presents a complex network connection, including original input 1, node 1, node 2 and node 3 and the connection relationship between them. Node 1 and original input 1 and node 2 transmit information to each other, node 2 and node 3 transmit information to each other, and terminal 4 and terminal 5 output control signals.

[0077] In the above-mentioned input and output communication process between nodes, the output information of the nodes is vectorized, so the gradient can be transmitted. Therefore, the above-mentioned multiple agents can be regarded as an agent as a whole for end-to-end gradient feedback. In the overall information transmission process, a noise can be added to the information transmitted between nodes during training and inference, which helps to improve the multimodality (distribution diversity) of the agent in processing the mapping between input and output.

[0078] Furthermore, when training each agent, training data is first collected. The data collection sources include: expert demonstration data, where the expert demonstration duration covers one or more complete tasks, or a series of tasks; the agent's automation strategy data, and the collected data is used as the original input (including images, voice, text, etc.) t representation) and raw output (including speech, text, actuator control signals, t The data contains various heterogeneous robot models, such as wheeled, wheeled-footed, quadrupedal, bipedal, and various tasks, such as gripping with claws, grasping with dexterous hands, and navigation and searching.

[0079] For the above collected data, an expert or automated evaluation mechanism is introduced to score the recorded sequential input and output. If the operation is wrong or the target is not achieved, negative feedback is given. If it is executed correctly or the target is achieved, positive feedback is given. Normally, zero feedback is given. The demonstration data sequence is as follows:

[0080]

[0081] Among them, the superscript k represents the kth trajectory, T represents the length of the collected sequence, and if it is confirmed that the quality of the trajectory meets expectations, the sequence can be evaluated without evaluation, and only the input and output sequences are retained, as shown in the following formula:

[0082]

[0083] Among them, if the collected data is demonstration and evaluation data, reinforcement learning is used for optimization. If the collected data is expert demonstration data, imitation learning is used for optimization.

[0084] Furthermore, we use massive amounts of corpus, images, videos, and multimodal alignment data to pre-train the central agent, initialize the node neural network, set all node neural networks to share or not share parameters, and for each sequence, initialize the memory tensors of all agents to 0 or random, calculate the input and output of each agent at each time moment, and obtain the output prediction of each actuator or terminal.

[0085] Specifically, if imitation learning is used, the loss function is as follows:

[0086]

[0087] Among them, L IL represents the loss value of imitation learning, k represents the input index of the real motion data, t represents the time, represents the output prediction, Output data representing the real motion data.

[0088] Specifically, if reinforcement learning is used, the loss function is as follows:

[0089] The specific formula for calculating the loss value based on the output prediction is as follows:

[0090]

[0091] Among them, L RL Represents the loss value of reinforcement learning.

[0092] Specifically, the operation data of each intelligent agent at each moment is obtained through the information gradient transmission between intelligent agents, the output prediction is obtained according to the operation data of each intelligent agent at each moment, the loss value is calculated according to the output prediction and the evaluation data, and the training is carried out until each intelligent agent converges, so as to obtain the trained intelligent agents, and each intelligent agent is deployed in the robot to control the operation of the robot.

[0093] Among them, the gradient is transmitted from the terminal output between each node through inter-node communication, and finally propagated to each agent. The pre-trained agents are used to hot-start some agents. For example, the central agent can be hot-started using a multi-modal pre-trained agent.

[0094] Through the above process, the embodiment of the present invention achieves a significant improvement in the flexibility and intelligence of the robot's behavior through the robot control method of adaptive cluster intelligent agents. By constructing intelligent agents for each joint of the robot and configuring connection relationships, as well as constructing a central intelligent agent for processing multiple input information, the robot's environmental adaptability and decision-making efficiency are enhanced. The parameter sharing and synchronous update mechanism ensures the collaborative work between intelligent agents. Combined with imitation learning and reinforcement learning, the robot can self-optimize based on real motion data and evaluation data, improve control accuracy and response speed, and provide an efficient and intelligent solution for robot applications in complex environments, thereby effectively solving the problem that the existing technology cannot achieve multi-task and multi-scenario robot control, and has poor generalization ability and flexibility.

[0095] The following is an embodiment of the device of the present invention, which can be used to execute the robot control method based on adaptive swarm intelligence involved in the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the method embodiment of the robot control method based on adaptive swarm intelligence involved in the present invention.

[0096] See also Figure 3 , a robot control device 800 is provided in an embodiment of the present invention.

[0097] The device 800 includes but is not limited to: a joint intelligent agent setting module 810, a central intelligent agent construction module 830, a parameter mechanism setting module 850 and a robot training module 870.

[0098] Among them, the joint intelligent body setting module 810 is used to construct joint intelligent bodies for each joint of the robot and configure the connection relationship between each joint intelligent body; the joint intelligent body includes an arm intelligent body and a waist intelligent body.

[0099] The central intelligent agent construction module 830 is used to construct a central intelligent agent for the robot and process input information through the central intelligent agent; the input information includes image signals, text signals and voice signals.

[0100] The parameter mechanism setting module 850 is used to set the parameter sharing and synchronous updating mechanism for each joint agent and the central agent; the central agent is used to output signals to control each joint agent.

[0101] The robot training module 870 is used to obtain real motion data and evaluation data, and to train and optimize each joint agent and central agent through imitation learning and reinforcement learning based on the real motion data and evaluation data, and then perform robot control.

[0102] It should be noted that the robot control provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the robot control device will be divided into different functional modules to complete all or part of the functions described above.

[0103] In addition, the robot control device provided in the above embodiment and the embodiment of the robot control method based on adaptive swarm intelligent agent belong to the same concept, and the specific way in which each module performs operations has been described in detail in the method embodiment and will not be repeated here.

[0104] Compared with the related art, the present invention has the following beneficial effects:

[0105] 1. The present invention constructs a joint intelligent body for each joint of the robot and configures the connection relationship between each joint intelligent body, constructs a central intelligent body for the robot, processes input information through the central intelligent body, and the input information includes image signals, text signals and voice signals. A parameter sharing and synchronous update mechanism is set for each joint intelligent body and the central intelligent body. The central intelligent body is used to output signals to control each joint intelligent body, obtain real motion data and evaluation data, and train and optimize each joint intelligent body and the central intelligent body through imitation learning and reinforcement learning according to the real motion data and evaluation data, and then control the robot. The robot control method of the adaptive cluster intelligent body is implemented. The flexibility and intelligence of the robot's behavior have been significantly improved. By building intelligent agents for each joint of the robot and configuring connection relationships, as well as building a central intelligent agent to process multiple input information, the robot's environmental adaptability and decision-making efficiency have been enhanced. Parameter sharing and synchronous update mechanisms ensure the collaborative work between intelligent agents. Combining imitation learning and reinforcement learning, the robot can self-optimize based on real motion data and evaluation data, improve control accuracy and response speed, and provide an efficient and intelligent solution for robot applications in complex environments, thereby effectively solving the problems that existing technologies cannot achieve multi-task and multi-scenario robot control, and have poor generalization ability and flexibility.

[0106] 2. The present invention can control heterogeneous robots and adapt to complex multi-environments, complex multi-task, and interactive robot intelligent body bases. The present invention can achieve unified control of heterogeneous robots through distributed control of joints, and can adapt to robots of arbitrary structures (wheeled, bipedal, quadrupedal, single-arm, double-arm, gripper, dexterous hand, special robot) through intelligent body parameters and the memory capacity of each joint.

[0107] 3. The present invention can utilize the memory capacity of each joint intelligent agent and the general knowledge ability of the central intelligent agent to realize the adaptation of a set of intelligent agents to different tasks and characteristics, thereby realizing self-learning and adaptive control after deployment.

[0108] 4. The present invention optimizes end-to-end so that the input and output between joints are implicitly optimized end-to-end, without the need to manually define the communication protocols and communication formats between the various structures. At the same time, it retains the use of dense vectors for efficient and accurate information transmission, more effective and accurate inter-module collaboration, avoids error transmission, and allows asynchronous control between joints, such as slower central reaction time and shorter control cycle of the underlying intelligent body, thereby achieving more flexible response.

[0109] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0110] The above descriptions are only some embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A robot control method based on adaptive swarm intelligence, characterized in that: The method comprises: Constructing joint intelligent bodies for each joint of the robot and configuring the connection relationship between each joint intelligent body; the joint intelligent body includes an arm intelligent body and a waist intelligent body; Building a central intelligent agent for the robot, and processing input information through the central intelligent agent; the input information includes image signals, text signals and voice signals; A mechanism for parameter sharing and synchronous updating is set for each joint intelligent agent and the central intelligent agent; the central intelligent agent is used to output signals to control each joint intelligent agent; Acquire real motion data and evaluation data, and perform robot control after training and optimizing the joint agents and the central agent through imitation learning and reinforcement learning based on the real motion data and evaluation data.

2. The robot control method based on adaptive swarm intelligence according to claim 1, characterized in that: The arm agent is connected to the finger root agent and the finger joint agent, and the waist agent is connected to the hip joint agent, the knee joint agent and the ankle joint agent.

3. The robot control method based on adaptive swarm intelligence as claimed in claim 2, characterized in that: The formula for information transmission between the interconnected agents is as follows: y t ,M t =F i (x t ;M t-1 ,θ i ); Among them, θ i represents the parameters of the ith agent, M t Represents the memory tensor at time t, y t represents the agent output at time t, F i represents the structure of the ith agent, x t Represents the input at time t.

4. The robot control method based on adaptive swarm intelligence according to claim 1, characterized in that: The step of constructing a central intelligent agent for the robot and processing input information through the central intelligent agent includes: If the input information is a text signal, signal processing is performed on the text signal to obtain a symbol sequence, and the symbol sequence is converted into a vector sequence through the embedded coding layer of the central intelligent agent; If the input information is an image signal, encoding the image signal into a vector sequence through a convolutional neural network; If the input information is a speech signal, the speech signal is processed into a vector sequence through a speech processing algorithm and a sequence coding network.

5. The robot control method based on adaptive swarm intelligence according to claim 1, characterized in that: The obtaining of real motion data and evaluation data, and training and optimizing the joint agents and the central agent through imitation learning and reinforcement learning according to the real motion data and evaluation data, comprises: Acquire real motion data and evaluation data of various robot models performing tasks; the robot models include wheeled, wheeled-footed, quadrupedal, and bipedal; the tasks include gripping with a gripper, grasping with a dexterous hand, and navigation and searching; Training the joint agents and the central agent according to the real motion data by using an imitation learning algorithm; The joint agents and the central agent are trained according to the real motion data and evaluation data through a reinforcement learning algorithm.

6. A robot training method based on adaptive swarm intelligence agent applied to any one of claims 1-8, characterized in that: The method comprises: Collect training data; the training data includes expert demonstration data and automation strategy data; the expert demonstration data is data generated by experts demonstrating the execution of tasks; the automation strategy data is data generated by the execution of intelligent agents; In the training process, an expert evaluation mechanism is introduced to automatically evaluate the intelligent agent, and the parameters of each intelligent agent are optimized by reducing the error between the expert demonstration action and the output of the intelligent agent, and the imitation learning training is performed on each intelligent agent; During the training process, an automated evaluation mechanism is introduced to score the input and output recorded in each intelligent agent, and reinforcement learning is used to optimize each intelligent agent based on the scoring results, and reinforcement learning training is performed on each intelligent agent.

7. The robot training method based on adaptive swarm intelligence according to claim 6, characterized in that: The optimizing of the parameters of each of the agents by reducing the error between the expert demonstration action and the agent output comprises: Inputting the input information recorded in the expert demonstration data into the agent for operation, obtaining output predictions based on the operation data of each agent at each moment, and calculating loss values ​​for training based on the output predictions and the output information recorded in the expert demonstration; The specific formula for calculating the loss value is as follows: Among them, L IL represents the loss value of imitation learning, k represents the input index of the expert demonstration data, t represents the time, represents the output prediction of the agent, Represents the output data of the expert demonstration.

8. The robot training method based on adaptive swarm intelligence according to claim 6, characterized in that: The optimizing each intelligent agent by using reinforcement learning according to the scoring results includes: Inputting the input and output recorded in each agent into the automated evaluation system to obtain an evaluation value, and inputting the input recorded in each agent into the agent to obtain an output prediction; Calculate the loss value for training based on the output prediction and the recorded output and the evaluation value; The specific formula for calculating the loss value is as follows: Among them, L RL represents the loss value of reinforcement learning, and γ is the attenuation coefficient, which takes values ​​between 0.95 and 0.

99.

9. The robot training method based on adaptive swarm intelligence according to claim 6, characterized in that: The method further comprises: Obtaining a central training set, initializing a node neural network in the central agent, and setting shared properties of the node neural network parameters; the central training set includes a large amount of corpus, images, videos, and multimodal alignment data; When the loss value is optimized, the gradient is transmitted through back propagation, and the gradient is transmitted by communication between nodes so that all agent parameters are optimized; The central intelligent agent is hot-started using a multimodal pre-training model, the central intelligent agent is pre-trained using the central training set, and the parameters of the central intelligent agent are optimized according to the input and output of the central intelligent agent at each moment.

10. A robot control device based on an adaptive swarm agent, characterized in that: The device comprises: A joint intelligent body construction module is used to construct joint intelligent bodies for each joint of the robot and configure the connection relationship between each joint intelligent body; the joint intelligent body includes an arm intelligent body and a waist intelligent body; A central intelligent agent building module, used to build a central intelligent agent for the robot, and process input information through the central intelligent agent; the input information includes image signals, text signals and voice signals; A parameter mechanism setting module, used to set a parameter sharing and synchronous updating mechanism for each joint agent and the central agent; the central agent is used to output signals to control each joint agent; The robot training module is used to obtain real motion data and evaluation data, and to control the robot after training and optimizing the joint agents and the central agent through imitation learning and reinforcement learning based on the real motion data and evaluation data.

Images