Multi-agent architecture based on online evolution

Abstract

The invention provides a multi-agent architecture based on online evolution, and relates to the technical field of multi-agent systems.According to the multi-agent architecture based on online evolution, actual rewards obtained by executing actions in the real world after control agents make decisions are compared with virtual rewards predicted in the virtual world; whether a sensing agent needs to be updated or not is determined according to a comparison result, expert knowledge of a generative model in a virtual world is used for assisting decision making of a control agent, generated information and existing information in the virtual world are fused to narrow the decision making range, collaboration of sensing, control and generation is achieved, and the control efficiency is improved. According to the architecture, feedback information obtained after a control agent executes actions in a physical environment serves as newly-added data to be added into a corresponding data set of a sensing agent, and a model of the sensing agent is adjusted through collaborative optimization of the newly-added data. The expandability, the interpretability and the convergence efficiency of the intelligent agent are ensured while the intelligent agent adapts to continuously changing environmental conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent systems, and in particular to a multi-agent architecture based on online evolution. Background Art Multi-agent systems consist of a group of autonomous entities and are often used to solve complex problems that require collective intelligence and collaboration. Multi-agent systems are widely used, and common application areas include cloud computing, autonomous driving, and smart warehousing. In multi-agent systems, collaborative perception and control technologies play a vital role. Multi-agent collaborative perception mainly studies how multiple agents can cooperate to perceive, interpret, and understand their environment. With the rapid development of artificial intelligence and deep learning, supervised learning has been widely used in most perception tasks. Among the many supervised learning network architectures, the architecture based on convolutional neural networks is the most successful and widely used. For the perception tasks of a single agent, target detection models such as RCNN and YOLO series can achieve rapid detection and positioning of targets. For the collaborative perception of multi-agents, it is necessary to perform pre-, mid-, or post-fusion of the perception data of multiple agents. Typical algorithms include Cooper, V2VNet, and Car2X. Multi-agent collaborative control focuses on studying how multiple autonomous agents coordinate their actions to achieve common goals. This method uses the collective cooperation of agents to enable them to complete tasks more efficiently than when a single agent acts alone. Currently, there are many algorithms that can realize multi-agent collaborative control, among which multi-agent reinforcement learning is one of the most mainstream algorithms. Multi-agent reinforcement learning can be divided into four categories: behavior analysis, communication learning, collaborative learning and imitation learning. Typical algorithms include IQL, IDQN, VDN, QMIX and MADDPG.

[0002] Typically, the workflow of a traditional multi-agent system is that the sensor first collects and interprets data from the environment, and then the controller makes decisions based on these results. In other words, most existing methods study multi-agent perception and multi-agent control algorithms separately. This model has been widely used in many fields, but this traditional workflow may have problems in dynamically changing environments. Since most perception algorithms rely on supervised learning and most perception models are trained from fixed data sets, when the environment changes, the sensor model may have recognition errors because the data set used for training does not contain new and changing environmental data, which may cause the controller to make wrong decisions and have adverse consequences during actual execution.

[0003] The development of end-to-end models can partially solve the above problems. These models take raw sensor data as input and directly output control actions. By minimizing the loss through back-propagation, they can jointly and globally optimize object perception and adapt to changing environments. Although end-to-end models are effective, they also face some challenges. First, end-to-end models are usually configured for specific tasks and may lack generality and scalability. In addition, they lack interpretability and are often referred to as "black boxes." Interpretability can not only provide performance and safety guarantees, but also improve public acceptance. However, current end-to-end models are still insufficient in this regard. In addition, end-to-end models require a large amount of data for training, which may consume a lot of computing resources and storage space, resulting in inefficient convergence.

[0004] Based on the above aspects, there is currently a lack of multi-agent architecture based on online evolution that can be used in non-stationary environments while ensuring scalability, explainability and convergence efficiency. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a multi-agent architecture based on online evolution, which enables the agents to adapt to the changing environmental conditions while ensuring scalability, interpretability and convergence efficiency. The architecture adds the feedback information obtained by the control agent after performing actions in the physical environment as new data to the corresponding data set of the perception agent, and adjusts the model of the perception agent through their collaborative optimization so that the model can adapt to the dynamic environment; at the same time, the architecture introduces a generative model, and uses the prior knowledge of the generative model as auxiliary information to further promote the collaborative cooperation of multiple agents.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a multi-agent architecture based on online evolution, specifically comprising the following steps: S1. Perception Agent Perception Process Each sensor learns from the real world Get the original data , including images, sounds and other types of information. After obtaining the original information, each sensor uses a collaborative perception algorithm to fuse the original information obtained by each sensor, and then identifies and detects the information to obtain the recognition result. , the perception process can be expressed as: S2. Virtual world construction process Based on perception results , you can build a virtual world , in building a virtual world When the sensory information is integrated, various sensory information (such as the state information of the agent, the type, position, size and other data information of the recognized object) is integrated together, and then the real world The information of each agent and various objects is mapped to the virtual world and use the function Prediction Rewards ; S3. Decision-making and execution of control agents The control agent makes decisions in the virtual world. This process can be divided into two aspects: on the one hand, the control agent uses the corresponding algorithm to select actions, and on the other hand, the selected actions are simulated in the virtual world to determine their correctness. The control agent can use different algorithms to achieve its goals, such as rule-based algorithms, genetic algorithms, particle swarm optimization and other optimization-based decision algorithms, Dijkstra, A * Heuristic search algorithms, reinforcement learning algorithms, algorithms based on game theory, etc. Among them, the more typical algorithm is the multi-agent reinforcement learning algorithm; S4. Online evolutionary learning process Get actual rewards for each controlled agent After that, it will be compared with the corresponding predicted reward in the virtual world. If the difference between the two values ​​exceeds a certain threshold, , it indicates that there is a deviation between the prediction of the perceptual intelligent agent and the actual situation. In this case, the model of the perceptual intelligent agent needs to be optimized. The function for judging whether the perceptual intelligent agent model needs to be optimized is expressed as , which is expressed as follows: in, equal This means that the model needs to be optimized. Since the perceptual agent model is trained based on supervised learning, we need to add new data as supervision information. Specifically, we will re-label the original data and represent the labeling process as a function , which is expressed as follows: in, Represents the current relabeled data, Represents an existing tag, Indicates a new tag. According to the control agent Feedback modified the label, express This means that under the premise of optimizing the perception agent model, when the real reward is equal to one of the predicted rewards, it means that the original perception agent model has made an error in recognition and the existing wrongly recognized label needs to be replaced with an existing label; if the real reward is not in the predicted reward, it means that an object that has not appeared before is encountered and a new label needs to be assigned to the object. Therefore, the new data added to the perception agent model dataset can be expressed as: in, represents the perceptual agent model dataset, , , the perceptual agent can detect Different types of data, each type of data has its corresponding training data set ,For perceptual agents that detect the same type of data, their model parameters can be shared or trained separately.

[0007] Preferably, in said S1, It is Data of perceptual agents The feature extraction model of , In the process of fusion Data of perceptual agents The weight of , It is a model for identifying fusion features, and the model parameters are .

[0008] Preferably, in S2, the virtual world There are three main purposes: First, it can be used to simulate in advance the next action selected by the corresponding decision algorithm of the multi-agent to determine whether a bad reward will occur to avoid performing the action in the real world, and secondly, with the help of the virtual world In the generative model, we can transform the virtual world Taking the current state as input and providing a hint, these models can generate relevant content for reference, helping the control agent choose the right action, and thirdly, by comparing the actual reward in the real world. and prediction rewards in virtual worlds The target recognition model of the perceptual agent can be optimized to achieve online evolutionary learning, thereby ensuring accuracy and safety.

[0009] Preferably, in S3, if a multi-agent reinforcement learning algorithm is used for decision making, a Markov game is used in the algorithm. The Markov game can be represented by a quintuple, which is represented by ,in, Indicates shared An intelligent agent, represents the state space set of the environment, Representing an Agent The set of action spaces that can be selected, is the state transition function, Representing an Agent In Status The rewards received are: is the decay factor. In a multi-agent system, each agent maximizes its own reward as a learning goal. For the overall environment, there is no unified indicator to measure the pros and cons of the entire system. At this time, Nash equilibrium is used to represent the common reinforcement learning goal of multiple agents. The process of coordination between multiple agents can be described as the process of finding Nash equilibrium, that is: in, represents the set of all agent strategies, Indicates that all agents reach Nash equilibrium when the agent The cumulative reward is: In the above formula It refers to the state-action value function of multiple agents, and its calculation formula can be expressed as follows: Therefore, according to the above formula, Nash equilibrium can be expressed as: Preferably, under the Nash equilibrium condition, after the action is selected using the multi-agent strategy selection algorithm, the control agent will act in the virtual world If If a negative predicted reward is received in , it indicates that the action may not be optimal and needs to be reselected. In simple scenarios, this can be achieved by continuing to simulate the remaining actions in the current state and using -The greedy strategy selects the action with the highest predicted reward. In more complex scenarios, such as continuous action scenarios with large action spaces or dynamically changing environments, the pre-trained expert knowledge and task understanding of generative models can be used to assist in action selection. These models can generate auxiliary information consistent with human preferences based on the current state of the virtual world. By fusing this generated information with existing information in the virtual world, the range of action selection can be narrowed down and the optimal action can be finally determined. The specific formula can be expressed as: in, In the virtual world In the simulation, the process of executing the action is simulated. represents the action selected by the corresponding decision algorithm of the multi-agent, where Represents a specific algorithm, In the virtual world In the simulation, the process of executing the action is simulated. represents the action selected by the corresponding decision algorithm of the multi-agent, where Represents a specific algorithm, Represents the control agent The current status of represents the auxiliary information provided by the generative model to help action selection, while Is an indicator function, when providing prompt information When , indicating the information of using the generative model. If no prompt information is provided , the function value is , the generative model is not used. After the action is selected, the actual reward is obtained by performing the action in the real world and move to the next state.

[0010] Preferably, the online evolution is a learning method that optimizes the online model of each agent in a multi-agent environment based on the interaction and fusion of multi-agent perception, control and generation information, autonomously adapts to time-varying environmental characteristics based on intelligent environmental constraints, and achieves networked general artificial intelligence. The agents are divided into three categories, namely perception agents, control agents and generative models. Perception agents (in ) is mainly responsible for perception. Controlling the agent (in ) for decision making, generative models (in ) generate auxiliary information to assist decision-making, and these three types of agents can collaborate with each other to achieve coordinated optimization.

[0011] Preferably, the real world It is the environment in which each agent actually performs actions, in which the perception agent obtains raw data, and in which the control agent performs actions and obtains feedback. It can refer to scenes in various fields. The virtual world In order to provide a subjective mapping of the real world to each intelligent agent within its perception range, it is necessary to integrate various perceived information. For perceptual agents The acquired raw data includes images, locations, sounds and other information.

[0012] Preferably, the model parameters For perceptual agents The parameters used in the perception process are learned through the training process.

[0013] Preferably, the perception result It is the perception result obtained by the perception agents in collaboration, including the status of various agents, the types and location information of different objects.

[0014] Preferably, the actual reward To control the agent in the real world The reward obtained after executing the action in Represents a specific control agent , represents the state transition of the control agent, and the predicted reward By function Computed virtual world The rewards in Represents the controlling agent , Represents state transfer, Mainly used with actual rewards to determine whether the relevant information needs to be used for coordinated optimization. To control the agent Based on the strategies and virtual worlds it has learned The current best action is selected based on the information.

[0015] The present invention provides a multi-agent architecture based on online evolution, which has the following beneficial effects: 1. The present invention provides a multi-agent architecture based on online evolution. The multi-agent architecture can realize adaptive learning of multi-agents through mutual collaborative optimization among perception agents, control agents and generative models when facing unknown or changing environments. Traditional multi-agent architectures usually have perception agents recognize information, and then control agents make decisions and execute them based on the recognition results. The models used by perception agents for recognition are mostly trained using fixed data sets based on supervised learning. Therefore, when facing unknown or changing environments, recognition errors may occur, resulting in subsequent control agent decision errors and bad consequences. The framework proposed in the present invention has made improvements based on this point and has contributed to the development of networked general artificial intelligence.

[0016] 2. The present invention provides a multi-agent architecture based on online evolution, which is also superior to the end-to-end model in terms of scalability, interpretability and convergence efficiency. In terms of scalability, the architecture proposed by the present invention is a general architecture that can accommodate heterogeneous agents and various algorithms and is applicable to a variety of scenarios. In contrast, most end-to-end models are tailored for specific tasks, which limits their applicability in other fields. In terms of interpretability, the architecture proposed by the present invention can obtain perception results through perception agents, thereby explaining the actions taken by control agents, and helps to identify whether the problem occurs in the perception link or the decision-making link. In addition, the generative model can be combined to provide explanations for the corresponding actions. On the contrary, the end-to-end model is like a "black box", and it is difficult to clearly determine whether there is a problem with the perception or control component, and it is difficult to predict the model's response to unforeseen situations. In addition, the architecture proposed by the present invention separates the perception, control and generation modules, reduces the parameter requirements and algorithm complexity of each module, and thus improves the training efficiency. In contrast, the end-to-end model processes the entire task in a single model, resulting in increased complexity and computing requirements, thereby reducing the training efficiency to a certain extent.

[0017] 3. The present invention provides a multi-agent architecture based on online evolution. The multi-agent architecture has a wide range of application scenarios, such as autonomous driving, embodied intelligence and other fields, showing obvious advantages. In the field of autonomous driving, autonomous driving systems usually involve different types of sensors (such as radars, cameras, lidar) and control algorithms (such as path planning, dynamic obstacle avoidance), etc. The architecture proposed in the present invention can handle these heterogeneous agents and adapt to the ever-changing environment and task requirements. It can also make it easier to understand and explain the decision-making process of the autonomous driving system and provide more reliable safety guarantees. In the field of embodied intelligence, robot control systems also usually need to process a variety of different actions and sensor inputs. The architecture proposed in the present invention can support a variety of different types of robots and tasks, and realize online evolutionary learning when facing unknown and changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is the overall architecture diagram of the present invention; Figure 2 It is a diagram of the collaborative optimization process of the control agent and the perception agent of the present invention; Figure 3 A process diagram of using the pre-knowledge of a generative model to assist a control agent in making decisions according to the present invention; Figure 4 A schematic diagram of a multi-agent path planning learning algorithm of the present invention; Figure 5 A schematic diagram of the accuracy of the perception agent in identifying obstacles before and after the collaborative optimization of the present invention; Figure 6 The present invention compares the convergence iteration number results of the multi-agent path planning task under different algorithms, different numbers of control agents and different scenarios.

[0019] in, Figure 1 In the process, each sensor obtains raw data from the real world, including images, sounds and other types of information. After obtaining the raw information, each sensor uses a collaborative perception algorithm to fuse the raw information obtained by each sensor, and then identifies and detects the information to obtain the recognition result; then, based on the perception result, a virtual world is constructed, and the information of each agent and various objects from the real world is mapped to the virtual world, and the reward is predicted using a function; then, the control agent makes a decision in the virtual world, selects an action using the corresponding decision algorithm and simulates the action in the virtual world. If a negative predicted reward is obtained, the action needs to be reselected, and a -greedy strategy can be used or the pre-trained expert knowledge and task understanding of the generative model can be used to assist in action selection; after the action is selected, the action is executed in the real world to obtain the actual reward and transfer to the next state; then the collaborative optimization of perception control is performed, Figure 2 In the process, after each control agent obtains the actual reward, it is compared with the corresponding predicted reward in the virtual world. If the difference between the two values ​​exceeds a certain threshold, it indicates that there is a deviation between the prediction of the perception agent and the actual situation. At this time, the perception agent model needs to be optimized, and the current data is relabeled and added to the data set as new data. When the new data reaches a certain amount, the perception agent model is retrained to update the parameters. Figure 3 In the process, the information in the virtual world and the questions provided are taken as input, and the generative model will output corresponding answers, such as guiding the control agent to make action selections, explaining the current state, describing the current scene, etc., to provide auxiliary information for the subsequent decision-making of the control agent. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] Example 1 like Figure 1-6As shown, an embodiment of the present invention provides a multi-agent architecture based on online evolution, uses OpenAI Gym to build a multi-agent path planning simulation environment, introduces obstacles into the simulation environment, and divides the obstacles into two different types to simulate a dynamically changing environment: one is an obstacle that can be recognized by the perceptual agent, and the other is an unknown object that cannot be recognized by the model initially trained by the perceptual agent. This setting is mainly used to evaluate the effectiveness of subsequent collaborative optimization. In the multi-agent path planning task, the goal of each control agent is to use an algorithm to learn the path from the starting point to the end point. If the agent collides with an obstacle or the agents collide with each other, the round ends. When all agents have successfully reached their respective end points in the last 50 rounds, it means successful convergence. The specific algorithm is as follows Figure 4 The detailed steps are as follows: Step 1: Initialize the real world Wr. For the multi-agent path planning simulation environment, you need to use openaigym to build the corresponding environment and initialize various parameters; Step 2: Initialize the states of each control agent and set the control agent to a pre-specified initial position; Step 3: Initialize the model parameters of each perceptual agent, mark the obstacles, agents, and end points in the simulation environment of multi-agent pathfinding, do not mark unknown objects, and make a data set for training so that the trained perceptron cannot recognize unknown objects; Step 4: Start executing the program, set the maximum number of rounds to 6000, and the initial number of rounds to 0. If it is greater than 6000 or the program itself can converge, end the program, otherwise go to step 5, and set the convergence condition that each agent can successfully find the path to the end point in the most recent 50 consecutive rounds; Step 5: Obtain the complete original image information as the original data, that is, the simulation environment image of the multi-agent path planning; Step 6: Each perception agent model recognizes the original data to obtain the recognition result Pr, and uses the YOLOv5 model to recognize the original image data; Step 7: Create a virtual world Wv based on the recognition result Pr and calculate the predicted reward Rv. The recognition result includes information such as the type and location of each object. In the multi-agent path-finding environment, a 10*10 array is created corresponding to the real world and the object type information is marked at the corresponding position of the array according to the recognition result to obtain the virtual world and predict the reward value, that is, the prediction of the reward value that each control agent may obtain after performing different actions in different states; Step 8: Each control agent simulates the action training strategy in the virtual world Wv to obtain the optimal action a*. Each control agent simulates the action at the next moment in the constructed virtual world and learns the strategy. The IQL and IDQN algorithms are used for strategy learning. The IQL learning rate is set to 0.2, the discount factor is set to 0.99, and ε is set to 0.4; the IDQN learning rate is set to 0.0005, the discount factor is set to 0.94, and ε is set to 0.4. The buffer pool size is set to 1000000, the batch size is set to 256, and the random number of the two algorithms is set to 42; Step 9: Each control agent performs actions in the real world to obtain a real reward Ro. The reward for reaching the end point is set to +1, the reward for hitting an obstacle is set to -1, and the reward for control agents colliding with each other is set to -0.1. The overall reward is the average reward of each control agent. Step 10: Determine whether all agents can reach the end point after the current moment is completed. If so, the current round ends and returns to step 4 to start a new round. If not, continue to step 11; Step 11: Calculate the difference between the actual reward and the predicted reward of the control agent. If an agent fails to reach the end point, it means that the perception agent may have made a recognition error, which led to an error in the decision of the control agent. At this time, compare the difference between the predicted reward value and the actual reward value of the corresponding control agent; Step 12: Determine whether the gap exceeds the threshold. The threshold should be set in a reasonable range according to the reward value. It is set to 0.5 in the path planning task. If it exceeds the threshold, it is considered that the perception agent has a recognition error and the process proceeds to step 13. Otherwise, the process proceeds to step 16. Step 13: Modify the labels of the corresponding unknown objects or objects with incorrect recognition, and add them to the perception agent model data set as new data. That is, the unknown object information fed back by the current control agent is marked as a new label on the image recognized by the original perception agent model, and used as new data to retrain the perception agent model and update the model parameters; Step 14: Determine whether the data added to the data set exceeds a certain number. The number setting can be changed according to the actual model. The value set here is 50. If it exceeds, then transfer to step 15, otherwise transfer to step 16; Step 15: Retrain the perceptron model and update the perceptron model parameters; Step 16: Determine whether each control agent has compared the actual reward with the predicted reward. If so, the current round ends and transfers to step 4. Otherwise, continue to judge the control agent and transfer to step 11.

[0022] The simulation was performed on a server with a 16-core Intel(R) Xeon(R) CPUE5-2682v4 processor and a 16GB P100 GPU. The specific results are as follows.

[0023] The accuracy of the perception agent in identifying obstacles before and after a collaborative optimization was compared. Figure 5 As shown, Figure 5 The change in accuracy in the case of 5 and 7 unknown objects is shown. From the curves, it can be seen that in all cases, the accuracy of the perception agent for obstacle recognition has improved after the collaborative optimization. In addition, in both scenarios, the accuracy using 3 control agents is higher than that using 2 control agents. This is because as the number of control agents increases, the probability of encountering unknown objects also increases, resulting in a larger number of unknown objects being marked. Therefore, after retraining, the number of unknown objects recognized also increases accordingly.

[0024] The number of convergence iterations of the multi-agent path planning task with and without collaborative optimization under different algorithms (IQL and IDQN), different numbers of control agents, and different scenarios was compared to evaluate whether the training efficiency has been improved. The comparison results are shown in Table 1. From the results in Table 1, it can be seen that, in any case, the number of convergence iterations of the path planning task is significantly lower when collaborative optimization is used than when it is not used, which shows that the use of optimization can improve the task efficiency; as the number of unknown objects in the simulation environment increases, the number of iterations required for convergence will increase regardless of whether collaborative optimization is used. This is mainly because as the number of unknown objects increases, the probability of the control agent colliding with these unknown objects also increases. The collision with the unknown object marks the end of the current iteration, thereby increasing the number of iterations required for convergence; in addition, the number of control agents also affects the number of convergence iterations. In scenarios without collaborative optimization, the number of iterations required for convergence usually increases as the number of control agents increases. However, in scenarios with collaborative optimization, the number of iterations required for convergence decreases as the number of control agents increases. The main reason for this performance improvement is that multiple control agents are more likely to encounter different unknown objects, which enables the system to accumulate data on newly added unknown objects more quickly and then use them to optimize the perception agent model; when different algorithms (such as IQL and IDQN) are used, the impact of the two algorithms on the number of convergence iterations is not significant, which reflects the versatility of the proposed architecture.

[0025] Example 2 The multi-agent architecture based on online evolution proposed in the present invention has a wide range of application scenarios, and the architecture can be applied to the field of autonomous driving.

[0026] In the field of autonomous driving, consider a local road segment that contains elements such as manned vehicles, autonomous vehicles (AVs), roadside units, cameras, radars, etc. Typically, autonomous vehicles are equipped with sensors, controllers, and actuators, while roadside units and cameras also contain sensors. Although collaborative perception technology has made significant progress in the field of autonomous driving, extreme weather conditions may still cause recognition errors, which in turn affect the decision-making of autonomous vehicles. In this case, a sensing-control collaborative framework can be used to enable the sensors of autonomous vehicles, as well as the sensors of roadside units and cameras to learn adaptively. It can be divided into the following steps: A1. Perception of the Perceptron For autonomous vehicles, collaborative perception can be achieved by combining the vehicle's own perception information with data obtained from other vehicles, roadside units, cameras, and other traffic elements through communication to obtain perception results.

[0027] A2. Construction of Virtual World Based on the perception results, a virtual world that includes factors such as the vehicle’s current state, road conditions, weather, lighting, etc. can be constructed, and the predicted reward can be calculated.

[0028] A3. Controller Decision and Execution The controller of the autonomous vehicle is simulated in the virtual world, utilizing the generative model if necessary, and making decisions based on these factors to perform autonomous maneuvers such as following, overtaking, and stopping.

[0029] A4. Online Evolutionary Learning Process After performing these actions, feedback from the physical environment (such as whether the overtaking was successful or the following distance) can be used to optimize the vehicle's sensors, as well as the sensors of cameras, roadside units and other traffic elements. The model is retrained regularly to achieve online evolutionary learning. In this way, autonomous driving can adapt to the changing environment and ultimately achieve the effect of artificial general intelligence.

[0030] Example 3 The multi-agent architecture based on online evolution proposed in the present invention has a wide range of application scenarios, and the architecture can be applied to the field of embodied intelligence.

[0031] In the field of embodied intelligence, consider a multi-robot operation scenario where multiple robots need to collaborate to move an object to a specified location while avoiding potential obstacles. Each robot is equipped with perception, control, and execution components, and makes decisions based on its perception results. However, in unfamiliar environments, conditions such as lighting and space may change, leading to perception errors. In this case, the perception-control collaborative framework can be applied to achieve online evolution of the perceptron. Specifically: B1. Perception of the Perceptron The sensors of each robot collect raw data from the environment for collaborative perception to obtain perception results.

[0032] B2. Construction of Virtual World Based on the perception results, a virtual world is constructed, which contains information such as the position of each robot, the position of the objects that need to be moved, and the position of obstacles, and the predicted reward is calculated.

[0033] B3. Controller Decision and Execution The robot's controller uses information from the virtual world to select a cooperative strategy and start moving the object.

[0034] B4. Online Evolutionary Learning Process During the operation, each robot obtains feedback from the environment, such as whether the object is successfully lifted, whether it hits an obstacle, whether the target position is reached, etc. These feedbacks are compared with the predicted rewards in the virtual world, and then the robot's sensor models are optimized with the information obtained, and these models are retrained regularly. This improves the correctness of perception and enables the robot to adapt to the changing environment.

[0035] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-agent architecture based on online evolution, characterized in that: The specific steps include: S1. Perception Agent Perception Process Each sensor learns from the real world Get the original data , including images, sounds and other types of information. After obtaining the original information, each sensor uses a collaborative perception algorithm to fuse the original information obtained by each sensor, and then identifies and detects the information to obtain the recognition result. , the perception process can be expressed as: ； S2. Virtual world construction process Based on perception results , you can build a virtual world , in building a virtual world When the sensory information is integrated, various sensory information (such as the state information of the agent, the type, position, size and other data information of the recognized object) is integrated together, and then the real world The information of each agent and various objects is mapped to the virtual world and use the function Prediction Rewards ; S3. Decision-making and execution of control agents The control agent makes decisions in the virtual world. This process can be divided into two aspects: on the one hand, the control agent uses the corresponding algorithm to select actions, and on the other hand, the selected actions are simulated in the virtual world to determine their correctness. The control agent can use different algorithms to achieve its goals, such as rule-based algorithms, genetic algorithms, particle swarm optimization and other optimization-based decision algorithms, Dijkstra, A * Heuristic search algorithms, reinforcement learning algorithms, algorithms based on game theory, etc. Among them, the more typical algorithm is the multi-agent reinforcement learning algorithm; S4. Online evolutionary learning process Get actual rewards for each controlled agent After that, it will be compared with the corresponding predicted reward in the virtual world. If the difference between the two values ​​exceeds a certain threshold, , it indicates that there is a deviation between the prediction of the perceptual intelligent agent and the actual situation. In this case, the model of the perceptual intelligent agent needs to be optimized. The function for judging whether the perceptual intelligent agent model needs to be optimized is expressed as , which is expressed as follows: ； in, equal This means that the model needs to be optimized. Since the perceptual agent model is trained based on supervised learning, we need to add new data as supervision information. Specifically, we will re-label the original data and represent the labeling process as a function , which is expressed as follows: ； in, Represents the current relabeled data, Represents an existing tag, Indicates a new tag. According to the control agent Feedback modified the label, express This means that under the premise of optimizing the perception agent model, when the real reward is equal to one of the predicted rewards, it means that the original perception agent model has made an error in recognition and the existing wrongly recognized label needs to be replaced with an existing label; if the real reward is not in the predicted reward, it means that an object that has not appeared before is encountered and a new label needs to be assigned to the object. Therefore, the new data added to the perception agent model dataset can be expressed as: ； in, represents the perceptual agent model dataset, , , the perceptual agent can detect Different types of data, each type of data has its corresponding training data set ,For perceptual agents that detect the same type of data, their model parameters can be shared or trained separately.

2. The multi-agent architecture based on online evolution according to claim 1, characterized in that: In S1, It is Data of perceptual agents The feature extraction model of , In the process of fusion Data of perceptual agents The weight of , It is a model for identifying fusion features, and the model parameters are .

3. The multi-agent architecture based on online evolution according to claim 1, characterized in that: In S2, the virtual world There are three main purposes: First, it can be used to simulate in advance the next action selected by the corresponding decision algorithm of the multi-agent to determine whether a bad reward will occur to avoid performing the action in the real world, and secondly, with the help of the virtual world In the generative model, we can transform the virtual world Taking the current state as input and providing a hint, these models can generate relevant content for reference, helping the control agent choose the right action, and thirdly, by comparing the actual reward in the real world. and prediction rewards in virtual worlds The target recognition model of the perceptual agent can be optimized to achieve online evolutionary learning, thereby ensuring accuracy and safety.

4. The multi-agent architecture based on online evolution according to claim 1, characterized in that: In S3, if a multi-agent reinforcement learning algorithm is used for decision making, a Markov game is used in the algorithm. The Markov game can be represented by a five-tuple, which is recorded as ,in, Indicates shared An intelligent agent, represents the state space set of the environment, Representing an Agent The set of action spaces that can be selected, is the state transition function, Representing an Agent In Status The rewards received are: is the decay factor. In a multi-agent system, each agent maximizes its own reward as a learning goal. For the overall environment, there is no unified indicator to measure the pros and cons of the entire system. At this time, Nash equilibrium is used to represent the common reinforcement learning goal of multiple agents. The process of coordination between multiple agents can be described as the process of finding Nash equilibrium, that is: ； in, represents the set of all agent strategies, Indicates that all agents reach Nash equilibrium when the agent The cumulative reward is: ； In the above formula It refers to the state-action value function of multiple agents, and its calculation formula can be expressed as follows: ； Therefore, according to the above formula, Nash equilibrium can be expressed as: 。 5. The multi-agent architecture based on online evolution according to claim 4, characterized in that: Under the Nash equilibrium condition, after the action is selected using the multi-agent strategy selection algorithm, the control agent will act in the virtual world. If If a negative predicted reward is received in , it indicates that the action may not be optimal and needs to be reselected. In simple scenarios, this can be achieved by continuing to simulate the remaining actions in the current state and using -The greedy strategy selects the action with the highest predicted reward. In more complex scenarios, such as continuous action scenarios with large action spaces or dynamically changing environments, the pre-trained expert knowledge and task understanding of generative models can be used to assist in action selection. These models can generate auxiliary information consistent with human preferences based on the current state of the virtual world. By fusing this generated information with existing information in the virtual world, the range of action selection can be narrowed down and the optimal action can be finally determined. The specific formula can be expressed as: ； in, In the virtual world In the simulation, the process of executing the action is simulated. represents the action selected by the corresponding decision algorithm of the multi-agent, where Represents a specific algorithm, Represents the control agent The current status of represents the auxiliary information provided by the generative model to help action selection, while Is an indicator function, when providing prompt information When , indicating the information of using the generative model. If no prompt information is provided , the function value is , the generative model is not used. After the action is selected, the actual reward is obtained by performing the action in the real world and move to the next state.

6. The multi-agent architecture based on online evolution according to claim 1, characterized in that: The online evolution is a learning method that optimizes the online model of each agent in a multi-agent environment based on the interaction and fusion of multi-agent perception, control and generation information. It can autonomously adapt to the time-varying environment characteristics based on the constraints of the intelligent environment and achieve networked general artificial intelligence. The agents are divided into three categories, namely perception agents, control agents and generative models. Perception agents (in ) is mainly responsible for perception. Controlling the agent (in ) for decision making, generative models (in ) generate auxiliary information to assist decision-making, and these three types of agents can collaborate with each other to achieve coordinated optimization.

7. The multi-agent architecture based on online evolution according to claim 1, characterized in that: The real world It is the environment in which each agent actually performs actions, in which the perception agent obtains raw data, and in which the control agent performs actions and obtains feedback. It can refer to scenes in various fields. The virtual world In order to provide a subjective mapping of the real world to each intelligent agent within its perception range, it is necessary to integrate various perceived information. For perceptual agents The acquired raw data includes images, locations, sounds and other information.

8. The multi-agent architecture based on online evolution according to claim 2, characterized in that: The model parameters For perceptual agents The parameters used in the perception process are learned through the training process.

9. The multi-agent architecture based on online evolution according to claim 1, characterized in that: The perceived results It is the perception result obtained by the perception agents in collaboration, including the status of various agents, the types and location information of different objects.

10. The multi-agent architecture based on online evolution according to claim 1, characterized in that: The actual reward To control the agent in the real world The reward obtained after executing the action in Represents a specific control agent , represents the state transition of the control agent, and the predicted reward By function Computed virtual world The rewards in Represents the controlling agent , Represents state transfer, Mainly used with actual rewards to determine whether the relevant information needs to be used for coordinated optimization. To control the agent Based on the strategies and virtual worlds it has learned The current best action is selected based on the information.

Images