A multi-agent architecture based on online evolution

Abstract

The present invention provides a multi-agent architecture based on online evolution, which relates to the technical field of multi-agent systems. The multi-agent architecture based on online evolution compares the actual rewards obtained by the control agent after performing actions in the real world after making decisions with the virtual rewards predicted in the virtual world, and decides whether the perception agent needs to be updated based on the comparison results. The expert knowledge of the generative model in the virtual world is used to assist the decision-making of the control agent. By fusing this generated information with the existing information in the virtual world to narrow the decision-making scope, the coordination of perception, control and generation is achieved. The architecture adds the feedback information obtained after the control agent performs actions in the physical environment as new data to the corresponding data set of the perception agent, and adjusts the model of the perception agent through their collaborative optimization, so that the agent can adapt to the ever-changing environmental conditions while also ensuring scalability, interpretability and convergence efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent systems, and in particular to a multi-agent architecture based on online evolution. Background Art

[0002] Multi-agent systems consist of a group of autonomous entities, typically used to solve complex problems requiring collective intelligence and collaboration. Multi-agent systems have a wide range of applications, with common areas including cloud computing, autonomous driving, and smart warehousing. Collaborative perception and control technologies play a crucial role in multi-agent systems. Multi-agent collaborative perception focuses on how multiple agents can collaborate to perceive, interpret, and understand their environment. With the rapid development of artificial intelligence and deep learning, supervised learning has become widely used in most perception tasks. Among the various supervised learning network architectures, convolutional neural network-based architectures are the most successful and widely used. For single-agent perception tasks, object detection models such as RCNN and YOLO can achieve rapid object detection and localization. Multi-agent collaborative perception requires front-end, mid-stage, or post-stage fusion of the perception data from multiple agents. Typical algorithms include Cooper, V2VNet, and Car2X. Multi-agent collaborative control focuses on how multiple autonomous agents coordinate their actions to achieve a common goal. This approach leverages the collective cooperation of agents, enabling them to complete tasks more efficiently than if each agent acted alone. Currently, there are many algorithms that can achieve multi-agent collaborative control, among which multi-agent reinforcement learning is one of the most mainstream algorithms. Multi-agent reinforcement learning can be divided into four categories: behavior analysis, communication learning, collaborative learning, and imitation learning. Typical algorithms include IQL, IDQN, VDN, QMIX, and MADDPG.

[0003] Typically, the workflow of a traditional multi-agent system is that the perceptron first collects and interprets data from the environment, and then the controller makes decisions based on these results. In other words, existing methods mostly study multi-agent perception and multi-agent control algorithms separately. This model has been widely used in many fields, but this traditional workflow may encounter problems in dynamically changing environments. Because most perception algorithms rely on supervised learning and most perception models are trained from fixed datasets, when the environment changes, the perceptron model may make recognition errors because the dataset used for training does not contain the new and changing environmental data. This may cause the controller to make incorrect decisions and have adverse consequences during actual execution.

[0004] The development of end-to-end models can partially solve the above problems. These models take raw sensor data as input and directly output control actions. By minimizing the loss through backpropagation, they can jointly and globally optimize object perception and adapt to changing environments. Although end-to-end models are effective, they also face some challenges. First, end-to-end models are usually configured for specific tasks and may lack versatility and scalability. In addition, they lack interpretability and are often called "black boxes." Interpretability can not only provide performance and safety guarantees, but also improve public acceptance. However, current end-to-end models are still insufficient in this regard. In addition, end-to-end models require a large amount of data for training, which may consume a lot of computing resources and storage space, resulting in low convergence efficiency.

[0005] Based on the above aspects, there is currently a lack of multi-agent architecture based on online evolution that can be used in non-stationary environments while ensuring scalability, interpretability and convergence efficiency. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this paper proposes a multi-agent architecture based on online evolution, enabling agents to adapt to changing environmental conditions while maintaining scalability, interpretability, and convergence efficiency. This architecture incorporates feedback from the control agent after executing actions in the physical environment as new data into the corresponding dataset of the perception agent. Through collaborative optimization, the perception agent's model is adjusted to adapt to dynamic environments. Furthermore, the architecture incorporates generative models, leveraging their prior knowledge as auxiliary information to further promote multi-agent collaboration.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a multi-agent architecture based on online evolution, specifically comprising the following steps:

[0008] S1. Perception Agent Perception Process

[0009] Each sensor learns from the real world Get the original data , including images, sounds and other types of information. After obtaining the original information, each sensor uses a collaborative perception algorithm to fuse the original information obtained by each sensor, and then performs recognition detection on the information to obtain the recognition result. , the perception process can be expressed as:

[0010]

[0011] S2. Virtual world construction process

[0012] Based on perception results , you can build a virtual world , in building a virtual world When the various perception information (such as the state information of the intelligent body, the type, position, size and other data information of the recognized object) are integrated together, and then the real world The information of each agent and various objects is mapped to the virtual world and use the function Prediction Rewards ;

[0013] S3. Decision-making and execution of control agents

[0014] The control agent makes decisions in the virtual world. This process can be divided into two aspects: on the one hand, the control agent uses the corresponding algorithm to select actions, and on the other hand, the selected actions are simulated in the virtual world to determine their correctness. The control agent can use different algorithms to achieve its goals, such as rule-based algorithms, genetic algorithms, particle swarm optimization and other optimization-based decision algorithms, Dijkstra, A * Heuristic search algorithms, reinforcement learning algorithms, game theory-based algorithms, etc. Among them, the more typical algorithm is the multi-agent reinforcement learning algorithm;

[0015] S4. Online evolutionary learning process

[0016] Get actual rewards for each controlled agent After that, it will be compared with the corresponding predicted reward in the virtual world Compare and if the difference between the two values ​​exceeds a certain threshold , it indicates that there is a deviation between the prediction of the perceptual agent and the actual situation. In this case, the model of the perceptual agent needs to be optimized. The function for judging whether the perceptual agent model needs to be optimized is expressed as , which is expressed as follows:

[0017]

[0018] in, equal This means that the model needs to be optimized. Since the perceptual agent model is trained based on supervised learning, we need to add new data as supervision information. Specifically, we will re-label the original data and express the labeling process as a function , which is expressed as follows:

[0019]

[0020] in, Represents the currently relabeled data, Represents an existing tag, Indicates a new tag. According to the control agent Feedback modified the label, express This means that under the premise of optimizing the perception agent model, when the real reward is equal to one of the predicted rewards, it means that the original perception agent model has made an error in recognition and the existing incorrectly recognized label needs to be replaced with an existing label. If the real reward is not in the predicted reward, it means that an object that has not appeared before is encountered and a new label needs to be assigned to the object. Therefore, the new data added to the perception agent model dataset can be expressed as:

[0021]

[0022] in, represents the perceptual agent model dataset, , , the perceptual agent can detect Different types of data, each type of data has its own corresponding training data set ,For perception agents that detect the same type of data, their model parameters can be shared or trained separately.

[0023] Preferably, in said S1, It is Data of a perceptual agent The feature extraction model of , In the fusion process Data of a perceptual agent The weight of , It is a model for identifying fusion features, and the model parameters are .

[0024] Preferably, in said S2, the virtual world There are three main purposes: First, it can be used to pre-simulate the next action selected by the corresponding decision algorithm of the multi-agent To determine whether there will be a bad reward to avoid performing the action in the real world, and secondly, with the help of the virtual world In the generative model, we can transform the virtual world Taking the current state of the agent as input and providing a hint, these models can generate relevant content for reference to help the control agent choose the right action. Third, by comparing the actual rewards in the real world and prediction rewards in virtual worlds The target recognition model of the perceptual agent can be optimized to achieve online evolutionary learning, thereby ensuring accuracy and safety.

[0025] Preferably, in S3, if a multi-agent reinforcement learning algorithm is used for decision making, a Markov game is used in the algorithm. The Markov game can be represented by a quintuple, which is represented by ,in, Indicates shared an intelligent agent, represents the state space set of the environment, Representing an agent The set of action spaces that can be selected, is the state transition function, Representing an agent In state The rewards received are: is the decay factor. In a multi-agent system, each agent maximizes its own reward as its learning goal. For the overall environment, there is no unified metric that can measure the performance of the entire system. In this case, Nash equilibrium is used to represent the common reinforcement learning goal of multiple agents. The process of coordinating multiple agents can be described as the process of finding a Nash equilibrium, that is:

[0026]

[0027] in, represents the set of all agent strategies, Indicates that all agents reach Nash equilibrium when the agent The cumulative reward is:

[0028]

[0029] In the above formula It refers to the state-action value function of multiple agents, and its calculation formula can be expressed as follows:

[0030]

[0031] Therefore, according to the above formula, Nash equilibrium can be expressed as:

[0032]

[0033] Preferably, under the Nash equilibrium condition, after the action is selected using the multi-agent strategy selection algorithm, the control agent will act in the virtual world If If a negative predicted reward is received in , it indicates that the action may not be optimal and needs to be reselected. In simple scenarios, this can be achieved by continuing to simulate the remaining actions in the current state and using The greedy strategy selects the action with the highest predicted reward. In more complex scenarios, such as continuous action scenarios with a large action space or dynamically changing environments, the pre-trained expert knowledge and task understanding of generative models can be used to assist in action selection. These models can generate auxiliary information consistent with human preferences based on the current state of the virtual world. By fusing this generated information with existing information in the virtual world, the range of action options can be narrowed down and the optimal action can be ultimately determined. The specific formula can be expressed as:

[0034]

[0035] in, In the virtual world In the simulation, the process of executing the action is simulated. represents the action selected by the corresponding decision algorithm of the multi-agent, where Represents a specific algorithm, In the virtual world In the simulation, the process of executing the action is simulated. represents the action selected by the corresponding decision algorithm of the multi-agent, where Represents a specific algorithm, Represents the control agent The current state, represents the auxiliary information provided by the generative model to help action selection, and Is an indicator function, when providing prompt information When , indicating the use of generative model information, if no prompt information is provided , the function value is , the generative model is not used. After the action is selected, the actual reward is obtained by performing the action in the real world. and move to the next state.

[0036] Preferably, the online evolution is a learning method that optimizes the online model of each agent in a multi-agent environment based on the interaction and fusion of multi-agent perception, control and generation information, autonomously adapts to the time-varying environment characteristics based on the constraints of the intelligent environment, and achieves networked general artificial intelligence. The agents are divided into three categories, namely perception agents, control agents and generative models. Perception agents (in ) is mainly responsible for perception. Control agent (in ) for decision making, generative models (in ) generate auxiliary information to assist decision-making, and these three types of agents can collaborate with each other to achieve coordinated optimization.

[0037] Preferably, the real world The virtual world is the environment in which each agent actually performs actions, in which the perception agent obtains raw data and the control agent performs actions and obtains feedback. It can refer to scenes in various fields. In order to provide a subjective mapping of the real world to each agent within its perception range, it is necessary to integrate various perceived information. For perceptual agents The acquired raw data includes images, locations, sounds, and other information.

[0038] Preferably, the model parameters For perceptual agents The parameters used in the perception process are learned through the training process.

[0039] Preferably, the perception result It is the perception result obtained by the collaborative perception of intelligent agents, including the status of various intelligent agents, the type and location information of different objects.

[0040] Preferably, the actual reward To control the agent in the real world The reward obtained after executing the action in Represents a specific control agent , Represents the state transition of the control agent, the predicted reward By function Computed virtual world The rewards in Represents the control agent , Represents state transfer, Mainly used for actual rewards Compare to determine whether relevant information needs to be used for coordinated optimization, the optimal action To control the agent Based on the strategies and virtual worlds it has learned The current best action is selected based on the information.

[0041] The present invention provides a multi-agent architecture based on online evolution, which has the following beneficial effects:

[0042] 1. The present invention provides a multi-agent architecture based on online evolution. When facing an unknown or changing environment, the multi-agent architecture can realize adaptive learning of the multi-agent through the mutual collaborative optimization among the perception agent, the control agent, and the generative model. The traditional multi-agent architecture usually has the perception agent recognize information, and then the control agent makes decisions based on the recognition results. The models used by the perception agent for recognition are mostly trained using fixed data sets based on supervised learning. Therefore, when facing an unknown or changing environment, recognition errors may occur, which may lead to subsequent control agent decision-making errors and adverse consequences. The framework proposed in the present invention has made improvements based on this point and has contributed to the development of networked general artificial intelligence.

[0043] 2. The present invention provides a multi-agent architecture based on online evolution, which also outperforms end-to-end models in terms of scalability, interpretability, and convergence efficiency. In terms of scalability, the proposed architecture is a general architecture that can accommodate heterogeneous agents and various algorithms and is applicable to a variety of scenarios. In contrast, most end-to-end models are tailored for specific tasks, limiting their applicability in other fields. In terms of interpretability, the proposed architecture can obtain perception results through the perception agent, thereby explaining the actions taken by the control agent and helping to identify whether the problem occurs in the perception link or the decision link. In addition, it can also be combined with a generative model to provide explanations for the corresponding actions. In contrast, the end-to-end model is like a "black box", making it difficult to clearly determine whether the problem lies in the perception or control component, and it is difficult to predict the model's response to unforeseen situations. In addition, the proposed architecture separates the perception, control, and generation modules, reducing the parameter requirements and algorithm complexity of each module, thereby improving training efficiency. In contrast, the end-to-end model processes the entire task in a single model, resulting in increased complexity and computational requirements, which to some extent reduces training efficiency.

[0044] 3. The present invention provides a multi-agent architecture based on online evolution. This multi-agent architecture has a wide range of application scenarios, such as autonomous driving, embodied intelligence and other fields, and has demonstrated obvious advantages. In the field of autonomous driving, autonomous driving systems usually involve different types of sensors (such as radars, cameras, lidar) and control algorithms (such as path planning, dynamic obstacle avoidance), etc. The architecture proposed in the present invention can handle these heterogeneous agents and adapt to the ever-changing environment and task requirements. It can also make it easier to understand and explain the decision-making process of the autonomous driving system and provide more reliable safety guarantees; in the field of embodied intelligence, robot control systems also usually need to process a variety of different actions and sensor inputs. The architecture proposed in the present invention can support a variety of different types of robots and tasks, and realize online evolutionary learning when facing unknown and changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is the overall architecture diagram of the present invention;

[0046] Figure 2 This is a diagram of the collaborative optimization process of the control agent and the perception agent of the present invention;

[0047] Figure 3 A diagram of the process of using the prior knowledge of a generative model to assist a control agent in making decisions according to the present invention;

[0048] Figure 4 Schematic diagram of the multi-agent path planning learning algorithm of the present invention;

[0049] Figure 5 This is a schematic diagram of the accuracy of the perception agent in identifying obstacles before and after the collaborative optimization of the present invention;

[0050] Figure 6 This is a graph comparing the convergence iteration times of multi-agent path planning tasks under different algorithms, different numbers of control agents, and different scenarios.

[0051] in, Figure 1 In the process, each sensor obtains raw data from the real world, including images, sounds and other types of information. After obtaining the raw information, each sensor uses a collaborative perception algorithm to fuse the raw information obtained by each sensor, and performs recognition detection on the information to obtain the recognition result; then, based on the perception result, a virtual world is constructed, and the information of each agent and various objects from the real world is mapped to the virtual world, and the reward is predicted using a function; then, the control agent makes a decision in the virtual world, selects an action using the corresponding decision algorithm and simulates the action in the virtual world. If a negative predicted reward is obtained, the action needs to be reselected, and a -greedy strategy can be used or the pre-trained expert knowledge and task understanding of the generative model can be used to assist in action selection; after the action is selected, the action is executed in the real world to obtain the actual reward and transfer to the next state; then the collaborative optimization of perception and control is performed, Figure 2 In the process, after each control agent obtains the actual reward, it is compared with the corresponding predicted reward in the virtual world. If the difference between the two values ​​exceeds a certain threshold, it indicates that there is a deviation between the prediction of the perception agent and the actual situation. At this time, the perception agent model needs to be optimized, and the current data is relabeled and added to the dataset as new data. When the new data reaches a certain amount, the perception agent model is retrained to update the parameters. Figure 3In the process, the information in the virtual world and the questions provided are taken as input, and the generative model will output corresponding answers, such as guiding the control agent to make action selections, explaining the current state, describing the current scene, etc., providing auxiliary information for the subsequent decision-making of the control agent. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] Example 1

[0054] like Figure 1-6 As shown, an embodiment of the present invention provides a multi-agent architecture based on online evolution, uses OpenAI Gym to build a multi-agent path planning simulation environment, introduces obstacles into the simulation environment, and divides the obstacles into two different types to simulate a dynamically changing environment: one is an obstacle that can be recognized by the perception agent, and the other is an unknown object that cannot be recognized by the model initially trained by the perception agent. This setting is mainly used to evaluate the effectiveness of subsequent collaborative optimization. In the multi-agent path planning task, the goal of each control agent is to use an algorithm to learn the path from the starting point to the end point. If the agent collides with an obstacle or the agents collide with each other, the round ends. When all agents successfully reach their respective end points in the last 50 rounds, it means successful convergence. The specific algorithm is as follows Figure 4 The detailed steps are as follows:

[0055] Step 1: Initialize the real world Wr. For the multi-agent path planning simulation environment, you need to use openaigym to build the corresponding environment and initialize various parameters.

[0056] Step 2: Initialize the state of each control agent and set the control agent to a pre-specified initial position;

[0057] Step 3: Initialize the model parameters of each perceptron agent. In a multi-agent pathfinding simulation environment, mark the obstacles, agents, and endpoints in the simulation environment. Do not mark unknown objects. Create a dataset for training so that the trained perceptron cannot recognize unknown objects.

[0058] Step 4: Start the program, set the maximum number of rounds to 6000, and the initial number of rounds to 0. If it is greater than 6000 or the program itself has converged, end the program. Otherwise, go to step 5. Set the convergence condition as that every agent can successfully find a path to the end point in the last 50 consecutive rounds.

[0059] Step 5: Obtain the complete original image information as the original data, that is, the simulation environment image of the multi-agent path planning;

[0060] Step 6: Each perception agent model recognizes the original data to obtain the recognition result Pr, and uses the YOLOv5 model to recognize the original image data;

[0061] Step 7: Create a virtual world Wv based on the recognition result Pr and calculate the predicted reward Rv. The recognition result includes information such as the type and location of each object. In the multi-agent pathfinding environment, a 10*10 array corresponding to the real world is created and the object type information is marked at the corresponding position in the array based on the recognition result. This creates a virtual world and predicts the reward value. In other words, it predicts the reward value that each controlled agent may receive after performing different actions in different states.

[0062] Step 8: Each control agent simulates the action training strategy in the virtual world Wv to obtain the optimal action a*. Each control agent simulates the action at the next moment in the constructed virtual world and learns the strategy. The IQL and IDQN algorithms are used for strategy learning. The IQL learning rate is set to 0.2, the discount factor is set to 0.99, and ε is set to 0.4; the IDQN learning rate is set to 0.0005, the discount factor is set to 0.94, and ε is set to 0.4. The buffer pool size is set to 1000000, the batch size is set to 256, and the random number of both algorithms is set to 42.

[0063] Step 9: Each controlled agent performs an action in the real world and obtains a real reward Ro. The reward for reaching the end point is set to +1, the reward for hitting an obstacle is set to -1, and the reward for controlling agents to collide with each other is set to -0.1. The overall reward is the average of the rewards of each controlled agent.

[0064] Step 10: Determine whether all agents can reach the end point after the current moment is completed. If so, the current round ends and returns to step 4 to start a new round. If not, continue to step 11;

[0065] Step 11: Calculate the difference between the actual reward of the control agent and the predicted reward. If an agent fails to reach the end point, it may indicate that the perception agent has made a recognition error, which has led to an incorrect decision of the control agent. In this case, compare the difference between the predicted reward value and the actual reward value of the corresponding control agent;

[0066] Step 12: Determine whether the gap exceeds the threshold. The threshold should be set in a reasonable range according to the reward value. In the path planning task, it is set to 0.5. If it exceeds the threshold, it is considered that the perception agent has made a recognition error and the process proceeds to step 13. Otherwise, the process proceeds to step 16.

[0067] Step 13: Modify the labels of the corresponding unknown objects or objects with incorrect recognition and add them as new data to the perception agent model dataset. In other words, the unknown object information fed back by the current control agent is added as a new label to the image originally recognized by the perception agent model. This new data is used to retrain the perception agent model and update the model parameters.

[0068] Step 14: Determine whether the data added to the dataset exceeds a certain number. The number setting can be changed according to the actual model. The value set here is 50. If it exceeds, then transfer to step 15, otherwise transfer to step 16;

[0069] Step 15: Retrain the perceptron model and update the perceptron model parameters;

[0070] Step 16: Determine whether each control agent has compared the actual reward with the predicted reward. If so, the current round ends and transfers to step 4. If not, continue to judge the control agent and transfer to step 11.

[0071] The simulation is performed on a server with a 16-core Intel(R) Xeon(R) CPU E5-2682v4 processor and a 16GB P100 GPU. The specific results are as follows.

[0072] The accuracy of the perception agent in identifying obstacles was compared before and after a collaborative optimization. Figure 5 As shown, Figure 5The following plots show the change in accuracy for scenarios with five and seven unknown objects. The curves show that, in all cases, the perception agent's obstacle recognition accuracy improved after collaborative optimization. Furthermore, in both scenarios, the accuracy using three control agents was higher than using two. This is because as the number of control agents increases, the probability of encountering unknown objects also increases, resulting in a greater number of labeled unknown objects. Consequently, after retraining, the number of recognized unknown objects also increases.

[0073] We compared the number of convergence iterations for a multi-agent path planning task with and without collaborative optimization under different algorithms (IQL and IDQN), different numbers of control agents, and different scenarios to assess whether training efficiency has been improved. The results are shown in Table 1. Table 1 shows that, regardless of the scenario, the number of convergence iterations for the path planning task is significantly lower when collaborative optimization is used than when it is not, indicating that optimization can improve task efficiency. As the number of unknown objects in the simulation environment increases, the number of iterations required for convergence increases, regardless of whether collaborative optimization is used. This is primarily because the probability of a control agent colliding with these unknown objects increases with the number of unknown objects. A collision with an unknown object signals the end of the current iteration, increasing the number of iterations required for convergence. Furthermore, the number of control agents also affects the number of convergence iterations. In scenarios without collaborative optimization, the number of iterations required for convergence generally increases with the number of control agents. However, in scenarios with collaborative optimization, the number of iterations required for convergence decreases with the number of control agents. The main reason for this performance improvement is that multiple control agents are more likely to encounter different unknown objects, which enables the system to accumulate data on newly added unknown objects more quickly and then use it to optimize the perception agent model; when different algorithms (such as IQL and IDQN) are used, the impact of the two algorithms on the number of convergence iterations is not significant, which reflects the versatility of the proposed architecture.

[0074] Example 2

[0075] The multi-agent architecture based on online evolution proposed in this invention has a wide range of application scenarios, and the architecture can be applied to the field of autonomous driving.

[0076] In the field of autonomous driving, consider a local road segment that includes elements such as manned vehicles, autonomous vehicles (AVs), roadside units (RSUs), cameras, and radars. Typically, AVs are equipped with sensors, controllers, and actuators, while RSUs and cameras also contain sensors. Although collaborative perception technology has made significant progress in the field of autonomous driving, extreme weather conditions can still cause recognition errors, which in turn affect the decision-making of autonomous vehicles. In this case, a sensing-control collaborative framework can be used to enable the sensors of the autonomous vehicle, as well as the sensors of the RSUs and cameras, to learn adaptively. This can be divided into the following steps:

[0077] A1. Perception of the Perceptron

[0078] For autonomous vehicles, collaborative perception can be achieved by combining the vehicle's own perception information with data obtained from other vehicles, roadside units, cameras, and other traffic elements through communication to obtain perception results.

[0079] A2. Construction of Virtual World

[0080] Based on the perception results, a virtual world that includes factors such as the vehicle’s current state, road conditions, weather, and lighting can be constructed, and the predicted reward can be calculated.

[0081] A3. Controller Decision-Making and Execution

[0082] The controller of the autonomous vehicle is simulated in the virtual world, utilizing the generative model if necessary, and making decisions based on these factors to perform autonomous maneuvers such as following, passing, and stopping.

[0083] A4. Online Evolutionary Learning Process

[0084] After executing these maneuvers, feedback from the physical environment (such as whether the overtaking attempt was successful or the following distance) is used to optimize the vehicle's sensors, as well as those of cameras, roadside units, and other traffic elements. The model is periodically retrained to achieve online evolutionary learning. In this way, autonomous driving can adapt to changing environments, ultimately achieving the effects of artificial general intelligence.

[0085] Example 3

[0086] The multi-agent architecture based on online evolution proposed in this invention has a wide range of application scenarios, and the architecture can be applied to the field of embodied intelligence.

[0087] In the field of embodied intelligence, consider a multi-robot operation scenario where multiple robots need to collaborate to move an object to a specified location while avoiding potential obstacles. Each robot is equipped with perception, control, and execution components and makes decisions based on its perception results. However, in unfamiliar environments, conditions such as lighting and space may change, leading to perception errors. In this case, a sensor-control collaborative framework can be applied to achieve online evolution of the perceptron. Specifically:

[0088] B1. Perception of the Perceptron

[0089] The sensors of each robot collect raw data from the environment for collaborative perception, thereby obtaining perception results.

[0090] B2. Construction of Virtual World

[0091] Based on the perception results, a virtual world is constructed, which contains information such as the position of each robot, the position of the objects that need to be moved, and the position of obstacles, and the predicted reward is calculated.

[0092] B3. Controller Decision-Making and Execution

[0093] The robot's controller uses information from the virtual world to select a cooperative strategy and start moving the object.

[0094] B4. Online Evolutionary Learning Process

[0095] During the execution of an action, each robot receives feedback from the environment, such as whether it successfully lifted an object, whether it hit an obstacle, or whether it reached its target location. This feedback is compared with the predicted rewards in the virtual world. The robot then uses this information to optimize its sensor models, which are then retrained periodically. This improves the accuracy of its perception and enables the robot to adapt to its ever-changing environment.

[0096] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-agent architecture based on online evolution, characterized by: The specific steps include: S1. Perception Agent Perception Process Each sensor learns from the real world Get the original data , including images, sounds and other types of information. After obtaining the original information, each sensor uses a collaborative perception algorithm to fuse the original information obtained by each sensor, and then performs recognition detection on the information to obtain the recognition result. , the perception process can be expressed as: S2. Virtual world construction process Based on perception results , you can build a virtual world , in building a virtual world When the various perception information (such as the state information of the intelligent body, the type, position, size and other data information of the recognized object) are integrated together, and then the real world The information of each agent and various objects is mapped to the virtual world and use the function Prediction Rewards ; S3. Decision-making and execution of control agents The control agent makes decisions in the virtual world. This process can be divided into two aspects: on the one hand, the control agent uses the corresponding algorithm to select actions, and on the other hand, the selected actions are simulated in the virtual world to determine their correctness. The control agent can use different algorithms to achieve its goals, such as rule-based algorithms, genetic algorithms, particle swarm optimization and other optimization-based decision algorithms, Dijkstra, A * Heuristic search algorithms, reinforcement learning algorithms, game theory-based algorithms, etc. Among them, the more typical algorithm is the multi-agent reinforcement learning algorithm; S4. Online evolutionary learning process Get actual rewards for each controlled agent After that, it will be compared with the corresponding predicted reward in the virtual world Compare and if the difference between the two values ​​exceeds a certain threshold , it indicates that there is a deviation between the prediction of the perceptual agent and the actual situation. In this case, the model of the perceptual agent needs to be optimized. The function for judging whether the perceptual agent model needs to be optimized is expressed as , which is expressed as follows: ； in, equal This means that the model needs to be optimized. Since the perceptual agent model is trained based on supervised learning, we need to add new data as supervision information. Specifically, we will re-label the original data and express the labeling process as a function , which is expressed as follows: ； in, Represents the currently relabeled data, Represents an existing tag, Indicates a new tag. According to the control agent Feedback modified the label, express This means that under the premise of optimizing the perception agent model, when the real reward is equal to one of the predicted rewards, it means that the original perception agent model has made an error in recognition and the existing incorrectly recognized label needs to be replaced with an existing label. If the real reward is not in the predicted reward, it means that an object that has not appeared before is encountered and a new label needs to be assigned to the object. Therefore, the new data added to the perception agent model dataset can be expressed as: ； in, represents the perceptual agent model dataset, , , the perceptual agent can detect Different types of data, each type of data has its own corresponding training data set ,For perception agents that detect the same type of data, their model parameters can be shared or trained separately.

2. The multi-agent architecture based on online evolution according to claim 1, characterized in that: In the S1, It is Data of a perceptual agent The feature extraction model of , In the fusion process Data of a perceptual agent The weight of , It is a model for identifying fusion features, and the model parameters are .

3. The multi-agent architecture based on online evolution according to claim 1, characterized in that: In S2, the virtual world There are three main purposes: First, it can be used to pre-simulate the next action selected by the corresponding decision algorithm of the multi-agent To determine whether there will be a bad reward to avoid performing the action in the real world, and secondly, with the help of the virtual world In the generative model, we can transform the virtual world Taking the current state of the agent as input and providing a hint, these models can generate relevant content for reference to help the control agent choose the right action. Third, by comparing the actual rewards in the real world and prediction rewards in virtual worlds The target recognition model of the perceptual agent can be optimized to achieve online evolutionary learning, thereby ensuring accuracy and safety.

4. The multi-agent architecture based on online evolution according to claim 1, characterized in that: In S3, if a multi-agent reinforcement learning algorithm is used for decision making, a Markov game is used in the algorithm. The Markov game can be represented by a five-tuple, which is recorded as ,in, Indicates shared an intelligent agent, represents the state space set of the environment, Representing an agent The set of action spaces that can be selected, is the state transition function, Representing an agent In state The rewards received are: is a decay factor. In a multi-agent system, each agent maximizes its own reward as its learning goal. For the overall environment, there is no unified indicator that can measure the quality of the entire system. In this case, Nash equilibrium is used to represent the common reinforcement learning goal of multiple agents. The process of coordination between multiple agents can be expressed as the process of finding Nash equilibrium, that is: ； in, represents the set of all agent strategies, Indicates that all agents reach Nash equilibrium when the agent The cumulative reward is: ； In the above formula It refers to the state-action value function of multiple agents, and its calculation formula can be expressed as follows: ； Therefore, according to the above formula, Nash equilibrium can be expressed as: 。 5. The multi-agent architecture based on online evolution according to claim 4, characterized in that: Under the Nash equilibrium condition, after the action is selected using the multi-agent strategy selection algorithm, the control agent will act in the virtual world. ; if If a negative predicted reward is received in , it indicates that the action may not be optimal and needs to be reselected; in simple scenarios, this can be achieved by continuing to simulate the remaining actions in the current state and using The greedy strategy selects the action with the highest predicted reward. In more complex scenarios, such as continuous action scenarios with a large action space or dynamically changing environments, the pre-trained expert knowledge and task understanding of generative models can be used to assist in action selection. These models can generate auxiliary information consistent with human preferences based on the current state of the virtual world. By fusing this generated information with existing information in the virtual world, the range of action selection can be narrowed down, and the optimal action can be ultimately determined. The specific formula can be expressed as: ； in, In the virtual world In the simulation, the process of executing the action is simulated. represents the action selected by the corresponding decision algorithm of the multi-agent, where Represents a specific algorithm, Represents the control agent The current state, represents the auxiliary information provided by the generative model to help action selection, and Is an indicator function, when providing prompt information When , indicating the use of generative model information, if no prompt information is provided , the function value is , the generative model is not used; after the action is selected, the action is performed in the real world to obtain actual rewards and move to the next state.

6. The multi-agent architecture based on online evolution according to claim 1, characterized in that: The online evolution is a learning method that optimizes the online model of each agent in a multi-agent environment based on the interaction and fusion of multi-agent perception, control and generation information. It can autonomously adapt to the time-varying environment characteristics based on the constraints of the intelligent environment and achieve networked general artificial intelligence. The agents are divided into three categories, namely perception agents, control agents and generative models. Perception agents are (in ) is mainly responsible for perception; controlling the intelligent body (in ) for decision making, generative models (in ) generate auxiliary information to assist decision-making, and these three types of agents can collaborate with each other to achieve coordinated optimization.

7. The multi-agent architecture based on online evolution according to claim 1, characterized in that: The real world The virtual world is the environment in which each agent actually performs actions, in which the perception agent obtains raw data and the control agent performs actions and obtains feedback. It can refer to scenes in various fields. In order to provide a subjective mapping of the real world to each agent within its perception range, it is necessary to integrate various perceived information. For perceptual agents The acquired raw data includes images, locations, sounds, and other information.

8. The multi-agent architecture based on online evolution according to claim 2, characterized in that: The model parameters For perceptual agents The parameters used in the perception process are learned through the training process.

9. The multi-agent architecture based on online evolution according to claim 1, characterized in that: The perceived results It is the perception result obtained by the collaborative perception of intelligent agents, including the status of various agents, the type and location information of different objects.

10. The multi-agent architecture based on online evolution according to claim 1, characterized in that: The actual reward To control the agent in the real world The reward obtained after executing the action in Represents a specific control agent , Represents the state transition of the control agent, the predicted reward By function Computed virtual world The rewards in Represents the control agent , Represents state transfer, Mainly used for actual rewards Compare to determine whether relevant information needs to be used for coordinated optimization, the optimal action To control the agent Based on the strategies and virtual worlds it has learned The current best action is selected based on the information.

Images