Multi-agent cooperative control method for traffic scene

Abstract

The invention discloses a traffic scene-oriented multi-agent cooperative control method. The method comprises the following steps of receiving sensing control data information of multiple traffic agents in an intelligent traffic system based on deep reinforcement learning; extracting multi-source sensing features and feedback features of the multi-source sensing information and the feedback information, and performing multi-source heterogeneous information fusion on the multi-source sensing features and the feedback features to obtain sensing and control fusion features; the sensing and control fusion features are transmitted to the multiple traffic agents in real time, the sensing supervision and fusion feedback technology and the multiple traffic agents are adopted to conduct real-time information interaction to achieve new category data detection, incremental learning is conducted based on the new category data, and online updating iteration of the model is achieved to assist in updating strategies and actions of the multiple traffic agents. According to the method, the generalization in a data distribution change environment is continuously improved without manual supervision information, meanwhile, decision sharing and decision improvement between traffic agents are realized, and the decision execution efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology, and in particular to a multi-agent collaborative control method for traffic scenarios. Background Art

[0002] With the widespread application of artificial intelligence (AI) technology in modern transportation, various types of transportation tools are endowed with the concept of intelligent agents, which are an important part of the transportation system. As one of the common research objects of machine learning, intelligent agents are usually defined as entities in the environment, which can obtain and interpret data reflecting events in the environment, select actions based on the interpretation, and finally perform actions that have an impact on the environment, that is, they have the ability to perceive and make decisions. Intelligent agents are usually composed of sensors, controllers, and actuators, among which the actuators that only perform actions are mainly related to the physical state of the intelligent agent itself. In traffic scenarios, multiple traffic intelligent agents are often required to work together to cope with complex and changing road environments and limited computing and communication resources. In order to meet different application requirements, many researchers focus on the field of collaborative perception and control of multiple intelligent agents.

[0003] In the field of multi-agent collaborative perception in large-scale networked systems, different agents compensate each other by sharing their sensor information, thereby improving the effectiveness of subsequent local decisions. Multi-agent perception can be divided into three basic modes according to the different shared information: front fusion, middle fusion, and back fusion. The information shared in the front fusion is the original data, which can achieve the fusion of each original data losslessly. However, the transmission of original data by the front fusion method will consume a lot of communication resources, which is difficult to meet the scene with high requirements for real-time perception. In contrast, the back fusion shares the perception results generated independently by each agent. The back fusion communication bandwidth consumption is small, but because it is based on the results of each incomplete observation, it has noise or errors, and the accuracy of the fusion result is low. In order to meet the requirements of low communication resource consumption and high detection result accuracy at the same time, the middle fusion method has been proposed and has received more attention from researchers. Middle fusion means that each agent processes the original data into intermediate layer features and then shares them, and obtains the final perception result based on the fused features.

[0004] At the same time, multi-agent cooperative control of large-scale networked systems optimizes the behavior of multiple agents by coordinating their actions. Unlike traditional control methods for single agents, cooperative control technology relies on communication between agents to exchange information about their states and goals. It must consider the complex interactions and dependencies between agents and be able to adapt to changing conditions in real time. For example, the ramp merging strategy based on game theory is used to optimize the merging coordination of vehicles in mixed traffic. The strategy can determine the dynamic merging sequence and the corresponding longitudinal / lateral control. In addition to the multi-agent cooperative control method based on game theory, some researchers have formulated the cooperative control problem as an optimization problem based on optimization methods. The goal is to find the optimal action for each agent to maximize the overall performance of the vehicle queue in mixed traffic. For example, the mixed integer linear programming (MILP) model coordinates the driving trajectory of autonomous vehicles from the perspective of total vehicle delay. In addition, the reinforcement learning-based method trains the control algorithm through trial and error, that is, the algorithm learns from its own experience in the environment and adapts to the changing environment. For example, the architecture of blockchain-integrated multi-agent deep reinforcement learning (Block-MADRL).

[0005] Most existing methods are based on the framework of sensors unilaterally providing environmental information to controllers, and the multi-agent collaborative algorithms of multi-agent collaborative perception and collaborative control in large-scale networked systems are executed separately. In this unidirectional framework, the upper limit of controller performance is often limited by the perception accuracy of the sensor. Therefore, it is urgent to introduce a sensing and control collaborative decision-making mechanism based on multi-agent online evolutionary learning, and form an intelligent agent bidirectional optimization framework through a reverse optimization channel from the controller to the sensor, so that the perception and control units can be collaboratively optimized to adapt to unknown changes in the dynamic environment and improve the operating efficiency of the transportation system. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a multi-agent collaborative control method for traffic scenarios. In the method, various perception and control task models are separated from artificial supervision information and continuously improve their generalization in the environment of data distribution changes, while realizing decision sharing and decision improvement among traffic agents. By using local collaborative decision-making technology, the local decision subsystem of a large-scale networked system can quickly adapt to non-stationary environmental changes, improving the efficiency and accuracy of decision execution.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] The multi-agent collaborative control method for traffic scenarios includes the following steps:

[0009] Receiving sensory control data information of multiple traffic agents in an intelligent traffic system based on deep reinforcement learning; the multiple agents include traffic agents with interactive functions and traffic agents without interactive functions, and the sensory control data information includes multi-source perception information of traffic agents with interactive functions and traffic agents without interactive functions and feedback information of traffic agents with interactive functions; each traffic agent is provided with a perception end model;

[0010] Extracting multi-source sensing features and feedback features of the multi-source sensing information and feedback information, and performing multi-source heterogeneous information fusion on the multi-source sensing features and feedback features to obtain sensing and control fusion features;

[0011] The sensor-control fusion features are transmitted to multiple traffic agents in real time, and the perception supervision and fusion feedback technology is used to conduct real-time information interaction with multiple traffic agents to realize new category data detection, and incremental learning is performed based on the new category data to achieve online update iteration of the model to assist multiple traffic agents in updating strategies and actions.

[0012] Preferably, the multi-source heterogeneous information fusion is performed on the multi-source perception features and feedback features to obtain the sensing and control fusion feature V s , self-supervised transfer learning is used, and the formula of its loss function is as follows:

[0013] E(θ n ,θ o )=ΣL(G n (V s ,θ n ),y s )-αΣL(G o (V S ,θ o ),y s )

[0014] Among them, the sensing and control fusion feature V s represents the target domain; the source domain y s To control the deterministic physical world characteristics in feedback; G n and G o represents the prediction results of the target classifier and the source domain classifier, L represents the classification loss, and θ n and θ o They correspond to the internal parameters of the target classifier and the source domain classifier respectively, and α is a balancing hyperparameter.

[0015] Preferably, perception supervision and fusion feedback technology is used to interact with multiple traffic agents in real time to detect new category data, and online training and learning are performed based on the new category data to achieve model update iteration to assist multiple traffic agents in updating strategies and actions, which specifically includes the following steps:

[0016] During the evolution of the data stream, the model uses entropy and probability to jointly derive the score of the new category data. The formula is as follows:

[0017] Sco(V s )=min(λc,1)Prob(V s )+max(1-λc,0)∑-V s logV s

[0018] In the formula, Sco is based on V s If the score of the new data is higher than the set threshold, it is judged as a new class; c is the number of blocks starting from 0 in each new class cycle, and the max / min operator keeps the weight between (0,1); λ is a trade-off parameter used to dynamically balance the two criteria; Prob is the degree of similarity between the new data and the discovered new class;

[0019] When the perception model is updated according to the new category data, the change of model parameters further updates the state information of the traffic agent. The traffic agent updates its strategy and action according to the real-time state information and surrounding environment information. The formula is as follows:

[0020]

[0021] In the formula, is the reward for action a in state s, is the state transition matrix, E π is the expected function, R t+1 is the reward obtained by the agent at time t+1, Q π is the action value function of the current policy function π, t is the current moment, S t is the state of the agent at time t, and its specific value is represented by s. t is the action that the agent is to perform at time t. Its specific value is represented by a. γ is the discount factor, and its value range is 0-1.

[0022] Preferably, the method further includes: based on the iterated model, realizing autonomous lane changing and autonomous collision avoidance of traffic agents with interactive functions in mixed traffic sections, and the specific steps are as follows:

[0023] During the movement of the interactive traffic agent, the driving efficiency is composed of the path plan completion rate and the moving deceleration ratio, and its formula can be constructed as follows:

[0024] μ=δ·σ(r,a)+(1-δ)·ρ(a)

[0025] Among them, δ is the balance coefficient, a represents the motion decision, r is the path decision, σ represents the path plan completion rate, which is the ratio of the vehicle's path plan successfully completed during the entire driving process, and ρ represents the mobile deceleration ratio, which is the proportion of the vehicle's deceleration movement in all the motion decisions taken during the entire driving process;

[0026] In the construction of the path and motion decision cost function, based on the surrounding environment of the interactive traffic agent, the driving status of traditional vehicles and the road environment, the total decision cost function θ can be constructed as follows:

[0027] θ=w s Q s (r,a)+w c Q c (r)+w d Q d (r,a)+Z

[0028] Where Q s , Q c and Q d are the cost functions for static safety, comfort, and dynamic safety, normalized to values ​​between 0 and 1, and w s 、w c and w d It is the weight of the three cost functions, and Z measures the cost function of violating traffic rules during vehicle driving:

[0029]

[0030] Among them, Q g The cost generated by the road geometry indicates that the vehicle exceeds the lane boundary and is repelled toward the passable lane. is the traffic rule constraint of the road section, Q a is the acceleration cost, w g 、w γ and w a are the weights of the three cost functions.

[0031] Based on the above technical solution, the beneficial effects of the present invention are:

[0032] 1) The present invention improves the utilization rate of integrated information: The integration of multi-source information of multiple traffic agents realizes the full mining of the value of multi-source and multi-modal data information, and further expands the general capabilities of the model. In the multi-traffic agent autonomous collision avoidance simulation experiment, the data utilization rate was increased by 5.4% compared with the model that did not adopt the multi-source information integration technology;

[0033] 2) The present invention enhances the generalization ability of the model: Online evolutionary learning allows the model to be separated from manual supervision information during the training process, thereby improving the model's generalization ability for novel categories. In the experiment, the recognition accuracy of the model using online evolution when facing new categories was about 2.7% higher than that of the model without online evolution;

[0034] 3) The present invention increases the self-evolution capability of the model: the introduction of online evolutionary learning and reinforcement learning enables the model to collect real-time information about changes in the real environment for self-iteration and evolution, and can make self-adjustments according to the real-time changing environment to cope with various scenarios that may be encountered in the future;

[0035] 4) The present invention reduces communication overhead: through the sensor-control collaborative technology, effective information is transmitted in real time between multiple traffic agents, and is applied to the construction of local system knowledge sharing and control mechanisms. In the experiment, the training efficiency is significantly improved compared with the traditional QMIX algorithm, and the communication overhead is reduced by 10%. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a multi-agent collaborative control method for a traffic scenario in an embodiment;

[0037] Figure 2 is a schematic diagram of multi-source heterogeneous information fusion in a multi-agent collaborative control method for traffic scenarios in an embodiment;

[0038] Figure 3 is a schematic diagram of collaborative online evolutionary learning in a multi-agent collaborative control method for traffic scenarios in an embodiment;

[0039] Figure 4 The diagram is a schematic diagram of a sensing and control collaborative online evolution framework in a multi-agent collaborative control method for traffic scenarios in one embodiment. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0041] like Figures 1 to 4 As shown, this embodiment aims at the local autonomous decision-making problem in the intelligent transportation system for traffic scenes, takes self-supervised learning and multi-agent reinforcement learning theory as the main theoretical basis, and aims at the association integration of agents and the adaptive learning problem of perception and control tasks. Through the association information integration technology, discrimination information fusion technology, and collaborative perception and control data adaptive learning technology among multiple traffic agents, a multi-agent collaborative control method for traffic scenes is proposed, including the following steps:

[0042] Step 1, receiving sensory control data information of multiple traffic agents in an intelligent traffic system based on deep reinforcement learning; the multiple agents include traffic agents with interactive functions and traffic agents without interactive functions, and the sensory control data information includes multi-source perception information of traffic agents with interactive functions and those without interactive functions and feedback information of traffic agents with interactive functions; a perception end model is set on each traffic agent side.

[0043] In this embodiment, the objects are traffic agents with interactive functions and traffic agents without interactive functions in mixed traffic scenarios. The traffic agents with interactive functions include buses, self-driving vehicles, drones, etc., and the traffic agents without interactive functions include traditional vehicles, roadside equipment, etc. Basic preparation stage: 1) Data preparation: The sensors equipped with buses, self-driving vehicles, traditional vehicles, roadside equipment, and drones capture the surrounding environment data and perform necessary preprocessing; 2) Basic model loading: Buses, self-driving vehicles, traditional vehicles, roadside equipment, and drones load the perception end model suitable for themselves to ensure that the traffic agent has basic capabilities.

[0044] Step 2: extracting multi-source perception features and feedback features of the multi-source perception information and feedback information, and performing multi-source heterogeneous information fusion on the multi-source perception features and feedback features to obtain sensing and control fusion features.

[0045] In this embodiment, multi-source heterogeneous information fusion is adopted: each traffic intelligent body transmits the data after feature extraction to the multi-source information fusion technology module, and transmits the fused data to the information fusion center for aggregation processing to obtain the sensing and control fusion feature V s The specific instructions are as follows:

[0046] Based on the theoretical foundation of hierarchical network system modeling and characterization, each agent in the hierarchical community structure model has independent dynamic behavior, and the traffic agent can independently adjust its speed, steering and other states. Through the analysis and judgment of the paths and motion planning of multiple traffic agents, the deep correlation between the perception data of multiple traffic agents and their paths and motion tasks can be effectively summarized. First, Figure 2 As shown in the figure, multi-source heterogeneous information (images, point clouds, ultrasound, controller information, etc.) is jointly modeled:

[0047] V S =φ(P1,P2,…,P n ,F1,F2,…,F m )

[0048] Among them, φ is the feature fusion function, V s represents the characteristics of multi-source sensory control data after fusion in different scenarios, P nrepresents the multi-source perception features of the same dimension obtained by using different feature extraction networks for different traffic agents, F m represents the feedback feature extracted from the controller feedback information of the interactive traffic agent. s Input into the subsequent task network to realize the online evolution of multiple traffic agents.

[0049] Based on the self-classifiers of multi-traffic agents, in order to better utilize the sensor-control fusion information obtained in real time to carry out online evolutionary learning, a self-supervised domain adaptation method is adopted, that is, the source domain and the target domain have the same tasks, the data of the two domains are similar but different, there are a large number of labels in the source domain, and there are no labels in the target domain. The sensor-control fusion feature V we just obtained s represents the target domain, and the source domain y s is the deterministic physical world feature in the control feedback. Therefore, it is hoped that a domain adaptation algorithm can be trained on the source domain to obtain the deterministic physical world feature in the control feedback. s To V s The mapping is used to map the target domain through y s Constrained Estimation V s At this point, we can get the loss for self-supervised transfer learning:

[0050] E(θ n ,θ o )=ΣL(G n (V s ,θ n ),y s )-αΣL(G o (V S ,θ o ),y s )

[0051] Among them G n and G o represents the prediction results of the target classifier and the source domain classifier, L represents the classification loss, and θ n and θ o are the internal parameters of the target classifier and the source domain classifier, respectively, and α is a balancing hyperparameter. For the multi-source information perceived by the traffic agent and the control feedback information of the interactive traffic agent, an adaptive method is used to realize the self-supervision constraint of the deterministic physical world characteristics in the control feedback on the fusion of the sensing and control information, and a more accurate overall feature estimate V is obtained than any single traffic agent involved in the fusion estimation. s By studying the real-time optimization and representation learning of multiple proxy tasks, we can obtain a pre-trained model suitable for the overall goal and accommodate complex knowledge from multi-source data.

[0052] Step 3: Fusion feature V sThe information is transmitted to multiple traffic agents in real time, and perception supervision and fusion feedback technology are used to interact with multiple traffic agents in real time to detect new category data. Incremental learning is performed based on new category data to achieve online update and iteration of the model to assist multiple traffic agents in updating strategies and actions.

[0053] In this embodiment, perception supervision and fusion feedback technology: according to the real-time status of each traffic agent and its surrounding environment information, and in accordance with the different capabilities of each traffic agent, perception supervision and fusion feedback technology are used to interact with the traffic agent in real time, so that the traffic agent can fully grasp the surrounding information to make the next decision. Collaborative online evolutionary learning: The perception side model of each traffic agent realizes new category data detection under the multi-traffic agent sensing and control collaborative framework, and performs training and learning based on the new category data to realize model update and iteration. The details are as follows:

[0054] Design a collaborative online evolutionary learning method for multiple traffic agents to achieve task-oriented adaptive learning of multiple traffic agents and reduce human intervention. Through the fusion of the perception information of multiple traffic agents and the control feedback information of interactive traffic agents, and based on the information sharing mechanism of each traffic agent in the multi-traffic agent system, using more accurate sensing and control fusion feature V s Give each traffic sensing agent corresponding supervision information to achieve online model optimization of the traffic sensing agent; at the same time, give each traffic control agent more comprehensive feedback information to achieve online model optimization of the traffic control agent. Figure 3 Example.

[0055] The model deployed at the sensing end of each traffic agent is initially trained for several known categories, and the characteristics of new categories are unknown. During the evolution of the data stream, the model uses entropy and probability to jointly derive the score of new data:

[0056] Sco(V s )=min(λc,1)Prob(V s )+max(1-λc,0)∑-V s logV s

[0057] In the formula, Sco is based on V sIf the new data score is higher than the set threshold, it is judged as a new class; c is the number of blocks starting from 0 in each new class cycle, and the max / min operator keeps the weight between (0,1); λ is a trade-off parameter used to dynamically balance the two criteria; Prob is the degree of similarity between the new data and the discovered new class. If an instance is similar to the new class prototype, it is likely to be a new class. Here is the normalized novel class similarity. New classes that appear at different stages are defined according to the new data score. After the new class data is updated, the model conducts incremental learning based on the new class data. As the data stream evolves, the model accumulates prior information and establishes appropriate feature prototypes for new instances for subsequent new class discrimination.

[0058] When the perception model is updated according to the new data, the changes in model parameters further update the state information of the traffic agent. The traffic agent updates its strategy and actions based on the real-time state information and surrounding environment information:

[0059]

[0060] In the formula, is the reward for action a in state s, is the state transition matrix, E π is the expected function, R t+1 is the reward obtained by the agent at time t+1, Q π is the action value function of the current policy function π, t is the current moment, S t is the state of the agent at time t, and its specific value is represented by s. t It is the action that the agent is to perform at time t, and its specific value is represented by a. γ is the discount factor, and its value range is 0-1. The traffic agent continuously takes different actions according to the surrounding environment information, and its state is also dynamically updated in real time. In this dynamic change and adjustment process, the perception information and control information realize real-time interactive coordination.

[0061] The advantages of multi-traffic intelligent body sensing and control coordination are: 1) Controlling traffic intelligent body can continuously s By obtaining more accurate fusion perception information and more comprehensive control feedback information, the controller can continuously optimize its online adaptation to non-stationary environmental changes to achieve better control effect and response speed; 2) The intelligent body of traffic perception can continuously obtain information from V s It can obtain more accurate feature supervision information than its own perception feature estimation, and at the same time continuously optimize the online adaptation of the sensor to non-stationary environmental changes to achieve better perception capabilities; 3) In the framework of multiple traffic agents, it can unify supervised learning and reinforcement learning based on deep networks, and form online evolutionary learning that does not rely on manually labeled information.

[0062] By exploring the correlation between the real-time reward and perception information obtained by the controller in multi-transportation agent reinforcement learning, the impact on online evolutionary learning is studied; and for path and motion planning tasks, the corresponding online evolutionary learning simulation environment is established to perform elastic search on the number of transportation agents. Through multiple rounds of judgment on the impact of the number of transportation agents on the overall, individual performance and generalization, a reliable large-scale deployment plan for transportation agents with sensor-control collaboration is established.

[0063] In one embodiment, the multi-agent cooperative control method for traffic scenarios also includes providing an iterative model to realize the autonomous lane changing and autonomous collision avoidance process of traffic agents with interactive functions in mixed traffic sections. The specific steps of the process are as follows:

[0064] In mixed traffic scenarios, based on the multi-source heterogeneous information fusion mechanism of multiple traffic agents and the multi-traffic agent sensing and control collaborative online evolution method, the autonomous lane changing and autonomous collision avoidance of vehicles in mixed traffic sections can be realized, driving efficiency can be improved, and thus the traffic efficiency of the section can be improved. In the collaborative path and motion decision-making task processing of multiple physical control traffic agents, realizing efficient autonomous reasoning and decision-making of multiple traffic agents is an urgent problem to be solved in this task. In mixed traffic scenarios, improving the driving efficiency of vehicles is the main goal. At the same time, how to define and measure the costs of different paths and motion decisions taken by vehicles during driving is also crucial. Therefore, the goal of collaborative decision-making of multiple traffic agents is transformed into maximizing the vehicle driving efficiency μ, while minimizing the decision-making cost θ caused by vehicle collisions and violations of traffic rules.

[0065] During the movement of physical traffic agents, driving efficiency is mainly composed of the path plan completion rate and the mobile deceleration ratio, and its formula can be constructed as follows:

[0066] μ=δ·σ(r,a)+(1-δ)·ρ(a)

[0067] Among them, δ is the balance coefficient, a represents motion decisions, such as acceleration, deceleration, and parking, and r represents path decisions, such as lane change behaviors such as changing left and right lanes. σ represents the path plan completion rate, which is the ratio of the vehicle's path plan successfully completed during the entire driving process, and ρ represents the mobile deceleration ratio, which is the proportion of the vehicle's deceleration movement in all the motion decisions taken during the entire driving process.

[0068] In the construction of the path and motion decision cost function, based on the surrounding environment of the physical control traffic agent, the driving state of the traditional vehicle and the road environment, the total decision cost function θ can be constructed as follows:

[0069] θ=w s Q s (r,a)+w c Q c(r)+w d Q d (r,a)+Z

[0070] Where Q s , Q c and Q d are the cost functions for static safety (the cost of avoiding static obstacles on the road), comfort, and dynamic safety (the cost of avoiding obstacles for traditional vehicles on the road), normalized to values ​​between 0 and 1. s 、w c and w d are the weights of the three cost functions, the higher w s May result in a safer but slower lane change path, while higher w c May result in shorter paths, higher w d It may allow the vehicle to maintain its original speed and overtake the moving vehicle. Z measures the cost function of violating traffic rules during vehicle driving:

[0071]

[0072] Among them, Q g The cost generated by the road geometry indicates that the vehicle exceeds the lane boundary and is subject to a repulsive force (cost) directed towards the passable lane. is the traffic rule constraint of the road section. a The acceleration cost is mainly due to the friction limit of the tires and the throttle limit of the vehicle. g , and w a are the weights of the three cost functions.

[0073] At the road section level of mixed traffic, multiple physical traffic agents (vehicles, roadside units, drones, etc.) follow traffic rules, traffic signal instructions and control instructions of upper-level road section-level traffic agents, and make autonomous decisions through sensor-control collaboration. The physical control traffic agents maximize driving efficiency μ and minimize decision costs θ, and obtain their own real-time paths and autonomous motion decisions.

[0074] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0075] The above is only a preferred implementation of the multi-agent collaborative control method for traffic scenarios disclosed by the present invention, and is not intended to limit the protection scope of the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of this specification shall be included in the protection scope of the embodiments of this specification.

Claims

1. Multi-agent collaborative control method for traffic scenarios, It is characterized in that The steps include: Receiving sensory control data information of multiple traffic agents in an intelligent traffic system based on deep reinforcement learning; the multiple agents include traffic agents with interactive functions and traffic agents without interactive functions, and the sensory control data information includes multi-source perception information of traffic agents with interactive functions and traffic agents without interactive functions and feedback information of traffic agents with interactive functions; each traffic agent is provided with a perception end model; Extracting multi-source sensing features and feedback features of the multi-source sensing information and feedback information, and performing multi-source heterogeneous information fusion on the multi-source sensing features and feedback features to obtain sensing and control fusion features; The sensor-control fusion features are transmitted to multiple traffic agents in real time, and the perception supervision and fusion feedback technology is used to conduct real-time information interaction with multiple traffic agents to realize new category data detection, and incremental learning is performed based on the new category data to achieve online update iteration of the model to assist multiple traffic agents in updating strategies and actions.

2. According to the multi-agent collaborative control method for traffic scenarios according to claim 1, It is characterized in that The multi-source heterogeneous information fusion of multi-source perception features and feedback features is performed to obtain the sensing and control fusion feature V s , self-supervised transfer learning is used, and the formula of its loss function is as follows: E(θ n ,the o )=∑L(G n (V s ,the n ),y s )-α∑L(G o (V S ,the o ),y s ) Among them, the sensing and control fusion feature V s represents the target domain; the source domain y s To control the deterministic physical world characteristics in feedback; G n and G o represents the prediction results of the target classifier and the source domain classifier, L represents the classification loss, and θ n and θ o They correspond to the internal parameters of the target classifier and the source domain classifier respectively, and α is a balancing hyperparameter.

3. The multi-agent collaborative control method for traffic scenarios according to claim 1, It is characterized in that The perception supervision and fusion feedback technology is used to interact with multiple traffic agents in real time to detect new category data, and online training and learning are carried out based on the new category data to achieve model update iteration to assist multiple traffic agents in updating strategies and actions. The specific steps include the following: During the evolution of the data stream, the model uses entropy and probability to jointly derive the score of the new category data. The formula is as follows: Sco(V s )=min(λc,1)Prob(Vs)+max(1-λc,0)∑-V s logV s In the formula, Sco is based on V s If the score of the new data is higher than the set threshold, it is judged as a new class; c is the number of blocks starting from 0 in each new class cycle, and the max / min operator keeps the weight between (0, 1); λ is a trade-off parameter used to dynamically balance the two criteria; Prob is the degree of similarity between the new data and the discovered new class; When the perception model is updated according to the new category data, the change of model parameters further updates the state information of the traffic agent. The traffic agent updates its strategy and action according to the real-time state information and surrounding environment information. The formula is as follows: In the formula, is the reward for action a in state s, is the state transfer matrix, E π is the expected function, R t+1 is the reward obtained by the agent at time t+1, Q π is the action value function of the current policy function π, t is the current moment, S t is the state of the agent at time t, and its specific value is represented by s. t is the action that the agent is to perform at time t. Its specific value is represented by a. γ is the discount factor, and its value range is 0-1.

4. The multi-agent collaborative control method for traffic scenarios according to claim 1, It is characterized in that Also includes: Based on the iterative model, autonomous lane changing and autonomous collision avoidance of traffic agents with interactive functions in mixed traffic sections are realized. The specific steps are as follows: During the movement of the interactive traffic agent, the driving efficiency is composed of the path plan completion rate and the moving deceleration ratio, and its formula can be constructed as follows: μ=δ·σ(r, a)+(1-δ)·ρ(a) Among them, δ is the balance coefficient, a represents the motion decision, r is the path decision, σ represents the path plan completion rate, which is the ratio of the vehicle's path plan successfully completed during the entire driving process, and ρ represents the mobile deceleration ratio, which is the proportion of the vehicle's deceleration movement in all the motion decisions taken during the entire driving process; In the construction of the path and motion decision cost function, based on the surrounding environment of the interactive traffic agent, the driving status of traditional vehicles and the road environment, the total decision cost function θ can be constructed as follows: θ=w s Q s (r,a)+w c Q c (r)+w d Q d (r,a)+Z Where Q s , Q c and Q d are the cost functions for static safety, comfort, and dynamic safety, normalized to values ​​between 0 and 1, and w s 、w c and w d It is the weight of the three cost functions. Z measures the cost function of violating traffic rules during vehicle driving: Among them, Q g The cost generated by the road geometry indicates that the vehicle exceeds the lane boundary and is repelled toward the passable lane. is the traffic rule constraint of the road section, Q a is the acceleration cost, w g 、w γ and w a are the weights of the three cost functions.

Images