Multi-agent asynchronous perception simulation method and system for vehicle-road cooperation

Abstract

The invention discloses a multi-agent asynchronous perception simulation method and system for vehicle-road cooperation, and the method comprises the steps: setting detailed parameters of a vehicle-road cooperation simulation scene, and starting an asynchronous world stream; creating a plurality of vehicle-road cooperative intelligent agents and common traffic participants, and planning traffic behaviors; a multi-source heterogeneous sensor is constructed on the vehicle-road cooperative intelligent agent, and asynchronous sensing is achieved; a data processing module is created on the vehicle-road cooperation intelligent agent, and processing and sharing of sensing data are achieved; a vehicle-road cooperation module is created on the vehicle-road cooperation agent, and cross-agent data fusion is realized; an algorithm training and testing process suitable for the simulation method is established, and integrated training and testing are achieved. The method can be used for vehicle-road collaborative simulation and perception algorithm development, the simulation process is closer to the real environment through asynchronous perception, asynchronous transmission and asynchronous processing, communication delay can be simulated more accurately, and the practicability and reliability of the perception algorithm can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent connected vehicles, and in particular to a multi-agent asynchronous perception simulation method and system for vehicle-road collaboration. Background Art

[0002] With the rapid development of intelligent technology and autonomous driving, intelligent connected vehicles and intelligent transportation systems are merging at an unprecedented speed, injecting new vitality into the modern transportation system. Among them, autonomous driving technology, as the most promising development direction in intelligent connected vehicles, is undergoing an evolution from traditional single-vehicle perception and decision-making to vehicle-road collaboration. Vehicle-road collaboration technology provides a new opportunity for autonomous driving. Automated driving vehicles can efficiently exchange data and share information with various entities such as transportation infrastructure, other vehicles and pedestrians, thereby maintaining a high degree of vehicle coordination and situational awareness in a more complex traffic environment. It not only improves road safety, but also greatly promotes traffic efficiency.

[0003] Although vehicle-road collaboration technology has made significant progress, its performance in complex traffic environments cannot fully meet actual needs. How to ensure the real-time and effectiveness of information transmission has become one of the main challenges. Considering the limited bandwidth resources, the collaboration process needs to balance performance and bandwidth to achieve reliable operation of autonomous vehicles in complex scenarios.

[0004] At present, most vehicle-road cooperative algorithms use data sets for training and testing, and simulate communication conditions by adding random noise. This type of method is quite different from the real environment, and the added random noise cannot accurately simulate the complex communication conditions and dynamic environment in the real world. In addition, some algorithms use simulation frameworks for training and testing, but most of them adopt synchronous mode. In synchronous mode, all parts of the simulation system run synchronously at a fixed time step. This method simplifies the implementation process of the algorithm, but it also ignores the asynchrony and non-determinism of the real environment, so it is difficult to meet real needs. Summary of the invention

[0005] The present invention proposes a multi-agent asynchronous perception simulation method and system for vehicle-road collaboration, which can simulate the asynchronous perception and communication process of vehicles in a real environment and provide an integrated training and testing solution. In the method proposed by the present invention, each agent can independently and autonomously perform real-time perception, communication and collaboration, thereby more accurately simulating the real world, helping to improve the coping ability of intelligent networked vehicles in complex environments, and providing new research ideas and experimental basis for the development and application of related technologies.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A multi-agent asynchronous perception simulation method for vehicle-road collaboration, the method comprising:

[0008] S1. Set the detailed parameters of the vehicle-road collaborative simulation scenario and start the asynchronous world flow;

[0009] S2, create multiple vehicle-road cooperative agents and ordinary traffic participants, and plan traffic behaviors;

[0010] S3: Build multi-source heterogeneous sensors on the vehicle-road cooperative agent to achieve asynchronous perception;

[0011] S4. Create a data processing module on the vehicle-road cooperative agent to realize the processing and sharing of perception data;

[0012] S5. Create a vehicle-road collaboration module on the vehicle-road collaboration agent to achieve cross-agent data fusion;

[0013] S6. Establish an algorithm training and testing process suitable for this simulation method to achieve integrated training and testing.

[0014] To further illustrate, the step S1 is specifically as follows:

[0015] S11, initializing a map model in the autonomous driving simulation platform Carla, and storing it as a file, so as to facilitate direct loading of the corresponding map later;

[0016] S12, reading and parsing the configuration file, loading the preset configuration parameters, creating the simulation world and the traffic manager according to the preset configuration parameters, and switching their operation mode to the asynchronous mode;

[0017] S13. Asynchronously create a unified signal system, configure a communication publisher, and use the communication main port to uniformly send a start signal and an end signal to control the start and stop of the simulation environment.

[0018] To further illustrate, the step S2 is specifically as follows:

[0019] S21, reading the agent information in the preset configuration parameters, and creating a corresponding number of agent instance modules according to the preset number of agents, each of which is an asynchronous process;

[0020] S22. Connect the Carla server in each agent instance module, read the map information, and generate an agent instance at the corresponding position on the map according to the preset road position and type of the current agent. When the agent type is a car, if its preset mode is automatic planning, it will be connected to the traffic manager, and Carla will automatically plan its behavior path; if its preset mode specifies the starting point and the end point, a path planner for the agent will be created, and local waypoints will be output through the control algorithm and perception results to achieve basic automatic driving under the specified path;

[0021] S23. After all agent instances are generated, the number, type and road position of ordinary traffic participants are obtained using random functions in the main process. Based on this, ordinary traffic participant instances are created at corresponding positions on the map and connected to the traffic manager. Carla automatically plans their behavior paths. If other instances already exist at the random position, create instances at other idle road positions.

[0022] S24. All intelligent agent instances and common traffic participant instances are connected to the unified signal system through the communication main port, and each instance object is destroyed by the unified signal system after the simulation is completed.

[0023] To further illustrate, the step S3 is specifically as follows:

[0024] S31, create a shared memory (named Shared_mem_origin) and a sensor management module in each agent instance module;

[0025] S32, in the sensor management module, based on the sensor type, location and attributes preset in the current agent configuration parameters, create multi-source heterogeneous sensors and bind them to the current agent instance;

[0026] S33, setting the callback function of the sensor, serializing the sensor's perception data into a byte stream, and writing it into the shared memory Shared_mem_origin for use by subsequent data processing modules.

[0027] To further illustrate, the step S4 is specifically as follows:

[0028] S41. Add external hardware devices (external hardware devices refer to devices that can forward network data, such as single-chip microcomputers and computers) to realize network data forwarding, and at the same time limit the bandwidth of each port used by the vehicle-road collaboration, so as to simulate the vehicle-road collaboration communication process in a real environment;

[0029] S42, create a data processing module in each agent instance module, which is a new asynchronous process. In the data processing module, create a shared memory (named Shared_mem_process) and a ZeroMQ communication channel, where the ZeroMQ communication channel is connected to the external hardware device through an IP address;

[0030] S43, read the byte stream data in the shared memory Shared_mem_origin, and deserialize it to obtain the sensor's perception data. Since the data processing module is an asynchronous process relative to the sensor management module, a mutex mechanism is introduced to isolate the operations of the two on the shared memory Shared_mem_origin to prevent the shared memory data from being confused;

[0031] S44, obtaining the traffic participants within the perception radius of the current intelligent agent through the Carla server, including other intelligent agents and ordinary traffic participants. If only fuzzy labels are required in the preset configuration parameters, the object IDs of these traffic participants in the Carla world are directly stored in the label list; if precise labels are required in the preset configuration parameters, the bounding box of each traffic participant is first obtained, and then it is determined whether the laser radar point cloud is projected into the bounding box, so as to screen out the perceived traffic participants and store their object IDs in the Carla world in the label list;

[0032] S45, the vehicle-road cooperative fusion mode includes four types: no fusion, early fusion, mid-term fusion and late fusion. In the data processing module, the sensor perception data is processed according to the preset fusion mode. In the no fusion mode, the data processing module stores the sensor perception data, posture, timestamp and label list of the current intelligent body in the data dictionary, and serializes it into a byte stream, which is only written to the shared memory Shared_mem_process;

[0033] S46. In the early fusion mode, the data processing module first stores the sensor perception data, position, timestamp and label list of the current agent in the data dictionary, and serializes it into a byte stream and writes it into the shared memory Shared_mem_process. At the same time, the data dictionary is published to the ZeroMQ communication channel, and other agents can obtain the data sent by the current agent by subscription;

[0034] S47. In the mid-term fusion mode, the data processing module first pre-processes the data according to the sensor data and posture of the current intelligent agent, voxelizes the lidar point cloud, and obtains voxel features, voxel coordinates and voxel points; then, the data processing module further processes the voxel features based on the detection algorithm. In the currently commonly used vehicle-road cooperative algorithm, after obtaining the voxel features, PillarVFE, PointPillarScatter and Backbone are usually used one by one to process them, and different convolution operations are performed according to the detection algorithm to obtain intermediate features; finally, the vehicle-road cooperative module stores the intermediate features, posture, timestamp and label list in the data dictionary, and serializes them into a byte stream and writes them into the shared memory Shared_mem_process. At the same time, the data dictionary is published to the ZeroMQ communication channel, and other intelligent agents can obtain the data sent by the current intelligent agent through the subscription method;

[0035] S48. In the late fusion mode, the data processing module further processes the intermediate features obtained by the mid-term fusion according to the detection algorithm, and outputs the detection results of the object, such as object classification, bounding box position, etc.; then, the detection results, posture, timestamp and label list are stored in the data dictionary, and serialized into a byte stream and written into the shared memory Shared_mem_process. At the same time, the data dictionary is also published to the ZeroMQ communication channel, and other agents can obtain the data sent by the current agent through the subscription method;

[0036] S49. The processing and sharing of perception data by the data processing module provides a data basis for the subsequent vehicle-road cooperative module. Since the data processing module is an asynchronous process relative to its superior intelligent agent instance module, its data processing process will not affect the real-time perception of the sensor.

[0037] To further illustrate, the step S5 is specifically as follows:

[0038] S51, reading the agent information in the preset configuration parameters, and creating a corresponding number of vehicle-road cooperative modules according to the preset number of agents, each of which is an asynchronous process;

[0039] S52. In the vehicle-road collaboration module, the current agent connects to the shared memory Shared_mem_process created in its data processing module and subscribes to the ZeroMQ communication channel of other agents.

[0040] S53, the vehicle-road collaboration module scans the shared memory Shared_mem_process when idle, and deserializes the byte stream data therein to restore the data dictionary. If the timestamp in the data dictionary has changed relative to the timestamp of the previous data, it indicates that the current agent has generated new data and the next step of processing is performed; if the timestamp has not changed, the scan continues. Since the vehicle-road collaboration module is an asynchronous process relative to the data processing module, a mutex mechanism is introduced to isolate the operations of the two on the shared memory Shared_mem_process to prevent confusion of shared memory data;

[0041] S54, after obtaining the new data of the current intelligent agent, the vehicle-road collaboration module obtains metadata in the ZeroMQ communication channel of other intelligent agents, and makes distance judgment based on the posture data. If it is judged that both parties are within the preset collaboration range, the complete data is received; if not, the data is skipped;

[0042] S55: If the vehicle-road collaboration module obtains complete data of other intelligent agents, it loads the data together with the data of the current intelligent agent into the list of data to be regularized; if the data of other intelligent agents are all out of the collaboration scope, it loads the data of the current intelligent agent alone into the list of data to be regularized;

[0043] S56. Continue to process the list of data to be regularized, convert part of the data therein into a Torch tensor, fill it with all 0 data if its dimension is smaller than the preset size of the model input, and remove the data if its dimension is larger than the preset size of the model input, so as to ensure that the data input to the model have the same dimension.

[0044] S57, merging the label lists of each agent in the data list to be regularized into a unified label list, removing duplicate object IDs therein, so as to generate labels in subsequent training or testing processes.

[0045] To further illustrate, the step S6 is specifically as follows:

[0046] S61. Establish a multi-agent asynchronous perception training process for vehicle-road collaboration. First, input data into the collaborative perception model in the vehicle-road collaboration module (used for target classification, positioning and tracking in perception tasks, etc. The simulation method and system proposed in the present invention can be connected to the currently commonly used collaborative perception model, so that the performance of each model can be compared), and obtain the output classification result and bounding box position; then, find the position of each object in the Carla server at the current moment according to the unified label list, so as to obtain the training label. In this way, the real-time processing capability of the model can be included in the evaluation scope; finally, calculate the loss function according to the model output and the training label, and perform back propagation to train the model;

[0047] S62. During the training process, setting a trigger condition for the end of the training, such as when the agent reaches the end point or reaches a specified time;

[0048] S63. After the training is completed, the collaborative perception model obtained in this round of training is saved, and the unified signal system closes all processes except the main process and destroys all objects in the Carla world. If the training rounds do not reach the preset rounds, the main process starts a new round of training, loads the Carla world from scratch and repeats all the above processes;

[0049] S64. Establish a multi-agent asynchronous perception test process for vehicle-road collaboration. Unlike the training process, the collaborative perception model is set to evaluation mode during the test, and it no longer calculates gradients. In order to conduct a comprehensive algorithm evaluation, four types of evaluation indicators are set in the test process, namely self-perception evaluation, collaborative perception evaluation, absolute range evaluation, and full map evaluation. Among them, the self-perception evaluation only includes objects within the self-perception range of the agent in the evaluation, the collaborative perception evaluation includes objects within the perception range of the agent itself and its collaborative agents in the evaluation, the absolute range evaluation includes all objects within a larger range around the agent according to preset parameters, and the full map evaluation includes all objects on the map in the evaluation. This method can better reflect the detection performance and beyond-visual-range performance of the algorithm;

[0050] S65. In order to better compare the performance of various perception algorithms under the same conditions, the test process is equipped with two modes. The first mode is the asynchronous free mode, in which each Carla object can move freely and the Carla world runs in asynchronous mode. This mode can simulate the real situation to the greatest extent and the world flow speed is faster, but in this mode it cannot guarantee the same world changes in each test; the second mode is the fixed world mode. In this mode, the test process in the asynchronous free mode must be run once, and then the position of the Carla object at each moment is recorded with a recorder and saved as a .csv file. After that, a fixed world can be run based on this file. The position of each Carla object can be accurately restored, and sensors can be added and sensed normally. In this mode, the performance of different perception algorithms can be quantitatively compared.

[0051] Based on the above-mentioned asynchronous simulation method, the present invention also proposes a multi-agent asynchronous perception system for vehicle-road collaboration, including traffic participants, wherein the traffic participants include multiple vehicle-road collaboration agents and other traffic participants, and multi-source heterogeneous sensors are constructed on the vehicle-road collaboration agent to realize asynchronous perception; the above-mentioned data processing module is set on the vehicle-road collaboration agent to realize the processing and sharing of perception data; the above-mentioned vehicle-road collaboration module is set on the vehicle-road collaboration agent to realize cross-agent data fusion.

[0052] Beneficial effects of the present invention:

[0053] 1. The present invention adopts an asynchronous mode for simulation, and each intelligent agent can independently perform real-time perception, communication and collaboration, thereby simulating the real road environment more accurately;

[0054] 2. The present invention adopts the method of external hardware equipment and network port forwarding, which can simulate different communication devices by limiting the port bandwidth, thus being closer to the actual communication situation;

[0055] 3. The present invention adopts an integrated training and testing solution, which can access a variety of vehicle-road cooperative algorithms and perform real-time training in an asynchronous environment, improving the ability of the model to migrate to the real environment and making it more practical;

[0056] 4. The present invention can set up a variety of vehicle-road collaborative simulation scenarios, including various traffic environments, communication bandwidths, roadside and vehicle-side equipment, etc., and can repeatedly verify and quantitatively compare various algorithms to improve the development efficiency of vehicle-road collaborative algorithms;

[0057] 5. This invention provides new research ideas and experimental basis for the further development of intelligent connected vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0059] Figure 2 It is a schematic diagram of the specific operation process of each part of the present invention;

[0060] Figure 3 It is a schematic diagram of the vehicle-road collaboration process of the present invention;

[0061] Figure 4 A schematic diagram of perception of the present invention in a vehicle-road collaboration scenario;

[0062] Figure 5 A schematic diagram of target detection in a vehicle-road collaboration scenario according to the present invention; DETAILED DESCRIPTION

[0063] The present invention will be described in detail below in conjunction with the various embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0064] like Figure 1 As shown, the present invention is a multi-agent asynchronous perception simulation method and system for vehicle-road collaboration, comprising the following steps:

[0065] S1. Set the detailed parameters of the vehicle-road collaborative simulation scenario and start the asynchronous world flow;

[0066] S2, create multiple vehicle-road cooperative agents and ordinary traffic participants, and plan traffic behaviors;

[0067] S3: Build multi-source heterogeneous sensors on the vehicle-road cooperative agent to achieve asynchronous perception;

[0068] S4. Create a data processing module on the vehicle-road cooperative agent to realize the processing and sharing of perception data;

[0069] S5. Create a vehicle-road collaboration module on the vehicle-road collaboration agent to achieve cross-agent data fusion;

[0070] S6. Establish an algorithm training and testing process suitable for this simulation method to achieve integrated training and testing.

[0071] The specific operation process of each part of the simulation method and system is as follows: Figure 2 shown.

[0072] To further illustrate, the step S1 is specifically as follows:

[0073] Set the detailed parameters of the vehicle-road collaborative simulation scenario and start the asynchronous world flow;

[0074] S11, initializing a map model in the autonomous driving simulation platform Carla, and storing it as a file, so as to facilitate direct loading of the corresponding map later;

[0075] S12, reading and parsing the configuration file, loading the preset configuration parameters, creating the simulation world and the traffic manager according to the preset configuration parameters, and switching their operation mode to the asynchronous mode;

[0076] S13, asynchronously create a unified signal system, configure a communication publisher, and use the communication main port to uniformly send a start signal and an end signal to control the start and stop of the simulation environment;

[0077] To further illustrate, the step S2 is specifically as follows:

[0078] Create multiple vehicle-road cooperative agents and ordinary traffic participants to plan traffic behavior;

[0079] S21, reading the agent information in the preset configuration parameters, and creating a corresponding number of agent instance modules according to the preset number of agents, each of which is an asynchronous process;

[0080] S22. Connect the Carla server in each agent instance module, read the map information, and generate an agent instance at the corresponding position on the map according to the preset road position and type of the current agent. When the agent type is a car, if its preset mode is automatic planning, it will be connected to the traffic manager, and Carla will automatically plan its behavior path; if its preset mode specifies the starting point and the end point, a path planner for the agent will be created, and local waypoints will be output through the control algorithm and perception results to achieve basic automatic driving under the specified path;

[0081] S23. After all agent instances are generated, the number, type and road position of ordinary traffic participants are obtained using random functions in the main process. Based on this, ordinary traffic participant instances are created at corresponding positions on the map and connected to the traffic manager. Carla automatically plans their behavior paths. If other instances already exist at the random position, create instances at other idle road positions.

[0082] S24, all agent instances and common traffic participant instances are connected to the unified signal system through the communication main port, and each instance object is destroyed by the unified signal system after the simulation ends;

[0083] To further illustrate, the step S3 is specifically as follows:

[0084] Build multi-source heterogeneous sensors on the vehicle-road cooperative agent to achieve asynchronous perception;

[0085] S31, creating a shared memory Shared_mem_origin and a sensor management module in each agent instance module;

[0086] S32, in the sensor management module, based on the sensor type, location and attributes preset in the current agent configuration parameters, create multi-source heterogeneous sensors and bind them to the current agent instance;

[0087] S33, setting the callback function of the sensor, serializing the sensor's perception data into a byte stream, and writing it into the shared memory Shared_mem_origin for use by subsequent data processing modules;

[0088] To further illustrate, the step S4 is specifically as follows:

[0089] Create a data processing module on the vehicle-road cooperative agent to realize the processing and sharing of perception data;

[0090] S41. Add external hardware devices to realize network data forwarding and limit the bandwidth of each port used by the vehicle-road collaboration to simulate the vehicle-road collaboration communication process in a real environment.

[0091] S42, creating a data processing module in each agent instance module, which is a new asynchronous process. In the data processing module, create a shared memory Shared_mem_process and a ZeroMQ communication channel, wherein the ZeroMQ communication channel is connected to an external hardware device through an IP address;

[0092] S43, read the byte stream data in the shared memory Shared_mem_origin, and deserialize it to obtain the sensor's perception data. Since the data processing module is an asynchronous process relative to the sensor management module, a mutex mechanism is introduced to isolate the operations of the two on the shared memory Shared_mem_origin to prevent the shared memory data from being confused;

[0093] S44, obtaining the traffic participants within the perception radius of the current intelligent agent through the Carla server, including other intelligent agents and ordinary traffic participants. If only fuzzy labels are required in the preset configuration parameters, the object IDs of these traffic participants in the Carla world are directly stored in the label list; if precise labels are required in the preset configuration parameters, the bounding box of each traffic participant is first obtained, and then it is determined whether the laser radar point cloud is projected into the bounding box, so as to screen out the perceived traffic participants and store their object IDs in the Carla world in the label list;

[0094] S45, the vehicle-road cooperative fusion mode includes four types: no fusion, early fusion, mid-term fusion and late fusion. In the data processing module, the sensor perception data is processed according to the preset fusion mode. In the no fusion mode, the data processing module stores the sensor perception data, posture, timestamp and label list of the current intelligent body in the data dictionary, and serializes it into a byte stream, which is only written to the shared memory Shared_mem_process;

[0095] S46. In the early fusion mode, the data processing module first stores the sensor perception data, position, timestamp and label list of the current agent in the data dictionary, and serializes it into a byte stream and writes it into the shared memory Shared_mem_process. At the same time, the data dictionary is published to the ZeroMQ communication channel, and other agents can obtain the data sent by the current agent by subscription;

[0096] S47. In the mid-term fusion mode, the data processing module first pre-processes the data according to the sensor data and posture of the current intelligent agent, voxelizes the lidar point cloud, and obtains voxel features, voxel coordinates and voxel points; then, the data processing module further processes the voxel features based on the detection algorithm. In the currently commonly used vehicle-road cooperative algorithm, after obtaining the voxel features, PillarVFE, PointPillarScatter and Backbone are usually used one by one to process them, and different convolution operations are performed according to the detection algorithm to obtain intermediate features; finally, the vehicle-road cooperative module stores the intermediate features, posture, timestamp and label list in the data dictionary, and serializes them into a byte stream and writes them into the shared memory Shared_mem_process. At the same time, the data dictionary is published to the ZeroMQ communication channel, and other intelligent agents can obtain the data sent by the current intelligent agent through the subscription method;

[0097] S48. In the late fusion mode, the data processing module further processes the intermediate features obtained by the mid-term fusion according to the detection algorithm, and outputs the detection results of the object, such as object classification, bounding box position, etc.; then, the detection results, posture, timestamp and label list are stored in the data dictionary, and serialized into a byte stream and written into the shared memory shared_mem_process. At the same time, the data dictionary is also published to the ZeroMQ communication channel, and other agents can obtain the data sent by the current agent by subscription;

[0098] S49, the processing and sharing of perception data by the data processing module provides a data basis for the subsequent vehicle-road cooperative module. Since the data processing module is an asynchronous process relative to its superior intelligent agent instance module, its data processing process will not affect the real-time perception of the sensor;

[0099] To further illustrate, the step S5 is specifically as follows:

[0100] Create a vehicle-road collaboration module on the vehicle-road collaboration agent to achieve cross-agent data fusion. The schematic diagram of the vehicle-road collaboration process is shown in the figure. Figure 3 As shown;

[0101] S51, reading the agent information in the preset configuration parameters, and creating a corresponding number of vehicle-road cooperative modules according to the preset number of agents, each of which is an asynchronous process;

[0102] S52. In the vehicle-road collaboration module, the current agent connects to the shared memory shared_mem_process created in its data processing module and subscribes to the ZeroMQ communication channel of other agents.

[0103] S53, the vehicle-road collaboration module scans the shared memory shared_mem_process when idle, and deserializes the byte stream data therein to restore the data dictionary. If the timestamp in the data dictionary has changed relative to the timestamp of the previous data, it indicates that the current agent has generated new data and the next step of processing is performed; if the timestamp has not changed, the scan continues. Since the vehicle-road collaboration module is an asynchronous process relative to the data processing module, a mutex mechanism is introduced to isolate the operations of the two on the shared memory Shared_mem_process to prevent confusion of shared memory data;

[0104] S54, after obtaining the new data of the current intelligent agent, the vehicle-road collaboration module obtains metadata in the ZeroMQ communication channel of other intelligent agents, and makes distance judgment based on the posture data. If it is judged that both parties are within the preset collaboration range, the complete data is received; if not, the data is skipped;

[0105] S55: If the vehicle-road collaboration module obtains complete data of other intelligent agents, it loads the data together with the data of the current intelligent agent into the list of data to be regularized; if the data of other intelligent agents are all out of the collaboration scope, it loads the data of the current intelligent agent alone into the list of data to be regularized;

[0106] S56. Continue to process the list of data to be regularized, convert part of the data therein into a Torch tensor, fill it with all 0 data if its dimension is smaller than the preset size of the model input, and remove the data if its dimension is larger than the preset size of the model input, so as to ensure that the data input to the model have the same dimension.

[0107] S57, merging the label lists of each agent in the data list to be regularized into a unified label list, removing duplicate object IDs therein, so as to generate labels in subsequent training or testing processes;

[0108] To further illustrate, the step S6 is specifically as follows:

[0109] Establish an algorithm training and testing process suitable for this simulation method to achieve integrated training and testing;

[0110] S61. Establish a multi-agent asynchronous perception training process for vehicle-road collaboration. First, input data into the collaborative perception model in the vehicle-road collaboration module to obtain the output classification results and bounding box positions. Then, find the positions of each object in the Carla server at the current moment according to the unified label list to obtain training labels. This method can include the real-time processing capability of the model in the evaluation scope. Finally, calculate the loss function according to the model output and the training labels, and perform back propagation to train the model.

[0111] S62. During the training process, setting a trigger condition for the end of the training, such as when the agent reaches the end point or reaches a specified time;

[0112] S63. After the training is completed, the collaborative perception model obtained in this round of training is saved, and the unified signal system closes all processes except the main process and destroys all objects in the Carla world. If the training rounds do not reach the preset rounds, the main process starts a new round of training, loads the Carla world from scratch and repeats all the above processes;

[0113] S64. Establish a multi-agent asynchronous perception test process for vehicle-road collaboration. Unlike the training process, the collaborative perception model is set to evaluation mode during the test, and it no longer calculates gradients. In order to conduct a comprehensive algorithm evaluation, four types of evaluation indicators are set in the test process, namely self-perception evaluation, collaborative perception evaluation, absolute range evaluation, and full map evaluation. Among them, the self-perception evaluation only includes objects within the self-perception range of the agent in the evaluation, the collaborative perception evaluation includes objects within the perception range of the agent itself and its collaborative agents in the evaluation, the absolute range evaluation includes all objects within a larger range around the agent according to preset parameters, and the full map evaluation includes all objects on the map in the evaluation. This method can better reflect the detection performance and beyond-visual-range performance of the algorithm;

[0114] S65. In order to better compare the performance of various perception algorithms under the same conditions, the test process is equipped with two modes. The first mode is the asynchronous free mode, in which each Carla object can move freely and the Carla world runs in asynchronous mode. This mode can simulate the real situation to the greatest extent and the world flow speed is faster, but in this mode it cannot guarantee the same world changes in each test; the second mode is the fixed world mode. In this mode, the test process in the asynchronous free mode must be run once, and then the position of the Carla object at each moment is recorded with a recorder and saved as a .csv file. After that, a fixed world can be run based on this file. The position of each Carla object can be accurately restored, and sensors can be added and sensed normally. In this mode, the performance of different perception algorithms can be quantitatively compared.

[0115] Based on the above asynchronous simulation method, a multi-agent asynchronous perception system for vehicle-road collaboration is established, including traffic participants, including multiple vehicle-road collaboration agents and other traffic participants. Multi-source heterogeneous sensors are constructed on the vehicle-road collaboration agent to realize asynchronous perception; the above data processing module is set on the vehicle-road collaboration agent to realize the processing and sharing of perception data; the above vehicle-road collaboration module is set on the vehicle-road collaboration agent to realize cross-agent data fusion. The multi-agent asynchronous perception system can visualize the perception and target detection process, thereby facilitating the development of perception algorithms. The visualization example is as follows: Figure 4 and Figure 5 shown.

[0116] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-agent asynchronous perception simulation method for vehicle-road collaboration, characterized in that: These include: S1. Set the vehicle-road collaborative simulation scenario parameters and start the asynchronous world flow; S2, create multiple vehicle-road cooperative agents and other traffic participants to plan traffic behavior; S3: Build multi-source heterogeneous sensors on the vehicle-road cooperative agent to achieve asynchronous perception; S4. Create a data processing module on the vehicle-road cooperative agent to realize the processing and sharing of perception data; S5. Create a vehicle-road collaboration module on the vehicle-road collaboration agent to achieve cross-agent data fusion; S6. Establish an algorithm training and testing process suitable for simulation to achieve integrated training and testing.

2. The asynchronous perception simulation method according to claim 1, characterized in that: The S1 specifically includes: S11, initializing a map model in the autonomous driving simulation platform Carla, and storing it as a file, so as to facilitate direct loading of the corresponding map later; S12, reading and parsing the configuration file, loading the preset configuration parameters, creating the simulation world and the traffic manager according to the preset configuration parameters, and switching their operation mode to the asynchronous mode; S13. Asynchronously create a unified signal system, configure a communication publisher, and use the communication main port to uniformly send a start signal and an end signal to control the start and stop of the simulation environment.

3. The asynchronous perception simulation method according to claim 1, characterized in that: The S2 specifically includes: S21, reading the agent information in the preset configuration parameters, and creating a corresponding number of agent instance modules according to the preset number of agents, each of which is an asynchronous process; S22, connect the Carla server in each agent instance module, read the map information, and generate an agent instance at the corresponding position of the map according to the preset road position and type of the current agent; when the agent type is a car, if its preset mode is automatic planning, connect it to the traffic manager, and let Carla automatically plan its behavior path; if its preset mode specifies the starting point and the end point, create a path planner for the agent, output local waypoints through the control algorithm and perception results, and realize basic automatic driving under the specified path; S23. After all agent instances are generated, the number, type and road position of ordinary traffic participants are obtained using random functions in the main process. Based on this, ordinary traffic participant instances are created at corresponding positions on the map and connected to the traffic manager. Carla automatically plans their behavior paths. If other instances already exist at the random position, create instances at other idle road positions. S24. All intelligent agent instances and common traffic participant instances are connected to the unified signal system through the communication main port, and each instance object is destroyed by the unified signal system after the simulation is completed.

4. The asynchronous perception simulation method according to claim 1, characterized in that: The S3 specifically includes: S31, creating a shared memory Shared_mem_origin and a sensor management module in each agent instance module; S32, in the sensor management module, based on the sensor type, location and attributes preset in the current agent configuration parameters, create multi-source heterogeneous sensors and bind them to the current agent instance; S33, setting the callback function of the sensor, serializing the sensor's perception data into a byte stream, and writing it into the shared memory Shared_mem_origin for use by subsequent data processing modules.

5. The asynchronous perception simulation method according to claim 1, characterized in that: The S4 specifically includes: S41. Add external hardware devices to realize network data forwarding and limit the bandwidth of each port used by the vehicle-road collaboration to simulate the vehicle-road collaboration communication process in a real environment. S42, creating a data processing module in each agent instance module, which is a new asynchronous process; in the data processing module, creating a shared memory Shared_mem_process and a ZeroMQ communication channel, wherein the ZeroMQ communication channel is connected to an external hardware device through an IP address; S43, read the byte stream data in the shared memory Shared_mem_origin, and deserialize it to obtain the sensor perception data; since the data processing module is an asynchronous process relative to the sensor management module, a mutex mechanism is introduced to isolate the operations of the two on the shared memory Shared_mem_origin to prevent the shared memory data from being confused; S44, obtaining the traffic participants within the perception radius of the current intelligent agent through the Carla server, including other intelligent agents and ordinary traffic participants. If only fuzzy labels are required in the preset configuration parameters, the object IDs of these traffic participants in the Carla world are directly stored in the label list; if precise labels are required in the preset configuration parameters, the bounding box of each traffic participant is first obtained, and then it is determined whether the laser radar point cloud is projected into the bounding box, so as to screen out the perceived traffic participants and store their object IDs in the Carla world in the label list; S45. The vehicle-road collaborative fusion mode includes four categories: no fusion, early fusion, mid-term fusion and late fusion. In the data processing module, the sensor's perception data is processed according to the preset fusion mode.

6. The asynchronous perception simulation method according to claim 5, characterized in that: The S45 specifically includes: In the non-fusion mode, the data processing module stores the current agent's sensor perception data, position, timestamp, and tag list in the data dictionary, serializes them into a byte stream, and writes them into the shared memory Shared_mem_process; In the early fusion mode, the data processing module first stores the sensor perception data, position, timestamp and label list of the current agent in the data dictionary, and serializes it into a byte stream and writes it into the shared memory Shared_mem_process. At the same time, the data dictionary is published to the ZeroMQ communication channel, and other agents can obtain the data sent by the current agent by subscription. In the mid-term fusion mode, the data processing module first pre-processes the data according to the sensor data and posture of the current intelligent agent, voxelizes the lidar point cloud, and obtains voxel features, voxel coordinates, and voxel points; then, the data processing module further processes the voxel features based on the detection algorithm. In the currently commonly used vehicle-road cooperative algorithm, after obtaining the voxel features, PillarVFE, PointPillarScatter, and Backbone are usually used one by one to process them, and different convolution operations are performed according to the detection algorithm to obtain intermediate features; finally, the vehicle-road cooperative module stores the intermediate features, posture, timestamp, and label list in the data dictionary, and serializes them into a byte stream and writes them into the shared memory Shared_mem_process. At the same time, the data dictionary is published to the ZeroMQ communication channel, and other intelligent agents can obtain the data sent by the current intelligent agent through the subscription method; In the late fusion mode, the data processing module further processes the intermediate features obtained by the mid-term fusion according to the detection algorithm, and outputs the detection results of the object, such as object classification, bounding box position, etc.; then, the detection results, posture, timestamp and label list are stored in the data dictionary, and serialized into a byte stream and written into the shared memory Shared_mem_process. At the same time, the data dictionary is published to the ZeroMQ communication channel, and other intelligent agents can obtain the data sent by the current intelligent agent by subscription.

7. The asynchronous perception simulation method according to claim 6, characterized in that: The S5 specifically includes: S51, reading the agent information in the preset configuration parameters, and creating a corresponding number of vehicle-road cooperative modules according to the preset number of agents, each of which is an asynchronous process; S52. In the vehicle-road collaboration module, the current agent connects to the shared memory Shared_mem_process created in its data processing module and subscribes to the ZeroMQ communication channel of other agents. S53, the vehicle-road collaboration module scans the shared memory Shared_mem_process when idle, and deserializes the byte stream data therein to restore the data dictionary; if the timestamp in the data dictionary has changed relative to the timestamp of the previous data, it indicates that the current agent has generated new data, and the next step of processing is carried out; if the timestamp has not changed, the scanning continues. The vehicle-road collaboration module is an asynchronous process relative to the data processing module, so a mutex mechanism is introduced to isolate the operations of the two on the shared memory Shared_mem_process to prevent confusion of shared memory data; S54, after obtaining the new data of the current intelligent agent, the vehicle-road collaboration module obtains metadata in the ZeroMQ communication channel of other intelligent agents, and makes distance judgment based on the posture data. If it is judged that both parties are within the preset collaboration range, the complete data is received; if not, the data is skipped; S55: If the vehicle-road collaboration module obtains complete data of other intelligent agents, it loads the data together with the data of the current intelligent agent into the list of data to be regularized; if the data of other intelligent agents are all out of the collaboration scope, it loads the data of the current intelligent agent alone into the list of data to be regularized; S56. Continue to process the list of data to be regularized, convert part of the data therein into a Torch tensor, fill it with all 0 data if its dimension is smaller than the preset size of the model input, and remove the data if its dimension is larger than the preset size of the model input, so as to ensure that the data input to the model have the same dimension. S57, merging the label lists of each agent in the data list to be regularized into a unified label list, removing duplicate object IDs therein, so as to generate labels in subsequent training or testing processes.

8. The asynchronous perception simulation method according to claim 1, characterized in that: The training of S6 specifically includes the following: S61. Establish a multi-agent asynchronous perception training process for vehicle-road collaboration. First, input data into the collaborative perception model in the vehicle-road collaboration module to obtain the output classification results and bounding box positions. Then, find the positions of each object in the Carla server at the current moment according to the unified label list to obtain training labels, and include the real-time processing capability of the model in the evaluation scope. Finally, calculate the loss function according to the model output and the training labels, and perform back propagation to train the model. S62. During the training process, setting a trigger condition for the end of training, such as when the agent reaches the end point or reaches a specified time; S63. After the training is completed, the collaborative perception model obtained in this round of training is saved, and the unified signal system closes all processes except the main process and destroys all objects in the Carla world. If the training rounds do not reach the preset rounds, the main process starts a new round of training, loads the Carla world from scratch and repeats the above process.

9. The asynchronous perception simulation method according to claim 1, characterized in that: The S6 test specifically includes the following: S64. During the test, the collaborative perception model is set to evaluation mode, and the gradient is no longer calculated. In order to conduct a comprehensive algorithm evaluation, four types of evaluation indicators are set during the test, including self-perception evaluation, collaborative perception evaluation, absolute range evaluation, and full map evaluation. Among them, the self-perception evaluation only includes objects within the self-perception range of the agent, the collaborative perception evaluation includes objects within the perception range of the agent itself and its collaborative agents, the absolute range evaluation includes all objects within a larger range around the agent according to preset parameters, and the full map evaluation includes all objects on the map; S65. Two modes are set in the test process. The first mode is the asynchronous free mode. Each Carla object can move freely, and the Carla world runs in asynchronous mode. This mode can simulate the real situation to the greatest extent, and the world flow speed is faster. The second mode is the fixed world mode. This mode first needs to run the test process in the asynchronous free mode, and then use the recorder to record the position of the Carla object at each moment and save it as a .csv file. After that, the fixed world can be run based on this file. The position of each Carla object can be accurately restored, and sensors can be added and sensed normally.

10. A multi-agent asynchronous perception system for vehicle-road collaboration, comprising traffic participants, wherein the traffic participants include multiple vehicle-road collaboration agents and other traffic participants, characterized in that: Multi-source heterogeneous sensors are constructed on the vehicle-road cooperative intelligent body to realize asynchronous perception; the data processing module described in any one of claims 1-9 is set on the vehicle-road cooperative intelligent body to realize the processing and sharing of perception data; the vehicle-road cooperative module described in any one of claims 1-9 is set on the vehicle-road cooperative intelligent body to realize cross-agent data fusion.

Images