A method and system for developing an offline multi-agent intelligent driving dataset

Abstract

The application discloses a kind of off-line multi-agent intelligent driving data set development method and system, it is related to the field of intelligent driving assistance, the method comprises: the first multi-agent real interaction data is preprocessed to obtain the second multi-agent real interaction data, and the original panoramic traffic scene representation corresponding to the second multi-agent real interaction data is extracted by traffic scene extraction model;From the autonomous driving simulation platform, the first multi-agent simulation interaction data corresponding to the target traffic interaction event is obtained;Based on conditional domain adversarial generative network, with real data feature space as benchmark, the first multi-agent simulation interaction data is domain self-adaptive enhanced to generate the second multi-agent simulation interaction data, and the domain gap between simulation data and real data is reduced;Through data encapsulation, standardized intelligent driving data set is generated.Therefore, high-quality intelligent driving data set with uniform structure, cross-domain alignment, multi-agent collaborative perception and decision-making is constructed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving assistance, and in particular to a method and system for developing an offline multi-agent intelligent driving dataset. BACKGROUND

[0002] The development of intelligent driving technology cannot be achieved without the support of high-quality datasets. Currently, commonly used intelligent driving datasets (such as KITTI, Waymo, etc.) mainly rely on online data recording and a single vehicle perspective during the collection process, making it difficult to fully depict the panoramic scene of multi-vehicle, multi-pedestrian, and multi-modal information interaction in complex traffic environments.

[0003] Specifically, existing intelligent driving datasets are mainly designed for single-vehicle intelligence, focusing more on the perception and decision-making capabilities of a single vehicle, while ignoring the needs of multi-agent collaborative driving scenarios. In real traffic environments, vehicles need to interact with other vehicles, pedestrians, bicycles, and even infrastructure in complex ways. For example, when merging on a highway, vehicles need to adjust their behavior based on the speed and position of surrounding vehicles; at urban intersections, vehicles may need to yield to pedestrians or compete for right-of-way with other vehicles. These multi-agent interaction behaviors pose higher requirements on autonomous driving algorithms, while existing datasets have obvious deficiencies in coverage and quality in this regard, resulting in the system's inability to fully understand the dynamic coordination relationship between each agent in complex traffic interactions.

[0004] On the other hand, although online data collection can record real road environments in real time, it is also subject to geographical location, time, and environmental factors, making it difficult to fully record and verify some extreme case boundary examples. In addition, online data collection also faces safety issues and legal restrictions. For example, collecting data in complex traffic environments may increase the risk of accidents, and in some countries or regions, publicly releasing traffic data may violate privacy protection regulations. SUMMARY

[0005] The present application provides a method and system for developing an offline multi-agent intelligent driving dataset, a storage medium, a computer program product, and an electronic device to at least solve the problems of the current related technology in depicting complex traffic scenarios and multi-agent interactions, and the limitations of online data collection.

[0006] In a first aspect, the embodiments of the present application provide a method for developing an offline multi-agent intelligent driving dataset, comprising: performing a preprocessing operation on first multi-agent real interaction data to obtain second multi-agent real interaction data, and extracting an original panoramic traffic scene representation corresponding to the second multi-agent real interaction data through a traffic scene extraction model; the first multi-agent real interaction data includes parameter data collected from a real traffic scene by a multi-modal sensor group and a V2X communication device deployed on a test vehicle; the traffic scene extraction model is based on a graph convolutional neural network as a model backbone; determining a target traffic interaction event corresponding to the original panoramic traffic scene representation, and obtaining first multi-agent simulation interaction data corresponding to the target traffic interaction event from an autonomous driving simulation platform; based on a conditional domain adversarial generation network, taking a real data feature space as a benchmark, performing domain self-adaptive enhancement on the first multi-agent simulation interaction data to generate second multi-agent simulation interaction data, thereby narrowing the domain gap between simulation data and real data; the conditional information of the conditional domain adversarial generation network is defined according to the target traffic interaction event and the original panoramic traffic scene representation; based on the target traffic interaction event, the corresponding second multi-agent simulation interaction data and the second multi-agent real interaction data are packaged to generate a standardized intelligent driving dataset.

[0007] In a second aspect, the embodiments of the present application provide a system for developing an offline multi-agent intelligent driving dataset, comprising: a real information acquisition unit configured to perform a preprocessing operation on first multi-agent real interaction data to obtain second multi-agent real interaction data, and extract an original panoramic traffic scene representation corresponding to the second multi-agent real interaction data through a traffic scene extraction model; the first multi-agent real interaction data includes parameter data collected from a real traffic scene by a multi-modal sensor group and a V2X communication device deployed on a test vehicle; the traffic scene extraction model is based on a graph convolutional neural network as a model backbone; an event simulation matching unit configured to determine a target traffic interaction event corresponding to the original panoramic traffic scene representation, and obtain first multi-agent simulation interaction data corresponding to the target traffic interaction event from an autonomous driving simulation platform; a domain self-adaptive enhancement unit configured to, based on a conditional domain adversarial generation network, take a real data feature space as a benchmark, perform domain self-adaptive enhancement on the first multi-agent simulation interaction data to generate second multi-agent simulation interaction data, thereby narrowing the domain gap between simulation data and real data; the conditional information of the conditional domain adversarial generation network is defined according to the target traffic interaction event and the original panoramic traffic scene representation; a data packaging unit configured to, based on the target traffic interaction event, package the corresponding second multi-agent simulation interaction data and the second multi-agent real interaction data to generate a standardized intelligent driving dataset.

[0008] In a third aspect, an electronic device is provided, which includes at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method for developing an offline multi-agent intelligent driving data set according to any of the embodiments of the present application.

[0009] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the method for developing an offline multi-agent intelligent driving data set according to any of the embodiments of the present application.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the steps of the method for developing an offline multi-agent intelligent driving data set according to any of the embodiments of the present application.

[0011] The method and system for developing an offline multi-agent intelligent driving data set provided by the present application can at least achieve the following technical effects:

[0012] (1) By structuring modeling and multi-modal information fusion of multi-agent real traffic interaction data, the spatio-temporal interaction relationship between agents in a complex traffic scene can be effectively restored and expressed. With the support of a graph convolutional neural network, a traffic scene extraction model can extract an original panoramic traffic scene representation with a structured relationship from preprocessed data, and then efficiently model the correlation between traffic behavior patterns and traffic elements. On this basis, the scene representation is further identified to target traffic interaction events, and traffic interaction events are introduced as a semantic driving bridge, not only establishing a logical mapping relationship from real data to simulation data, but also providing accurate target guidance for data augmentation and generation.

[0013] (2) The first multi-agent simulation interaction data is subjected to domain self-adaptive enhancement processing by using a conditional domain adversarial generation network, and the conditional information used not only includes target traffic interaction events, but also further introduces an original panoramic traffic scene representation, so that the generation network can consider the information features of the semantic layer and the structure layer during the simulation data style migration process. By using a joint conditional definition method, the restoration degree and credibility of the enhanced simulation data in terms of spatial organization, agent behavior logic and scene consistency are significantly improved, effectively reducing the domain difference between virtual and real data. On this basis, the enhanced simulation interaction data and the real interaction data are standardized and packaged based on unified event labels, and a high-quality intelligent driving data set with unified structure, cross-domain alignment, and multi-agent collaborative perception and decision-making is constructed.

[0014] By means of the technical solution, relying on the offline processing and simulation platform construction mechanism, the dependence of data development on actual road collection is reduced, the efficiency and safety of data development are improved, and the developed multi-agent intelligent driving data set has rich traffic interaction semantic expression ability and higher data diversity, thereby providing a stable and scalable data basis for training and evaluation of collaborative automatic driving algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0016] Figure 1 A flowchart of an example of a method for developing an offline multi-agent intelligent driving data set according to an embodiment of the present application is shown;

[0017] Figure 2 A structural connection diagram of an example of a traffic scene extraction model according to an embodiment of the present application is shown;

[0018] Figure 3 A structural connection diagram of an example of a conditional domain generative adversarial network according to an embodiment of the present application is shown;

[0019] Figure 4 A structural block diagram of an example of a development system of an offline multi-agent intelligent driving data set according to an embodiment of the present application is shown;

[0020] Figure 5 A structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0021] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0022] It should be noted that some current automatic driving simulation platforms (such as CARLA, LGSVL, SUMO, etc.) can generate various automatic driving related data. In the current related technology, some experts and scholars propose to use the data generated by the above automatic driving simulation platforms to construct a multi-agent simulation data set of intelligent driving, so as to realize the simulation of the interaction between vehicles, pedestrians, cyclists and other multi-agents.

[0023] However, there are still some obvious deficiencies in directly using the data generated by the automatic driving simulation platform to construct the multi-agent simulation data set: first, the virtual environment is difficult to perfectly reproduce the physical characteristics such as noise, light and weather of real sensors, resulting in "domain gap" between data and actual situation; second, the behavior of multi-agent is mostly dependent on preset rules or simplified models, which is difficult to capture the complex, nonlinear and uncertain decision-making process in real driving; third, in extreme or emergency scenarios, the details and dynamics of the simulation data are often insufficient.

[0024] It should be understood that the purpose of the above description of the current related technology is only to facilitate the public to better understand the spirit and motivation of the application, and is not considered as a limitation of the application. In addition, the technical solutions described in the above current related technology are not prior art, and can also be undisclosed technical solutions, such as solutions under research or in the laboratory stage.

[0025] In the technical solutions of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information, etc. comply with the relevant laws and regulations, and do not violate public order and good customs.

[0026] Figure 1 A flowchart showing an example of a method for developing an offline multi-agent intelligent driving data set according to an embodiment of the present application is shown.

[0027] Regarding the execution subject of the method of the present application, it can be any controller or processor with computing or processing capability. Specifically, it can be implemented by an automatic driving data set construction platform, which constructs a closed-loop multi-level data generation system, realizes efficient integration and cooperation of multi-agent information through multi-modal data preprocessing, panoramic traffic scene reconstruction driven by graph convolutional neural network and accurate positioning of target traffic interaction events.

[0028] In some examples, it can be integrated and configured in an electronic device or terminal by software, hardware or a combination of software and hardware, and the type of terminal or electronic device can be diversified, such as mobile phone, tablet computer or desktop computer, etc.

[0029] As Figure 1As shown, in step S110, a pretreatment operation is performed on the first multi-agent real interaction data to obtain second multi-agent real interaction data, and an original panoramic traffic scene representation corresponding to the second multi-agent real interaction data is extracted through a traffic scene extraction model.

[0030] Specifically, the first multi-agent real interaction data contains parameter data collected from a real traffic scene by a multi-modal sensor group and a V2X communication device deployed on a test vehicle; and the traffic scene extraction model is a graph convolutional neural network.

[0031] In some embodiments, a plurality of multi-source heterogeneous data acquired when a plurality of test vehicles run in a real road environment is first collected, including but not limited to data collected by a vehicle-mounted laser imaging detection and ranging (LiDAR), a millimeter wave radar, a camera, an inertial measurement unit (IMU), and a V2X communication module, which has time synchronization and multi-agent correlation. Subsequently, a series of pretreatment operations are performed on the data, including time alignment, space correction, coordinate transformation, agent label unification, and trajectory fusion, to eliminate abnormal values and construct a time and space continuous, entity identification clear data format, to obtain second multi-agent real interaction data.

[0032] It should be understood that V2X (Vehicle-to-Everything) is a vehicle networking technology, including V2V (Vehicle-to-Vehicle), V2I (Vehicle-to-Infrastructure), V2P (Vehicle-to-Pedestrian), and V2N (Vehicle-to-Network) communication modes, which is used to improve traffic safety and automatic driving capability.

[0033] It should be noted that the multi-modal sensor group mainly includes a camera, a laser radar, a millimeter wave radar, etc., which is responsible for directly sensing the physical environment around the vehicle, such as object detection, lane line identification, obstacle identification, etc., and can realize the collection of original visual, distance, speed, etc. Data. V2X communication equipment focuses on the exchange of information between vehicles and other traffic elements. Through wireless communication technology, V2X equipment can receive information from other vehicles or road infrastructure, such as real-time traffic conditions, cooperative driving information, and early warning information. Some of the information can be provided by other vehicles or traffic management systems. In this way, through the multi-modal sensor group for environmental perception, and based on the V2X communication equipment to capture more comprehensive information sharing, the combination of the two forms a more comprehensive traffic scene representation, providing stronger data support for autonomous driving. Therefore, the fusion and standardization of multi-modal heterogeneous perception data are realized, and the time synchronization and spatial consistency of the data are improved.

[0034] Further, a traffic scene extraction model with a graph convolutional network (GCN) as the main trunk is adopted to automatically extract the topological relationship and semantic interaction features between traffic subjects from the second multi-agent real interaction data, and a structured original panoramic traffic scene representation is constructed, which includes the position, speed, motion trend of each multi-agent and the relationship graph between them. Therefore, through graph structure modeling, the complex dynamic relationship between multi-agents is captured, the effective fusion of multi-vehicle, multi-pedestrian and other multi-agent information is realized, and the semantic complete panoramic scene information is obtained, so as to more comprehensively depict the multi-modal interaction in the complex traffic environment.

[0035] In step S120, the target traffic interaction event corresponding to the original panoramic traffic scene representation is determined, and the first multi-agent simulation interaction data corresponding to the target traffic interaction event is obtained from the autonomous driving simulation platform.

[0036] In one example of the embodiments of the present application, the system can specify the traffic interaction event corresponding to the real traffic scene by receiving user input information. In another example of the embodiments of the present application, the system can automatically identify the corresponding target traffic interaction event by calling a deep learning model to identify and classify the features of the original panoramic traffic scene representation, and both belong to the implementation scope of the embodiments of the present application.

[0037] On the other hand, in the automatic driving simulation platform, there is a multi-agent simulation interaction dataset pre-classified according to traffic interaction events, thereby supporting the calling of simulation data matched with the interaction events. Exemplarily, the automatic driving simulation platform (such as CARLA, SUMO, etc.) can generate a virtual traffic environment and multi-agent interaction behavior according to the definition of the target event, which meets the corresponding scene requirements. For example, in the scene of "yielding to pedestrians at intersections", the simulation platform can simulate the random crossing behavior of pedestrians and the avoidance strategy of vehicles.

[0038] It should be understood that the event type of the traffic interaction event can be diversified. In some examples of the embodiments of the present application, in combination with the actual needs of the multi-agent cooperative driving scene for the dataset, the traffic interaction event contains any one of the following: highway merging event, urban intersection conflict event, lane changing and overtaking event, following and distance adjustment event, and emergency collision avoidance and braking event.

[0039] Specifically, the highway merging event refers to when vehicles merge into the main lane from the entrance ramp, multiple vehicles need to be coordinated and adjusted according to their own speed, distance, and vehicle-to-vehicle communication information. The urban intersection conflict event refers to the complex scene of multiple traffic participants converging at the urban intersection, including straight, turning, pedestrian crossing, and cyclist crossing behaviors. The lane changing and overtaking event refers to when vehicles change lanes and overtake on urban or highway roads, the dynamic state of surrounding vehicles needs to be accurately evaluated, and the driving strategy needs to be adjusted in real time. This event often involves close interaction and cooperative decision-making between vehicles. The following and distance adjustment event refers to the process of following a vehicle, in which the vehicle adjusts the speed and distance dynamically to maintain a safe following state. The emergency collision avoidance and braking event refers to when a sudden obstacle or abnormal behavior (such as sudden stop, illegal lane change) occurs, the vehicle needs to make an emergency braking or avoidance decision quickly.

[0040] It should be emphasized that the above traffic interaction events not only cover single type of vehicle behavior, but also comprehensive embodiment of multiple traffic subjects interacting in complex environment. Each type of event reflects the dynamic interaction and cooperative behavior between subjects in the traffic scene from different angles, such as merging, lane changing, following, and emergency avoidance between vehicles, so that the dataset covers various key scenes from highway merging to urban intersection, from lane changing to emergency collision avoidance, providing rich scenes for training and verifying multi-agent cooperative decision-making.

[0041] In step S130, based on the conditional domain adversarial generation network, the first multi-agent simulation interaction data is domain-adaptively enhanced to generate the second multi-agent simulation interaction data based on the real data feature space as a benchmark, thereby narrowing the domain gap between the simulation data and the real data.

[0042] Here, the conditional information of the Conditional Domain Adversarial Network (CDAN) is defined according to the target traffic interaction event and the original panoramic traffic scene representation.

[0043] By adopting the Conditional Domain Adversarial Network, the feature distribution of real data is taken as the target domain to guide the domain adaptive enhancement of simulation data. Specifically, the first multi-agent simulation interaction data is input into the CDAN model, and the conditional information input is set according to the original panoramic traffic scene representation and the target traffic interaction event label. The distribution deviation of simulation data in a specific event and scene context is learned by the conditional discriminator, and the generator is trained to optimize the simulation data features so that they gradually approach the real data in the high-dimensional feature space. Finally, the second multi-agent simulation interaction data is output to realize the consistency alignment of data distribution. Thus, through domain adversarial training, the simulation data is closer to the real data in appearance and feature distribution; through the introduction of conditional information, the generated data can retain the details of key traffic interaction events, enhance the expression ability of data and the pertinence of application scenarios, realize fine-grained feature alignment based on scene semantics and interaction behavior, reduce the domain difference between simulation and real multi-agent data, ensure that the simulation data is not only "similar" to the real data, but also "usable", and improve the effectiveness and usability of simulation data.

[0044] In step S140, the corresponding second multi-agent simulation interaction data and second multi-agent real interaction data are encapsulated based on the target traffic interaction event to generate a standardized intelligent driving data set.

[0045] In some embodiments, the system combines and encapsulates the second multi-agent simulation interaction data and the second multi-agent real interaction data, indexes and classifies them according to the target traffic interaction event, and ensures the consistency of the data in the storage, reading and transmission process. In addition, a data interface for the automatic driving simulation and testing platform can be designed to ensure that the data set can be easily integrated into the existing training, verification and evaluation framework. Thus, the generated intelligent driving data set has a unified format and rich metadata information, and can facilitate data exchange and algorithm evaluation between different platforms.

[0046] Through the embodiments of the present application, by integrating real multi-agent interaction data (collected by multi-modal sensor groups and V2X devices) and simulation data, and using a graph convolutional neural network to extract a panoramic traffic scene representation, and then determining a target traffic interaction event according to the representation, and then using a conditional domain adversarial generation network to perform domain self-adaptive enhancement on the simulation data, the domain gap between the simulation data and the real data is effectively reduced, and by fusing the conditional information of the target event and the scene context, the usability of the enhanced simulation data in the corresponding traffic event scene is ensured. Finally, the enhanced simulation data and the real data are packaged into a standardized data set, providing high-quality, low-domain-gap and rich traffic interaction semantic training and verification data for intelligent driving systems, thereby significantly improving the generalization and robustness of the automatic driving algorithm in complex traffic scenes.

[0047] As for the implementation details of the preprocessing operation in step S110, in some examples of the embodiments of the present application, outlier detection and noise filtering are performed on each data source parameter in the first multi-agent real interaction data.

[0048] Specifically, for different data sources (such as cameras, lidar, millimeter wave radar, V2X communication data, GPS, vehicle control information, etc.) in the first multi-agent real interaction data, first, the expected parameter range and statistical distribution model of each data source are established. Statistical metadata, including sampling frequency, data format, environmental information at the time of collection, etc. In outlier detection, statistical indicators such as mean, variance, and box plot can be used to preliminarily analyze the data and detect abnormal data points that exceed the normal range. In noise filtering, methods such as median filtering, Kalman filtering, or Gaussian filtering can be used to smooth the signals of each data source, reducing high-frequency noise. In addition, filter parameters can also be adjusted according to data dynamics to ensure that noise can be effectively filtered out without losing effective information in different scenarios. As a result, after removing abnormal data and filtering noise, the quality of the original data output by each data source is significantly improved, reducing the cumulative effect of initial noise and outliers in the data processing chain.

[0049] On the other hand, each sensor parameter in the first multi-agent real interaction data is time-synchronized and calibrated in the coordinate system, so that each sensor parameter can be accurately aligned in a unified space-time reference system.

[0050] In some embodiments, the data of different sampling frequencies are time-aligned by linear interpolation or Kalman filtering, etc. using the timestamps of each sensor. Then, the data from different sensors and V2X devices are aligned to the same time axis by establishing a global clock reference, ensuring the consistency of data acquisition time. After time synchronization, all sensor data are consistent in time domain, ensuring the effective fusion of multi-modal data at the same time. In addition, the external parameters (rotation matrix and translation vector) of each sensor are calculated using a calibration board or known physical landmarks, and the data of each sensor are converted to a unified global coordinate system. The coordinate transformation matrix is further refined using the Iterative Closest Point (ICP) algorithm, ensuring the accurate overlap of different sensor data in space. Through coordinate system calibration, the data of all sensors are converted to a unified global reference frame, greatly improving the accuracy of spatial data and making the overall system more accurate and robust in capturing complex multi-agent interactions.

[0051] Figure 2 A structural connection diagram of an example of a traffic scene extraction model according to an embodiment of the present application is shown.

[0052] As shown in Figure 2 The traffic scene extraction model 200 includes a cascaded graph construction module 210, a graph convolution module 220, and a global pooling output layer 230.

[0053] The graph construction module 210 is used to construct an initial traffic scene graph structure corresponding to the second multi-agent real interaction data.

[0054] Here, the graph nodes in the graph structure are defined by the traffic participants (i.e. dynamic entities: vehicles, pedestrians, cyclists, etc.) or road environment elements (i.e. static environment elements: traffic lights, road signs, lane lines, etc.) in the traffic scene, and each graph node can be attached to a feature vector, such as a feature vector containing features extracted from multi-modal sensors or V2X communication devices, such as: position (x, y, z), velocity vector, acceleration, orientation angle, behavior state, traffic intention, etc. The edge connection in the graph structure is defined by the physical spatial distance, motion direction and motion speed between the connected graph nodes. Exemplarily, the edge connection between different graph nodes can be established according to the physical spatial proximity (such as the Euclidean distance being less than a certain threshold), the motion direction similarity (such as the included angle being less than a certain threshold), and the speed correlation (such as the speed difference being within a certain range), to form the initial traffic scene graph structure. In addition, the various relationships between the above graph nodes can be weighted to obtain the edge weight values corresponding to the edge connections.

[0055] Thus, considering the dynamic behavior subjects and static traffic elements, the graph structure is provided with multi-dimensional perception information of "behavior-environment", unstructured traffic scenes are converted into structured graph data, and the semantic association and physical interaction information between multi-agent are retained through multi-dimensional feature modeling of the edges, so as to improve the granularity and accuracy of scene understanding.

[0056] The graph convolution module 220 is configured to dynamically update the adjacency matrix corresponding to the initial graph structure of the traffic scene through a self-attention mechanism, and aggregate neighbor information by using a multi-layer graph convolution layer to update the node feature representation of each graph node layer by layer.

[0057] Specifically, the graph convolution module 220 is configured to model the initial graph structure by using a graph neural network, and to realize node feature updating and high-order relationship perception by using a combination of graph convolution and self-attention mechanism. By dynamically adjusting the adjacency matrix by using the self-attention mechanism, the attention weight between each pair of nodes is learned, so that the model can automatically mine key interaction paths, and the optimized adjacency matrix can emphasize important interaction pairs (such as pairs of vehicles that are currently performing avoidance or merging behaviors), and solve the information transmission limitation that may be caused by the static adjacency relationship.

[0058] Here, the graph convolution module 220 adopts a multi-layer structure design, expands the graph neural network into a multi-layer stack (for example, 3-5 layers), and each layer is responsible for fusing the features of neighbor nodes. By using multi-layer graph convolution, the information of neighbor nodes is gradually aggregated, and the feature representation of each node is updated layer by layer, so that it contains richer context information.

[0059] The global pooling output layer 230 is configured to globally pool the updated node feature representations of all graph nodes to generate an original panoramic traffic scene representation of the overall traffic scene.

[0060] For example, assuming that the node feature of the graph node output by the final graph convolution layer is , then the global scene representation is obtained by global pooling calculation:

[0061] , Equation (1)

[0062] In the equation, represents the original panoramic traffic scene representation, represents the pooling operation, is the total number of graph nodes in the graph structure.

[0063] ​It should be understood that the manner of the pooling operation can be diverse and can be adjusted according to development needs or traffic scenarios, for example, Mean Pooling, Max Pooling or attention-based pooling strategy can be adopted. Thus, by integrating the local node features into the global semantic representation of the whole scene through the global pooling layer, the global pooling can compress the high-dimensional node features into a compact representation, which can reflect the comprehensive information between each traffic participant and the road environment in the traffic scene, and provide comprehensive environmental perception support for the intelligent driving system.

[0064] In some examples of the embodiments of the present application, the graph convolution module can adopt a context-aware graph convolution module based on physical attribute driving, and be used to perform the following operations:

[0065] Based on the edge weights of each edge connection in the initial graph structure of the traffic scene, an initial adjacency matrix is constructed:

[0066] , Equation (2)

[0067] In the equation, denotes the initial edge weight between the graph node and the graph node ; denotes a complex Gaussian kernel function for mapping the physical indicators between different graph nodes into non-negative edge weights; denotes the physical space distance between the graph node and the graph node , denotes the absolute difference between the motion direction angle of the graph node and the graph node , denotes the absolute difference of the motion speed between the graph node and the graph node , and respectively denote the scale parameters of the spatial distance, the angle difference and the speed difference.

[0068] Regarding Equation (2), reflects the proximity of two subjects in space. In a complex traffic scene, the closer the subjects are, the more likely they are to interact, so distance is an important factor of edge weight. As the Euclidean distance increases, the exponential term rapidly decays, indicating that nodes far apart in space have weak influence on each other, while controls the decay rate.

[0069] Subjects with the same or similar directions often have higher relevance when interacting, and the direction difference It can better depict the possibility of separation or interaction of the motion trajectories of different subjects. The greater the difference in direction, The smaller the value, the better. It is used to adjust the degree of influence of angle differences on edge weights.

[0070] When two traffic participants are traveling at similar speeds, it suggests that they may be engaging in synchronized or coordinated behavior during their journeys. It can reflect the degree of dynamic matching between subjects. The greater the speed difference, The more pronounced the edge weight decay, The contribution of differences in adjustment speed to edge weights. Therefore, the greater the difference in physical indices, the smaller the generated edge weights, thus exhibiting a negative correlation when constructing the graph.

[0071] It should be noted that the scale parameter , and It can also be set based on data statistics from specific traffic scenarios. For example, in dense urban traffic scenarios, A smaller setting can be used to more precisely distinguish similar subjects; while in highway scenes, a larger setting can be used. .

[0072] Therefore, by using a composite Gaussian kernel function, the differences in physical spatial distance, direction of motion, and speed are mapped to non-negative edge weights, ensuring that each edge in the graph reflects the physical and motion relationships between real traffic participants. Gaussian forms are applied to the three physical indicators (square terms for distance and speed, and absolute values ​​for direction) to achieve attenuation, and the influence of each indicator is controlled by appropriate scaling parameters.

[0073] Arrange the initial edge weights to obtain a Initial adjacency matrix:

[0074] Equation (3)

[0075] In the formula, Represents the initial adjacency matrix. This represents the total number of nodes in the graph structure; regarding There are no matrix elements corresponding to the initial edge weights. All of them are set to 1, while the other remaining matrix elements are set to 0.

[0076] Here, for node pairs that do not have a real connection (or node pairs that exceed the edge weight threshold), their edge weights are set to 0, while self-connections ( If the number of neighbors of a node is less than the ring number (usually set to 1), a self-loop is added to ensure that the node's own information is not lost in subsequent aggregation. In this way, an initial adjacency matrix that can truly reflect the physical interaction of multiple agents in the traffic scene is constructed. The modeling based on physical indicators makes the graph structure have a more detailed description of the mutual relationship of vehicles, pedestrians and other agents in the real traffic scene, and provides an accurate and sparse physical structure basis for the subsequent graph convolution layer.

[0077] In the first layer graph convolution layer, the attention score between the graph node and the graph node is calculated to update the connection weight between the graph nodes based on the current node features:

[0078] , formula (4)

[0079] In the formula, and respectively represent the linear transformation matrix and the self-attention weight vector of the first layer graph convolution layer, represents vector splicing, represents the Leaky ReLU activation function, represents the vector transpose of , and respectively represent the input feature representation of the graph node and the graph node in the first layer.

[0080] The attention score is normalized to obtain the element in the dynamic adjacency matrix:

[0081] , formula (5)

[0082] In the formula, represents the neighbor node set of the graph node , represents the exponential sum of the attention scores of all neighbor nodes of the graph node , used for normalization processing; represents the normalized attention coefficient of the graph node and the graph node in the first layer, which constitutes the element of the dynamic adjacency matrix.

[0083] Here, through the self-attention mechanism, for each layer, the attention score between nodes is calculated by using linear transformation, splicing and Leaky ReLU activation function, and then the dynamic attention coefficient is obtained through normalization, which is directly used as the weight of the updated adjacency matrix, so that the adjacency matrix is dynamically updated, and the connection weight between nodes can be adjusted in real time according to the current feature state. Therefore, the dynamically adjusted adjacency matrix enables the graph convolution module to sensitively capture the instantaneous and nonlinear interaction changes in the traffic scene.

[0084] In the first layer graph convolution layer, the updated dynamic adjacency matrix is used to aggregate and update the graph node features:

[0085] , Equation (6)

[0086] , Equation (7)

[0087] In the formula, ReLU activation function, the intermediate representation of the graph node obtained by the layer graph convolution layer through neighbor aggregation; the output feature representation of the graph node in the first layer, which is used as the input feature representation of the first layer.

[0088] After layers, the node feature of the graph node output by the final graph convolution module is obtained, the total number of graph convolution layers in the graph convolution module.

[0089] Here, the multi-layer graph convolution is used to aggregate neighbor information layer by layer, realize the continuous update of node features and deep semantic extraction. By weighting and aggregating neighbor features, and combining residual connection and ReLU activation to complete node feature update, it is ensured that the information can not only fuse neighborhood features when passing between layers, but also will not lose its own features. Therefore, the multi-layer aggregated node features have stronger expression ability, which not only reflects local details, but also fuses global context, providing high-quality semantic information for subsequent panoramic scene representation.

[0090] Through the embodiments of the present application, the initial adjacency matrix is constructed by using physical indicators, and the connection weight is dynamically updated by combining the self-attention mechanism, so that the model can accurately capture the instantaneous and nonlinear interaction between multiple agents in the traffic scene. In addition, the semantic expression ability of node features is gradually improved through the multi-layer graph convolution module, and the local information is integrated into the global scene representation through the global pooling output layer, realizing deep semantic fusion and high-quality representation.

[0091] As to the details of the determination of the traffic interaction event in step S120, in some examples of the embodiments of the present application, a deep learning model can be used to automatically identify the target traffic interaction event corresponding to the original panoramic traffic scene representation.

[0092] Specifically, the original panoramic traffic scene representation is processed based on an event classifier to determine the corresponding target traffic interaction event, and the event classifier includes a cascaded Transformer encoder and a multi-layer perception and a Softmax classification layer.

[0093] The Transformer encoder is used to utilize the multi-head self-attention mechanism to model the global context and enhance the semantics of the original panoramic traffic scene representation, thereby extracting deep encoding features with global dependency and deep semantic information.

[0094] It should be noted that for traffic interaction events, there are often local key features and global synergies, and the Transformer encoder can capture global context relationships based on the multi-head self-attention mechanism, even if the input is a global scene vector, it can also adaptively mine potential regional correlations and semantic details. Therefore, it can adaptively capture local details and global relationships in the panoramic scene representation, output more rich traffic scene context information, thereby improving the accuracy of event recognition.

[0095] The multi-layer perception is used to perform nonlinear mapping on the deep encoding features, and the Softmax classification layer is used to determine the corresponding target traffic interaction event.

[0096] For example, the multi-layer perception can include multiple fully connected layers, each layer uses a ReLU activation function, and the deep encoding features are further mapped to an event classification space through nonlinear transformation. The Softmax classification layer maps the features to a probability distribution of each preset traffic interaction event, which can accurately realize automatic classification of specific traffic interaction events, such as accurate classification of multiple traffic interaction events such as high-speed merging, intersection conflict, lane changing, following and vehicle distance adjustment, and emergency collision avoidance.

[0097] Figure 3 A structural connection diagram of an example of a conditional domain adversarial generative network according to an embodiment of the present application is shown.

[0098] It should be noted that the conditional domain adversarial network (CDAN) is a deep learning model that combines conditional generative adversarial network (CGAN) and domain-adversarial training, which can effectively solve the problem of inconsistent cross-domain data distribution. In CDAN, the task-related features are explicitly utilized by introducing conditional information, making the domain adaptation process more accurate and effective.

[0099] In domain-adversarial training, it is through adversarial learning that the model learns a domain-insensitive feature representation, and can be distinguished by the domain classifier whether the feature comes from the source domain or the target domain; at the same time, the feature extractor of the main task will try to deceive the domain discriminator, so that the extracted features cannot be correctly distinguished by the domain.

[0100] However, the traditional domain-adversarial training only focuses on the alignment of feature distribution, ignoring the relationship between features and task labels. In this regard, CDAN introduces conditional information into domain-adversarial training, thereby better aligning the feature distribution of the source domain and the target domain.

[0101] As shown in Figure 3 , the conditional domain adversarial network 300 includes a comprehensive condition calculation module 310, a feature extractor 320, a generator 330, a feature inverse mapping module 340, and a domain classifier 350.

[0102] The comprehensive condition calculation module 310 is used to determine the comprehensive condition information of the conditional domain adversarial network according to the target traffic interaction event and the original panoramic traffic scene representation:

[0103] , Equation (8)

[0104] In the formula, represents the comprehensive condition information, represents the target traffic interaction event, represents the original panoramic traffic scene representation; represents an event encoding function for converting into a low-dimensional embedding vector; represents an attention mechanism to dynamically integrate event embedding and scene features.

[0105] Here, the attention mechanism is used to integrate the coarse-grained traffic interaction event and the scene fine-grained information of the environment summary description, and the attention is weighted and aggregated, so that the final output of the comprehensive condition information has both scene environment context and event features, realizing the adaptability to different environments and events, and supporting finer-grained feature alignment and domain adaptation in complex traffic scenes.

[0106] The feature extractor 320 is configured to extract original simulation features corresponding to the first multi-agent simulation interaction data, and to optimize the features in combination with the comprehensive condition information to determine corresponding simulation optimized features:

[0107] Equation (9)

[0108] Equation (10)

[0109] In the equation, represents the first multi-agent simulation interaction data, represents the feature extractor network, represents the original simulation features, represents the learnable parameter matrix and bias term in the feature extractor, represents the ReLU activation function, represents the feature concatenation operation, represents the simulation optimized features.

[0110] Here, the network type of the feature extractor network can be diversified, such as based on convolutional network, graph neural network or Transformer structure, etc., for uniformly mapping multi-modal input data to a general representation space. In addition, the comprehensive condition information is explicitly injected into the original simulation features of the source domain data, so that the network can consider the specific event / environment context in subsequent mapping.

[0111] The generator 330 is configured to adjust the simulation optimized features to domain enhanced simulation features closer to the real data feature distribution under the condition constraint by using the domain adaptation ability learned in the training process:

[0112] Equation (11)

[0113] Equation (12)

[0114] In the equation, represents the learnable parameter matrix and bias term in the generator; represents the generator mapping under the guidance of the comprehensive condition information represents the domain enhanced simulation features.

[0115] Here, during the domain adversarial training, the generator can learn how to align the simulation feature distribution with the real distribution, so that the enhanced features are closer to the real data feature space. Through the domain adaptive mapping of the generator, the simulation optimized features are further combined with the condition ​Fusion, output domain enhanced simulation features, realize simulation data to real data distribution close. Thus, the conditional information is used again inside the generator, strengthen the context and event elements in cross-domain mapping, the generated More consistent with real data distribution, provide high quality of domain alignment representation.

[0116] Feature inverse mapping module 340 is used to map the domain enhanced simulation features back to the simulation data space, generate a second multi-agent simulation interaction data:

[0117] , Equation (13)

[0118] , Equation (14)

[0119] In the formula, represents the second multi-agent simulation interaction data, represents the Sigmoid activation function, represents the feature inverse mapping module, represents the learnable parameter matrix and bias term of the feature inverse mapping module.

[0120] The feature inverse mapping module maps the domain enhanced features back to the simulation data space through linear mapping and Sigmoid activation function, so that the generated second multi-agent simulation interaction data is consistent with the original simulation data in terms of spatio-temporal structure, sensor data format, etc.

[0121] Through the conditional domain adversarial generation network in the embodiments of the present application, the comprehensive conditional information is explicitly introduced in the original feature extraction and feature domain enhancement optimization, so that the environmental context and target traffic event elements fully participate in feature transformation at different levels, ensuring the fine perception of event semantics and scene semantics in cross-domain mapping. Further, through the feature inverse mapping module, the simulation features enhanced by domain adaptation are mapped back to the original data space, realizing the complete restoration of the data structure, ensuring that the generated second multi-agent simulation interaction data is consistent with the original data in terms of format, space-time and multi-modal information, while integrating the feature advantages of real data, thereby significantly reducing the domain gap between simulation data and real data.

[0122] It should be noted that in the current automatic driving data set construction, the traditional method can only realize rough global feature alignment, and it is difficult to capture the fine-grained event semantics and structural information in the traffic scene, which limits the reliability and generalization ability of simulation data in practical applications.

[0123] In view of this, in some examples of the embodiments of the present application, a high-level and refined loss function design is proposed, which can not only realize global alignment of cross-domain features, but also ensure semantic consistency at the class and structure levels, thereby generating simulation data with high quality and low inter-domain gap.

[0124] Specifically, the loss function of the conditional domain adversarial generation network is:

[0125] , formula (15)

[0126] In the formula, denotes the total loss function, denotes the domain adversarial loss, denotes the semantic alignment loss, denotes the reconstruction loss, and denote the semantic alignment loss super parameter and the reconstruction loss super parameter, respectively.

[0127] , formula (16)

[0128] In the formula, and denote the input samples of the target domain corresponding to the real data and the source domain corresponding to the simulation data, respectively, denotes the domain classifier, and denote the real data feature distribution and the simulation data feature distribution, respectively; denotes the expectation operation on each sample sampled from the source domain data distribution , denotes the expectation operation on each sample sampled from the target domain data distribution .

[0129] Regarding formula (16), for the target domain data, the probability output by the domain classifier is expected to be close to 1; and for the source domain data, the output after the generator mapping is expected to make the domain classifier output close to 0. The log-likelihood loss (cross-entropy loss) is used for calculation, wherein the part for the source domain data passes the negative gradient through gradient reversal, prompting the feature extractor to adjust the generated feature distribution. In addition, the comprehensive conditional information is transmitted to the domain classifier together with the features, so that it considers the specific target traffic interaction event and environmental context when distinguishing the data source, so that the domain adversarial is not only the alignment of the global distribution, but also the matching at the fine-grained semantic level.

[0130] By means of the domain adversarial loss, the domain classifier and the gradient reversal layer, the feature extractor and the generator learn domain-independent feature representations (i.e., domain-invariant features), so that the source domain (simulation data) and the target domain (real data) are aligned as much as possible in the feature space, thereby reducing the difference between the real data and the simulation data in the domain feature distribution. In addition, with the assistance of conditional information, the generator can not only achieve global alignment, but also ensure that the feature alignment under different event types is more fine-grained, which is helpful for processing diverse interactive behaviors in complex traffic scenarios.

[0131] Equation (17)

[0132] In the formula, N denotes the total number of source domain samples, denotes the i-th simulation data sample, denotes the i-th sample output by the generator, denotes the i-th sample output by the generator, denotes the i-th sample output by the generator, denotes the i-th sample output by the generator, denotes the prototype vector corresponding to the i-th sample in the real data feature space, which is the mean of all real data features under the traffic interaction event; denotes the prototype vector corresponding to the i-th sample in the real data feature space, which is the mean of all real data features under the traffic interaction event; denotes the square of the Euclidean norm.

[0133] In the semantic alignment loss, to ensure that the generated simulation features are consistent with the real data in terms of time category at the semantic level, for each target event category, the prototype vector of the category in the real data is first calculated, and then the generated simulation features are made as close as possible to the corresponding category prototype, thereby ensuring that the features of each generated sample are consistent with the semantic distribution of the corresponding category of real data at a fine-grained level, thereby enhancing the fine-grained alignment capability of the generator at the event category level.

[0134] By means of the semantic alignment loss, the feature distribution of the simulation data under different traffic event categories is aligned with the corresponding distribution of the real data, greatly improving the applicability in complex traffic scenarios (such as merging, lane changing, pedestrian interaction, etc.).

[0135] Equation (18)

[0136] In the conditional domain adversarial generative network, the enhanced features are restored to the simulation data space through the introduction of the feature inverse mapping module, so that by designing a reconstruction loss term in the loss function, when the generator and the inverse mapping module work together, the second multi-agent simulation interaction data output finally not only consistent with the real data in terms of feature distribution, but also maintains a certain similarity in the original data structure.

[0137] By the embodiment of the present application, the domain adversarial loss, the semantic alignment loss and the reconstruction loss are weighted and summed to construct a comprehensive loss function to simultaneously realize cross-domain feature alignment, class / event level semantic consistency and preserve the original feature structure of the simulation data in the same framework. In addition, the domain adversarial loss and the semantic alignment loss are both guided by comprehensive condition information (including traffic event labels and traffic scene structure context), so that the network can adaptively adjust to specific event categories and scenes during the alignment process, enhance the generalization, and realize the construction of high-quality, low-inter-domain-gap offline multi-agent intelligent driving data set.

[0138] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of actions, but those skilled in the art should know that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0139] Figure 4 A structural block diagram of an example of a development system of an offline multi-agent intelligent driving data set according to an embodiment of the present application is shown.

[0140] As shown in Figure 4 The development system of the offline multi-agent intelligent driving data set 400 includes a real information acquisition unit 410, an event simulation matching unit 420, a domain adaptive enhancement unit 430 and a data packaging unit 440.

[0141] The real information acquisition unit 410 is configured to perform a preprocessing operation on the first multi-agent real interaction data to obtain second multi-agent real interaction data, and extract an original panoramic traffic scene representation corresponding to the second multi-agent real interaction data through a traffic scene extraction model; the first multi-agent real interaction data includes parameter data collected from a real traffic scene by a multi-modal sensor group and a V2X communication device deployed by a test vehicle; the traffic scene extraction model is a graph convolutional neural network as a model backbone.

[0142] The event simulation matching unit 420 is configured to determine a target traffic interaction event corresponding to the original panoramic traffic scene representation, and obtain first multi-agent simulation interaction data corresponding to the target traffic interaction event from an autonomous driving simulation platform.

[0143] The domain adaptation enhancement unit 430 is configured to perform domain adaptation enhancement on the first multi-agent simulation interaction data based on a conditional domain adversarial network to generate second multi-agent simulation interaction data, so as to narrow the domain gap between simulation data and real data, wherein the conditional information of the conditional domain adversarial network is defined according to the target traffic interaction event and the original panoramic traffic scene representation.

[0144] The data packaging unit 440 is configured to package the second multi-agent simulation interaction data and the second multi-agent real interaction data corresponding to the target traffic interaction event to generate a standardized intelligent driving data set.

[0145] In some embodiments, the embodiments of the present application provide a non-volatile computer readable storage medium, wherein one or more programs including execution instructions are stored in the storage medium, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to perform the steps of any one of the above-mentioned offline multi-agent intelligent driving data set development methods.

[0146] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer readable storage medium, and the computer program includes program instructions, when the program instructions are executed by a computer, the computer program instructions make the computer execute the steps of any one of the above-mentioned offline multi-agent intelligent driving data set development methods.

[0147] In some embodiments, the embodiments of the present application also provide an electronic device, which includes at least one processor and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the steps of the offline multi-agent intelligent driving data set development method.

[0148] Figure 5 is a hardware structure schematic diagram of an electronic device for executing the offline multi-agent intelligent driving data set development method provided by another embodiment of the present application, as shown in Figure 5 The device includes:

[0149] one or more processors 510 and a memory 520, Figure 5 The processor 510 is taken as an example in the memory 520.

[0150] The device for executing the offline multi-agent intelligent driving data set development method can also include an input device 530 and an output device 540.

[0151] The processor 510, the memory 520, the input device 530 and the output device 540 can be connected through a bus or other means, Figure 5 The bus connection is taken as an example.

[0152] The memory 520 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the method for developing an offline multi-agent intelligent driving data set in the embodiments of the present application. The processor 510 executes various functions and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 520, that is, implements the method for developing an offline multi-agent intelligent driving data set.

[0153] The memory 520 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created during use of the electronic device, etc. In addition, the memory 520 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 520 can optionally include a memory remotely arranged with respect to the processor 510, which can be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0154] The input device 530 can receive input digital or character information, and generate signals related to user settings and function control of the electronic device. The output device 540 can include a display device such as a display screen.

[0155] The one or more modules are stored in the memory 520, and when executed by the one or more processors 510, perform the method for developing an offline multi-agent intelligent driving data set in any of the above method embodiments.

[0156] The above product can perform the method provided by the embodiments of the present application, and has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the embodiments can be referred to the method provided by the embodiments of the present application.

[0157] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0158] (1) Mobile communication device: the characteristic of this kind of device is that it has mobile communication function, and the main target is to provide voice and data communication. This kind of terminal includes: smart phone, multimedia phone, functional phone, and low-end phone, etc.

[0159] (2) Ultra Mobile Personal Computer (UMPC) devices: These devices belong to the category of personal computers, have computing and processing capabilities, and usually have mobile Internet access. Such terminals include PDA, MID and UMPC devices.

[0160] (3) Portable entertainment devices: These devices can display and play multimedia content. Such devices include audio and video players, handheld game consoles, e-books, and smart toys and portable car navigation devices.

[0161] (4) Other on-board electronic devices with data interaction functions, such as car-mounted devices installed on vehicles.

[0162] The device embodiments described above are only illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0163] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in each embodiment or some part of the embodiment.

[0164] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for developing an offline multi-agent intelligent driving dataset, comprising: The real interaction data of the first multi-agent agent is preprocessed to obtain the real interaction data of the second multi-agent agent, and the original panoramic traffic scene representation corresponding to the real interaction data of the second multi-agent agent is extracted by the traffic scene extraction model. The first multi-agent real interaction data includes parameter data collected from real traffic scenarios through the multimodal sensor group and V2X communication equipment deployed on the test vehicle; the traffic scenario extraction model is based on a graph convolutional neural network as the model backbone. Determine the target traffic interaction event corresponding to the original panoramic traffic scene representation, and obtain the first multi-agent simulation interaction data corresponding to the target traffic interaction event from the autonomous driving simulation platform; Based on the conditional domain adversarial generative network, the first multi-agent simulation interaction data is enhanced by domain adaptation using the real data feature space as a benchmark to generate the second multi-agent simulation interaction data, thereby narrowing the domain gap between the simulation data and the real data. The conditional information of the conditional domain generative adversarial network is defined based on the target traffic interaction event and the original panoramic traffic scene representation; Based on the target traffic interaction event, the corresponding simulated interaction data of the second multi-agent and the real interaction data of the second multi-agent are encapsulated to generate a standardized intelligent driving dataset. The traffic scene extraction model includes a cascaded graph construction module, a graph convolution module, and a global pooling output layer. The graph construction module is used to construct an initial graph structure of the traffic scene corresponding to the real interaction data of the second multi-agent; the graph nodes in the graph structure are defined by traffic participants or road environment elements in the traffic scene, and the edge connections in the graph structure are defined by the physical spatial distance, direction of movement and speed of movement between the connected graph nodes; The graph convolution module is used to dynamically update the adjacency matrix corresponding to the initial graph structure of the traffic scene through a self-attention mechanism, and to aggregate neighbor information using multiple graph convolution layers to update the node feature representation of each graph node layer by layer. The global pooling output layer is used to globally pool the updated node feature representations of all graph nodes to generate the original panoramic traffic scene representation of the overall traffic scene. The conditional domain adversarial generative network includes a comprehensive conditional computation module, a feature extractor, a generator, a feature inverse mapping module, and a domain classifier. The comprehensive condition calculation module is used to determine the comprehensive condition information of the conditional domain generative adversarial network based on the target traffic interaction event and the original panoramic traffic scene representation: ， In the formula, This indicates comprehensive condition information. Indicates the target traffic interaction event. This represents the original panoramic traffic scene. This represents an event encoding function used to... Convert to a low-dimensional embedding vector; This represents an attention mechanism that dynamically fuses event embeddings and scene features; The feature extractor is used to extract the original simulation features corresponding to the first multi-agent simulation interaction data, and to perform feature optimization in combination with the comprehensive condition information to determine the corresponding simulation optimization features: ， ， In the formula, This represents the simulation interaction data of the first multi-agent agent. Represents the feature extractor network. Indicates the original simulation features. This represents the learnable parameter matrix and bias terms in the feature extractor. Represents the ReLU activation function. This indicates a feature concatenation operation. Represents the simulation optimization features; The generator is used to leverage the domain adaptation capabilities learned during training to adjust the simulation optimization features into domain-enhanced simulation features that more closely approximate the distribution of real data features under constraints. ， ， In the formula, This represents the learnable parameter matrix and bias terms in the generator; Indicates comprehensive condition information Guided generator mapping Representation domain enhances simulation features; The feature inverse mapping module is used to map the domain-enhanced simulation features back to the simulation data space to generate the second multi-agent simulation interaction data. ， ， In the formula, This represents the simulation interaction data of the second multi-agent system. This represents the Sigmoid activation function. This represents the feature inverse mapping module. This represents the learnable parameter matrix and bias terms of the feature inverse mapping module.

2. The method according to claim 1, wherein, Traffic interaction events include any one of the following: highway merging events, urban intersection conflict events, lane changing and overtaking events, following and distance adjustment events, and emergency collision avoidance and braking events.

3. The method according to claim 1, wherein, The preprocessing operation on the real interaction data of the first multi-agent agent to obtain the real interaction data of the second multi-agent agent includes: Outlier detection and noise filtering are performed on the parameters of each data source in the real interaction data of the first multi-agent agent. Time synchronization and coordinate system calibration are performed on the sensor parameters in the real interaction data of the first multi-agent system so that the sensor parameters can be accurately aligned under a unified spatiotemporal reference system.

4. The method according to claim 1, wherein, The graph convolution module employs a physically-driven, context-aware graph convolution module and is used to perform the following operations: Based on the edge weights of each edge connected in the initial graph structure of the traffic scene, an initial adjacency matrix is ​​constructed: ， In the formula, Represents the graph nodes in the initial graph structure With graph nodes Initial edge weights between them; This represents the composite Gaussian kernel function, used to map physical indices between different graph nodes to non-negative edge weights; Represents graph nodes With graph nodes The physical distance between them Represents graph nodes With graph nodes The absolute difference between the angles of the direction of motion, Represents graph nodes The absolute difference in motion velocity between nodes. , and These represent the scale parameters for spatial distance, angular difference, and velocity difference, respectively. Arrange the initial edge weights to obtain a Initial adjacency matrix: ， In the formula, Represents the initial adjacency matrix. This represents the total number of nodes in the graph structure. about There are no matrix elements corresponding to the initial edge weights. All of them are set to 1, while the other remaining matrix elements are set to 0; In the In a layered graph convolutional layer, computation graph nodes Graph Nodes Attention scores between Update the connection weights between graph nodes based on the current node's features: ， In the formula, and They represent the first The linear transformation matrix and self-attention weight vector of the layered graph convolutional layer This represents vector concatenation. This represents the Leaky ReLU activation function. express The vector transpose, and Representing graph nodes Graph Nodes In the The input feature representation of the layer; Normalizing the attention score yields the elements in the dynamic adjacency matrix: ， In the formula, Represents graph nodes The set of neighboring nodes, Represents graph nodes The exponential sum of the attention scores of all neighboring nodes is used for normalization. Represents graph nodes Graph Nodes In the The normalized attention coefficients of the layer constitute the elements of the dynamic adjacency matrix; In the In the graph convolutional layer, the updated dynamic adjacency matrix is ​​used to aggregate and update the graph node features: ， ， In the formula, Represents the ReLU activation function. Indicates the first The graph convolutional layer obtains graph nodes through neighbor aggregation. The middle representation; Represents graph nodes In the The output feature representation of the layer is used as the first layer. The input feature representation of the layer; go through The final graph nodes are obtained after layers from the graph convolution module. Node features , This indicates the total number of graph convolution layers in the graph convolution module.

5. The method according to claim 1, wherein, The determination of the target traffic interaction event corresponding to the original panoramic traffic scene representation includes: The original panoramic traffic scene representation is processed based on an event classifier to determine the corresponding target traffic interaction events; the event classifier includes a cascaded Transformer encoder, a multilayer perceptron, and a Softmax classification layer. The Transformer encoder is used to perform global context modeling and semantic enhancement on the original panoramic traffic scene representation using a multi-head self-attention mechanism, thereby extracting deep encoded features with global dependencies and deep semantic information. The multilayer perceptron is used to perform nonlinear mapping on the deep encoded features and to determine the corresponding target traffic interaction events through the Softmax classification layer.

6. The method according to claim 1, wherein, The loss function of the conditional domain adversarial generative network is: ， In the formula, Represents the total loss function. Indicates domain adversarial loss, Represents semantic alignment loss, Indicates the reconstruction loss. and These represent the semantic alignment loss hyperparameter and the reconstruction loss hyperparameter, respectively. ， In the formula, and These represent the input samples corresponding to the target domain of the real data and the source domain of the simulation data, respectively. Representation domain classifier, and These represent the feature distributions of real data and simulated data, respectively. Indicates the distribution of data from the source domain. Each sample in the middle Perform expectation calculation. Indicates the distribution of data from the target domain. Each sample in the middle Perform expectation calculation; ， In the formula, This represents the total number of samples in the source domain. Indicates the first One simulation data sample, The first output of the generator Enhanced features of each sample This refers to the traffic interaction event corresponding to this sample; For the corresponding real data feature space The prototype vector is the mean of all real data features under this traffic interaction event; Represents the square of the Euclidean norm; 。 7. A development system for an offline multi-agent intelligent driving dataset, used to implement the method as described in any one of claims 1-6; the system comprises: The real information acquisition unit is used to preprocess the real interaction data of the first multi-agent agent to obtain the real interaction data of the second multi-agent agent, and extract the original panoramic traffic scene representation corresponding to the real interaction data of the second multi-agent agent through the traffic scene extraction model. The first multi-agent real interaction data includes parameter data collected from real traffic scenarios through the multimodal sensor group and V2X communication equipment deployed on the test vehicle; the traffic scenario extraction model is based on a graph convolutional neural network as the model backbone. The event simulation matching unit is used to determine the target traffic interaction event corresponding to the original panoramic traffic scene representation, and to obtain the first multi-agent simulation interaction data corresponding to the target traffic interaction event from the autonomous driving simulation platform. The domain adaptive enhancement unit is used to perform domain adaptive enhancement on the first multi-agent simulation interaction data based on the real data feature space of the conditional domain adversarial generative network to generate the second multi-agent simulation interaction data, thereby reducing the domain gap between the simulation data and the real data. The conditional information of the conditional domain generative adversarial network is defined based on the target traffic interaction event and the original panoramic traffic scene representation; The data encapsulation unit is used to encapsulate the corresponding simulated interaction data and real interaction data of the second multi-agent based on the target traffic interaction event, so as to generate a standardized intelligent driving dataset.

Images