A multi-source data coordination and fusion perception method for networked automatic driving

Abstract

The application discloses a kind of multi-source data coordination and fusion perception methods for network connection automatic driving, it is related to the field of automatic driving, comprising: establishing vehicle-roadside multi-agent network relationship graph and preliminary sharing the metadata information of each agent;Image pixel features and point cloud features are extracted and fused using a multi-modal feature extraction network, and a convolutional autoencoder is applied to compress multi-modal fusion feature data and send it to the target network-connected vehicle and decompress it;Attention weight estimation operation, key point information transmission operation and feature linear aggregation operation are performed on the multi-modal fusion features of each agent to obtain the aggregated features;Then apply a multi-scale window attention method to obtain the fused self-attention features;Finally, the environmental target position and class within the visual range of each agent are predicted using a dual-branch decoupled perception head.The application solves the problem of single-vehicle automatic driving in the blind area, obstacle blocking and over-the-horizon conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, and in particular to a multi-source data coordination and fusion perception method for networked automatic driving. BACKGROUND

[0002] Environment perception is one of the most critical technologies in autonomous driving systems, which uses vehicle-mounted vision and radar sensors to collect scene data. Perception algorithms process and analyze traffic target categories, positions, motion states, and other information through computing units to provide important support for vehicle path planning and motion decision-making. Common environment perception algorithms include single-sensor-based and multi-sensor data fusion-based target recognition: (1) Single-sensor-based methods use RGB images or point cloud data as input to learn target visual / spatial feature representations and identify traffic participants of interest. However, cameras are easily affected by natural conditions such as light and weather, and lack accurate depth information. Laser radars cannot obtain specific color, texture, and other appearance information of targets, making it difficult to accurately perceive distant and similar targets. (2) Multi-sensor fusion methods combine image and point cloud inputs and design corresponding data fusion strategies to organically integrate multi-sensor heterogeneous data to improve target recognition accuracy. The core of this method is how to design an effective fusion method to complement different modal data information and enhance environmental target feature representation. In addition, single-vehicle autonomous driving is limited by factors such as sensor installation location, perception distance, angle, and others, making it difficult to accurately perceive and locate dynamic / static blind areas, obstacle occlusions, and over-the-horizon scenarios in complex traffic environments, which poses a huge challenge to the large-scale application of autonomous driving.

[0003] Networked autonomous driving systems are built on the basis of single-vehicle intelligence, combining vehicle-road cooperation, Internet of Vehicles, wireless communication, and other technologies to interact and share information with surrounding networked vehicles and roadside intelligent facilities. The environment perception module can further combine different perspectives and large-scale scene target data to overcome its own blind spots and over-the-horizon perception inaccuracies, improving long-distance recognition accuracy and perception robustness in different scenarios. Existing vehicle-road multi-source data fusion perception methods include two categories: raw data coordination and perception result fusion. (1) Raw data coordination methods directly transmit surrounding intelligent agent (networked vehicles and roadside facilities) sensor data to the current target vehicle for fusion. However, a large amount of sensor data requires high communication bandwidth and computing resources, which is difficult to achieve in real-world applications. (2) Perception result fusion methods process and analyze data under their own perspective and transmit the perception results to the current vehicle for fusion decision-making. This method is simple and efficient, but it is difficult to avoid perception defects under each perspective, and incorrect recognition results may mislead the target vehicle's decision-making process.

[0004] In general, thanks to the scene data provided by surrounding vehicles and roadside facilities, the connected automatic driving system can usually better identify and perceive the traffic target state. However, how to design a vehicle-road multi-source data collaboration and fusion scheme under the condition of limited communication bandwidth, effectively utilize multi-view environmental target information, and solve the challenges of single-vehicle automatic driving in the blind area of vision, obstruction and out-of-sight scene perception error is a key problem that needs to be urgently solved in the development of connected automatic driving. SUMMARY

[0005] The purpose of the present application is to provide a multi-source data collaboration and fusion perception method for connected automatic driving, which can solve the problem of single-vehicle automatic driving in the blind area of vision, obstruction and out-of-sight situation.

[0006] To achieve the above purpose, the present application provides the following scheme:

[0007] A multi-source data collaboration and fusion perception method for connected automatic driving, characterized in that the method comprises:

[0008] Randomly selecting a target connected vehicle, establishing a vehicle-roadside multi-agent network relationship graph within the communication range of the target connected vehicle, and preliminarily sharing the meta data information of each agent in the vehicle-roadside multi-agent network relationship graph; the agents include the target connected vehicle and non-target agents; the non-target agents include non-target connected vehicles and roadside facilities; the meta data information includes pose data, sensor extrinsic matrix and agent type;

[0009] For the meta data information of each agent, a multi-modal feature extraction network is used to extract and fuse image pixel features and point cloud features to obtain multi-modal fusion features;

[0010] For the multi-modal fusion features of each non-target agent, a convolutional autoencoder is applied for data compression, and the compressed data is sent to the target connected vehicle and decompressed; the multi-modal fusion features are pixel-point cloud fusion features;

[0011] After the target connected vehicle receives each compressed data and decompresses it, the multi-modal fusion features corresponding to each agent are subjected to attention weight estimation operation, key point information transmission operation and feature linear aggregation operation to obtain aggregated features;

[0012] For the aggregated features corresponding to each agent, a multi-scale window attention method is applied to obtain fused self-attention features;

[0013] The dual-branch decoupling perception head is used to predict the environmental target position and category within the visual range of each intelligent agent based on the fused self-attention features of each intelligent agent.

[0014] Optionally, the meta-data information of each intelligent agent in the vehicle-roadside multi-agent network relationship graph is preliminarily shared, specifically including:

[0015] Each non-target intelligent agent in the vehicle-roadside multi-agent network relationship graph receives the meta-data information of the target connected vehicle, and each non-target intelligent agent projects its own meta-data information to the target connected vehicle coordinate system through spatial change, so as to preliminarily share the meta-data information among the intelligent agents.

[0016] Optionally, for the meta-data information of each intelligent agent, a multi-modal feature extraction network is used to extract and fuse image pixel features and point cloud features to obtain multi-modal fusion features, specifically including:

[0017] A residual convolutional neural network is applied to the RGB image collected by each intelligent agent to obtain an RGB image feature map of each intelligent agent.

[0018] Point cloud data in the point cloud image collected by each intelligent agent is transformed into a series of columnar tensors and aggregated and compressed into a 2D pseudo image.

[0019] The 2D pseudo image is input into a columnar neural network structure to extract a point cloud feature map of each intelligent agent; the spatial dimensions of the RGB image feature map and the point cloud feature map are the same.

[0020] For each intelligent agent, each point in the point cloud feature map is projected into the RGB image feature map and the corresponding pixel of each point is determined according to the sensor extrinsic parameter matrix.

[0021] The neighboring region features of each pixel are aggregated by using a bilinear interpolation method, and the aggregated neighboring features corresponding to each pixel are projected to the point cloud feature plane to obtain the pixel-point cloud fusion features corresponding to each intelligent agent; the neighboring region features of each pixel are the region features within a preset range of each pixel.

[0022] Optionally, before applying the convolutional autoencoder to the multi-modal fusion features of each non-target intelligent agent for data compression, the following steps are included:

[0023] According to the spatial position information of each non-target intelligent agent, a pose offset coordinate of each non-target intelligent agent during a data transmission delay period is calculated by using a space-time correction method; the data transmission delay period refers to the time difference between the time when the non-target intelligent agent sends information to the target connected vehicle and the time when the target connected vehicle receives the information.

[0024] determine a pose offset coordinate change amount of the non-target intelligent agent during the data transceiving time delay according to the feature channel number and the data transceiving time delay value of the multi-modal fusion feature of each non-target intelligent agent;

[0025] compensate the multi-modal fusion feature of each non-target intelligent agent according to the pose offset coordinate and the pose offset coordinate change amount, to obtain a multi-modal fusion feature after time delay compensation.

[0026] Optionally, the expression of the pose offset coordinate is:

[0027]

[0028] In the formula, (X t , Y t ) represents the pose offset coordinate of the non-target intelligent agent during the data transceiving time delay; (X s , Y s ) represents the pose coordinate collected by the non-target intelligent agent at the moment of sending information; represents a rotation and translation affine transformation matrix; R 11 , R 12 , R 21 , R 22 represents the rotation of the sensor of the non-target intelligent agent in four directions; δ X and δ Y represent the translation of the sensor of the non-target intelligent agent in the X direction and the Y direction.

[0029] The expression of the pose offset coordinate change amount is:

[0030] DPE(Δt)=f(p c (Δt));

[0031] wherein,

[0032] In the formula, DPE(Δt) represents the pose offset coordinate change amount; f(·) represents a linear mapping layer; Δt represents the data transceiving time delay; c represents the feature channel number of the multi-modal fusion feature; C represents the maximum value of the feature channel number of the multi-modal fusion feature; c=2k represents that c is an even value; c=2k+1 represents that c is an odd value.

[0033] Optionally, a convolutional self-encoder is applied to the multi-modal fusion feature of each non-target intelligent agent for data compression, and the compressed data is sent to the target networked vehicle and decompressed, specifically including:

[0034] An encoder with a cascaded convolutional structure is used to compress the multi-modal fusion features along the feature channel dimension; the multi-modal fusion features are the time-delay compensated multi-modal fusion features;

[0035] The compressed data is sent to the target connected vehicle;

[0036] When the target connected vehicle receives each compressed data, a decoder with a deconvolutional structure is used to restore the feature channel dimension before compression.

[0037] Optionally, the multi-modal fusion features corresponding to each agent are subjected to attention weight estimation, key point information transmission and feature linear aggregation operations to obtain aggregated features, specifically including:

[0038] A vehicle-roadside directed graph representation is constructed according to the vehicle-roadside multi-agent network relationship graph;

[0039] For each pair of associated nodes in the vehicle-roadside directed graph representation, an attention weight value of each pair of associated nodes is calculated according to the multi-modal fusion features of the associated nodes using the attention weight estimation operation; each pair of associated nodes refers to two nodes having an edge connection relationship; the two nodes are respectively denoted as a source node and a target node; the information of the source node is transmitted to the target node;

[0040] An attention feature of the edge relationship between each pair of associated nodes is calculated according to the multi-modal fusion features of the target node using the key point information transmission operation, and an information transmission result of the edge relationship is obtained;

[0041] The attention weight value of each pair of associated nodes and the information transmission result of the corresponding edge relationship are subjected to feature linear aggregation to obtain the aggregated features of each node in the vehicle-roadside directed graph representation.

[0042] Optionally, the expression of the attention weight estimation operation is:

[0043]

[0044] wherein,

[0045]

[0046]

[0047] In the formula, LWE(i, j) represents the attention weight value of the associated nodes; i and j are the numbers of the two nodes in a pair of associated nodes; Softmax() represents an activation function; and Concat() represents an aggregation operation. represents the associated node attention weight value obtained by the attention head h; m represents the number of attention heads; MLP() represents a multi-layer perception, i.e., a linear mapping layer; Fi represents the multi-modal fusion feature corresponding to node i in the associated node; F j represents the multi-modal fusion feature corresponding to node j in the associated node; represents a learnable linear layer parameter; Φ(e ij ) represents the edge relationship between node i and node j; T represents transposition; C represents the maximum value of the feature fusion of the multi-modal fusion feature.

[0048] The expression of the key point information transmission operation is as follows:

[0049]

[0050] wherein,

[0051] In the formula, KMP(i, j) represents the attention feature of the edge relationship between node i and node j. represents the edge relationship attention feature obtained by the attention head h.

[0052] Optionally, a multi-scale window attention method is applied to the aggregated feature corresponding to each agent to obtain a fused self-attention feature, and the method specifically comprises the following steps:

[0053] The aggregated feature of each agent is divided into multiple window features through multi-window feature division.

[0054] For each window, the window feature is combined with a visual multi-head self-attention mechanism to obtain the attention feature of each attention head.

[0055] For each attention head, the attention features of each attention head of each window are combined to obtain the multi-window attention feature of each attention head.

[0056] The multi-window attention features of each attention head are combined to obtain the multi-head attention feature of each attention branch.

[0057] The multi-head attention features of each attention branch are combined through a split attention structure to obtain a multi-branch attention feature, i.e., the fused self-attention feature.

[0058] According to the specific embodiments of the present application, the following technical effects are provided:

[0059] The application provides a vehicle-road data coordination and fusion perception method for networked automatic driving, under certain communication range and bandwidth resource conditions, multi-view environment target data provided by different intelligent agents (surrounding networked vehicles and roadside facilities) is combined, a multi-agent heterogeneous data fusion method and a multi-scale window attention fusion module are designed, intelligent agent information in different spatial positions is distinguished, and important features in different regions are organically combined, so that the long-distance and wide-range environment target perception accuracy of the automatic driving vehicle is improved, and the perception generalization ability in different scenarios is improved. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0061] Figure 1 A multi-source data coordination and fusion perception method for networked automatic driving is provided for the first embodiment of the present application.

[0062] Figure 2 A multi-source data coordination and fusion perception method for networked automatic driving is provided for the first embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0064] As one of the most advanced technologies in the field of intelligent transportation science, autonomous driving (AD) technology mainly relies on various vehicle-mounted sensor devices (including visual cameras, laser radars, millimeter wave radars, etc.), computing units, and line control systems to recognize environmental information and judge motion states, endows vehicles with the ability of perception, planning, decision-making, and control, and independently and autonomously completes driving tasks without human assistance. On the basis of single-vehicle autonomous driving, connected automated vehicles (CAVs) further utilize vehicle networking and wireless communication technologies to organically link the “human-vehicle-road-cloud” traffic participants, and combine the multi-perspective environmental information provided by surrounding connected vehicles and roadside sensing facilities to upgrade the capabilities of autonomous driving vehicles in environmental perception, computing decision-making, and control execution, and accelerate the application research and industrialization landing of autonomous driving technology. Specifically, based on dedicated short range communications (DSRC) or cellular vehicle-to-everything (C-V2X) technology, data sharing and information interaction in connected autonomous driving mainly has two levels: on the one hand, for vehicle-to-vehicle (V2V) systems, the current vehicle (ego vehicle) receives data sent from surrounding vehicle nodes, such as the positions and speeds of neighboring vehicles, environmental target information under different perspectives, etc., the current vehicle analyzes and controls the vehicle attitude and future motion path according to the decision-making results, avoids collision accidents caused by blind area obstruction by obstacles, and realizes vehicle following driving, overtaking and lane changing, and other vehicle-to-vehicle cooperative applications. On the other hand, for vehicle-to-infrastructure (V2I) systems, the current vehicle (ego vehicle) can also interact and share with sensing data arranged at the roadside / crossing; since the roadside sensing devices are erected at a high position, they have a wider field of view and angle, and can provide long-distance, large-range environmental global information to assist autonomous vehicles in accurately perceiving targets in the beyond-visual-range and dynamic blind area, predicting the front emergency / abnormal / potential risk situation, and making planning and decision-making in advance to control the vehicle to drive safely.

[0065] Environment perception is an important part of the networked automatic driving system, and accurate scene target information can provide important support for vehicle path planning and motion control. On the one hand, identify and locate the surrounding vehicles, obstacles, road conditions (such as signs, traffic lights, etc.), and other information to assist vehicles in avoiding collisions and accident risks, and to achieve functions such as vehicle platooning, positioning navigation, etc. On the other hand, detect uncertain situations such as traffic emergencies, abnormalities, accidents, etc. in front of the road, predict potential risk areas, and help vehicles plan motion paths and make safety control decisions in advance. This is the premise and foundation of ensuring the safety and efficiency of vehicle driving; however, existing automatic driving perception methods have certain defects and cannot meet the needs of real-world applications, and networked automatic driving cooperative perception technology still needs further research.

[0066] To this end, the present application provides a multi-source data cooperative and fusion perception method for networked automatic driving, which respectively deploys sensors (cameras and lidar), communication units and data processing units, etc. on vehicles and roadside ends, obtains environmental target features under their respective perspectives through preprocessing; taking the current vehicle (ego vehicle) as the center vehicle (target networked vehicle), using Dedicated Short Range Communications (DSRC) or Cellular Vehicle-to-Everything (C-V2X) technology, designing a multi-agent heterogeneous data fusion method and a multi-scale window attention module, organically fusing multi-perspective target representations of different networked vehicles and roadside facilities from local to global, achieving more accurate and robust global data cooperative and target fusion perception; the present application fully utilizes the multi-source heterogeneous data resources in the intelligent vehicle-road cooperative system under different perspectives, adaptively selects and combines target state information under different resolutions, and can effectively solve the single-vehicle perception inaccuracy problem in abnormal traffic environments (occlusion, truncation, blind area), ensuring the safety of large-scale automatic driving.

[0067] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0068] Embodiment 1

[0069] As shown in Figure 1 and 2 , the present embodiment provides a multi-source data cooperative and fusion perception method for networked automatic driving, which comprises:

[0070] S1: randomly selecting a target connected vehicle, establishing a vehicle-roadside multi-agent network relationship graph within the communication range of the target connected vehicle, and preliminarily sharing metadata information of each agent in the vehicle-roadside multi-agent network relationship graph; the agents include the target connected vehicle and non-target agents; the non-target agents include non-target connected vehicles and roadside facilities; the metadata information includes pose data, sensor extrinsic matrix and agent type.

[0071] Here, the connected vehicle and the roadside facility are regarded as an agent (agent), denoted as a∈(1,…,N), where N represents the number of agents within the communicable range. A current vehicle (e) is randomly selected as a target vehicle, and a vehicle-vehicle and vehicle-roadside network relationship graph is constructed within a 70 m communicable range (according to the DSRC standard). Each agent in the network will share metadata information, including its own pose, sensor extrinsic matrix, and agent type representing a roadside facility or a connected vehicle. Therefore, each non-target agent a can receive the metadata of the current vehicle e at time t Meanwhile, the metadata of the current vehicle e is Through spatial transformation projection into the current vehicle coordinate system, the preliminary sharing of basic information between different agents is realized.

[0072] S2: for the metadata information of each agent, a multi-modal feature extraction network is used to extract and fuse image pixel features and point cloud features to obtain multi-modal fusion features.

[0073] For all agents (including connected vehicles and roadside facilities) within the communicable range, first, the spatial position relationship and the extrinsic matrix information in the metadata are used to convert the sensor data of different agents into the current vehicle coordinate system, i.e., to obtain different perspective data in the current vehicle coordinate system. Based on this, a multi-modal feature extraction network (MFN) is used to extract and learn image-point cloud features for each agent. Finally, the respective feature representations are compressed and the time offset error of different agents is compensated, and the corrected features are transmitted and shared to the current vehicle, realizing the preliminary fusion of multi-agent multi-source data.

[0074] The multi-modal feature extraction network MFN mainly includes an image feature extraction branch, a point cloud feature extraction branch, and a pixel-point fusion module. The image feature extraction branch takes the RGB image I∈R H×W×3 as input, H is the height of the image, W is the width of the image, and a classical residual convolutional neural network structure (ResNet) is used to obtain abstract high-dimensional feature representations represents the size of the agent e at time t RGB image feature map. The point cloud feature extraction branch extracts the original point cloud P ∈ R N×3 The spatial point cloud is first transformed into a series of columnar tensors and clustered into a corresponding 2D pseudo-image representation, and then the 2D pseudo-image is input into a traditional columnar neural network structure (PointPillar) to extract a point cloud feature map The point cloud feature map and the RGB image feature map have the same spatial dimensions. The pixel-point fusion module projects the points on the point cloud feature map to the RGB image feature map according to the sensor extrinsic matrix (including the camera and lidar parameter matrix), and finds the corresponding pixels. The bilinear interpolation method is used to aggregate the features in the adjacent region of the pixel, and the aggregated adjacent features corresponding to each pixel are projected back to the point cloud feature plane. In this way, the point-pixel information at each position of the feature map is fused, and this one-to-one modal feature method is more accurate. The adjacent region features of each pixel are the region features within the preset range of each pixel. The pixel-point feature fusion calculation process is shown in the following formula (1):

[0075]

[0076] wherein, denotes the multi-modal fusion feature, Concate(·) denotes the element-wise addition operation, and B(·) denotes the bilinear interpolation method. In this way, the image-point cloud fusion representation set at time t obtained from different agent perspectives can be represented as

[0077] S3: applying a convolutional autoencoder to the multi-modal fusion feature of each non-target intelligent agent for data compression, sending the compressed data to the target connected vehicle and decompressing; the multi-modal fusion feature is a pixel-point cloud fusion feature.

[0078] Data compression transmission aims to lightweight the multi-modal fusion features of different non-target intelligent agents to reduce communication bandwidth consumption. In order to greatly improve the data transmission efficiency and relieve the communication bandwidth demand, a simple and efficient convolutional encoder-decoder structure is further adopted to compress the multi-modal fusion features of each non-target intelligent agent. In the encoder part, a cascaded 1x1 convolutional layer is used for compression transmission along the feature channel dimension, and the compressed feature channel number is denoted as C'(C' << 1024); the current target connected vehicle e receives feature data from different agents, and a decoder with a 1x1 deconvolution structure is used to restore the feature channel dimension before compression, so as to reduce the feature loss before and after compression as much as possible, and at the same time, to maximize the reduction of resource consumption of data transmission. In this way, the current target connected vehicle e aggregates the environmental multi-view data features provided by the surrounding connected vehicles and roadside facilities.

[0079] Considering the displacement deviation caused by the time delay in the process of different agents transmitting data to the current target connected vehicle, further, before applying the convolution self-encoder to the multi-modal fusion features of each non-target agent for data compression, the spatial-temporal correction method (STC) and the delay-aware positional encoding method (DPE) are proposed to compensate for the data misalignment problem, realizing lossless transmission and sharing of data between agents.

[0080] Among them, the spatial-temporal correction method (STC) uses a differentiable 2D spatial affine transformation Γ ξ (·) to compensate for the pose displacement deviation of the current target connected vehicle from the non-target agent during the period from sending information at t1 to receiving data at t2 (data transmission and reception time delay period Δt = t2-t1), that is, the spatial-temporal correction method is used to calculate the pose displacement coordinates STC(Δt) of each non-target agent during the data transmission and reception time delay period, which can be specifically represented as formula (2) as follows:

[0081]

[0082] Among them, (X t ,Y t ) represents the pose displacement coordinates of the non-target agent during the data transmission and reception time delay period; (X s ,Y s ) represents the pose coordinates collected by the non-target agent at the time of sending information; represents the rotation and translation affine transformation matrix; R 11 , R 12 , R 21 , R 22 represents the rotation of the sensor of the non-target agent in 4 directions; δ X and δ Y represent the translation of the sensor of the non-target agent in the X direction and the Y direction. In addition, before the affine transformation, the original input coordinates are sampled using the bilinear interpolation method to ensure that the target coordinates after transformation are integer values, realizing the correction of the displacement error of the data transmission and reception time delay between agents.

[0083] The delay perception position encoding method (DPE) further perceives the spatial position change caused by the feature target motion during the time delay and performs coordinate compensation. The DPE is composed of a linear mapping layer and a learnable position embedding vector, and is initialized based on the time delay Δt and the feature channel number c∈[1,C] information using sine and cosine functions. That is, the pose offset coordinate change amount DPE(Δt) of the non-target intelligent agent during data transmission and reception delay is determined according to the feature channel number of the multi-modal fusion feature of each non-target intelligent agent and the data transmission and reception delay value, and the specific calculation process is as follows formula (3) and (4):

[0084]

[0085] DPE(Δt)=f(p c (Δt),R C →R C (4)

[0086] Wherein, f(·) represents a linear mapping layer, c represents the feature channel number of the multi-modal fusion feature; C represents the maximum value of the feature channel number of the multi-modal fusion feature; c=2k represents that c is an even value; c=2k+1 represents that c is an odd value. Further learn a more general time delay coordinate compensation result according to different embedding vectors, compensate the multi-modal fusion feature of each non-target intelligent agent according to the pose offset coordinate and the pose offset coordinate change amount, and obtain the multi-modal fusion feature after time delay compensation. Therefore, the image-point cloud fusion feature set after time delay compensation of different intelligent agents at time t can be updated as Wherein

[0087] Therefore, when data compression is performed in step S3, the data to be compressed is the multi-modal fusion feature after time delay compensation.

[0088] S4: After the target connected vehicle receives each compressed data and decompresses, the multi-modal fusion feature corresponding to each intelligent agent is subjected to attention weight estimation operation, key point information transmission operation and feature linear aggregation operation, and the aggregated feature is obtained.

[0089] Through the above steps, the current target connected vehicle receives multi-view feature representations from different intelligent agents (surrounding connected vehicles and roadside facilities). Assuming that the configuration of the sensing devices of the same intelligent agent is consistent, a simple and efficient vehicle-road directed graph representation is first constructed according to the vehicle-road multi-agent network relationship graph. Among them, the node (node) type includes roadside facilities and connected vehicles; the edge (edge) relationship includes car-car, car-roadside, roadside-car and roadside-roadside, which is represented as Φ(e ij}. Since the on-board and roadside end sensor data have different characteristics, the multi-agent heterogeneous fusion module (MHF) considers the fusion features from different connected vehicles and roadside ends differently by analyzing the different node types and edge relationships of the directed graph. The fusion module consists of three parts: attention weight estimation (AWE), keypoint message passing (KMP), and linear feature aggregation (LFA) operations.

[0090] The attention weight estimation (LWE) operation is used to calculate the importance score of each pair of nodes in the associated node and edge relationship combination. Specifically, the associated nodes are abstracted as a set of key-value pairs Query (Q) and Key (K), and a multi-head self-attention layer is used to calculate and aggregate the group components of each attention head, and finally the activation probability value is output as the importance score of the corresponding node feature through the Softmax regularization function.

[0091] The specific steps of the attention weight estimation (LWE) operation are as follows: for each pair of associated nodes in the vehicle-roadside directed graph representation, the attention weight value of each pair of associated nodes is calculated using the attention weight estimation operation according to the multi-modal fusion features of the associated nodes; each pair of associated nodes refers to two nodes with edge connection relationship; the two nodes are respectively denoted as source node and target node; the information of the source node is transmitted to the target node. The formulas involved in the whole process are shown in the following formulas (5)-(8), where the subscript t is simplified:

[0092]

[0093]

[0094]

[0095]

[0096] where LWE(i,j) represents the attention weight value of the associated nodes; i and j are the numbers of the two nodes in a pair of associated nodes; Softmax() represents the activation function; Concat() represents the aggregation operation; represents the attention weight value of the associated nodes obtained by attention head h; m represents the number of attention heads; MLP() represents a multi-layer perceptron, i.e., a linear mapping layer; F i represents the multi-modal fusion features corresponding to node i in the associated nodes; Fj denotes the multi-modal fusion feature corresponding to node j in the associated node; denotes the learnable linear layer parameters; Φ(e ij denotes the edge relationship between node i and node j; T denotes the transpose; C denotes the maximum value of the feature fusion of the multi-modal fusion feature.

[0097] After obtaining the importance score of each pair of source node and target node, the key point message passing (KMP) further calculates the attention feature of the corresponding edge relationship between each pair of the associated nodes according to the multi-modal fusion feature of the target node, as shown in the following formulas (9) and (10):

[0098]

[0099]

[0100] In the formula, KMP(i, j) denotes the attention feature of the edge relationship between node i and node j; denotes the edge relationship attention feature obtained by the attention head h. In the associated node, node i can be regarded as a source node, and node j can be regarded as a target node; then Fj denotes the multi-modal fusion feature corresponding to node j, that is, the multi-modal fusion feature corresponding to the target node.

[0101] Finally, the attention weight value (LWE(i, j)) of each pair of associated nodes is multiplied by the corresponding key point message passing result (that is, the edge relationship information passing result KMP(i, j)), and the feature linear aggregation (LFA) operation is performed to adaptively select the intelligent agent feature with high importance, to obtain the aggregated feature of each node in the vehicle-roadside directed graph representation. As shown in the following formula (11):

[0102] F i = MLP i (LWE(i, j)·KMP(i, j)), j∈N(i) (11)

[0103] S5: applying a multi-scale window attention method to the aggregated feature corresponding to each intelligent agent to obtain a fused self-attention feature.

[0104] For each intelligent agent feature, the cascaded multi-scale window attention (MWA) method emphasizes different regional target information, that is, a large window obtains long-distance target visual clues to compensate for positioning errors, and a small window focuses on local neighborhood fine-grained features. Specifically, it is assumed that there are k parallel attention branches, each attention head has m number, and the dimension of the attention head is The g-th branch window size is denoted as p g ×p g The aggregated feature F of the input agent i is then i ∈R H×W×C In the form of a window, it can be expressed as the following formula (12):

[0105]

[0106] Wherein, is the number of divided windows; for each window feature, the visual multi-head self-attention is used to calculate the importance of each component, as shown in the following formula (13):

[0107]

[0108] Wherein, B represents the relative position encoding, which is formalized as a parameter matrix Further, the multi-window attention results of different attention heads are aggregated to obtain the corresponding feature representation As shown in the following formula (14)-(16):

[0109]

[0110] Wherein,

[0111] Wherein, g represents the number of windows, h represents the number of attention heads, h = 1, 2, …, m.

[0112] Wherein, represents an agent aggregated multi-head attention feature result (i.e. a multi-head attention feature corresponding to one attention branch), and represents the multi-window attention feature result of the h-th attention head of an agent. Finally, the separation attention structure is used to organically combine the attention features of k branches, so that the model can adaptively screen the potential target area, as shown in the following formula (17):

[0113] Y = ATT split [Y 1 ,Y 2 ,…,Y k ] (17)

[0114] Wherein, ATT split [·] represents a separation attention layer, which is composed of multiple 1×1 and 3×3 convolution calculations, and calculates and retains the regions with high importance of different branch feature components.

[0115] Therefore, step S5 specifically comprises:

[0116] (1) For each of the agents, the aggregated features are divided into multiple window features through multi-window feature division.

[0117] (2) For each window, the window features are combined with a visual multi-head self-attention mechanism to obtain attention features of each attention head.

[0118] (3) For each attention head, the attention features of each attention head of each window are combined to obtain multi-window attention features of each attention head.

[0119] (4) The multi-window attention features of each attention head are combined to obtain multi-head attention features of each attention branch.

[0120] (5) The multi-head attention features of each attention branch are combined using a separate attention structure to obtain multi-branch attention features Y, which are the fused self-attention features.

[0121] S6: For each of the agents, the fused self-attention features are used to predict the location and category of the environmental targets within the visual range of each agent using a dual-branch decoupled perception head.

[0122] For the fused self-attention features Y, a dual-branch perception head structure is used to decouple the fused self-attention features of each agent for position coordinate regression and category classification prediction. Two branches use two layers of 1x1 convolution layers, respectively. The regression branch outputs (x, y, z, w, l, h, θ), representing the spatial coordinates, 3D size, and orientation angle of the target. The classification branch outputs the confidence score of each region belonging to the target or background. In addition, smooth-L1 regression loss and focal loss (FL) are introduced to supervise the prediction results of the two branches, as shown in the following equations (18)-(19):

[0123]

[0124] FL = -α(1-p t ) γ log(p t ) (19)

[0125] where b p and b g represent the predicted and ground-truth rectangular boxes, respectively, p t represents the probability value of being a positive sample, and α and γ represent the harmonic factor parameters, respectively.

[0126] The embodiment has the following advantages:

[0127] (1) The vehicle-road multi-source data collaborative and fusion perception method for networked automatic driving solves the problem of inaccurate perception of single vehicle automatic driving in the blind area, obstacle shielding and over-the-horizon conditions. Under certain computing resources and communication bandwidth conditions, the multi-view sensing data from different networked vehicles and roadside facilities are fully utilized to realize long-distance, large-range and global collaborative and target fusion perception, and to ensure the motion safety of automatic driving vehicles and the application deployment of intelligent vehicle-road cooperative systems.

[0128] (2) The vehicle-road multi-source data collaborative and fusion perception method for networked automatic driving greatly alleviates the data heterogeneity problem caused by different intelligent agents (networked vehicles and roadside facilities) due to sensor layout, hardware configuration and other problems. The proposed multi-agent heterogeneous data fusion method constructs a node-edge intelligent agent directed graph network structure, differentially calculates and fuses different node types and edge relationship features, and the time delay compensation strategy can reduce the influence of heterogeneous sensor data noise and displacement deviation, providing accurate information for the subsequent target perception process.

[0129] (3) The vehicle-road multi-source data collaborative and fusion perception method for networked automatic driving effectively makes up for the problems of lack of target feature information and low recognition and positioning accuracy in a single scale. The designed multi-scale window attention module focuses on the global and local context information of the target under different resolutions by introducing large and small window attention operations, and organically fuses multi-scale target importance feature extraction to achieve more accurate and robust perception results. In addition, the setting of different scale window attention further improves the generalization ability of the algorithm under different positioning errors.

[0130] Embodiment two

[0131] The embodiment provides a vehicle-road multi-source data collaborative and fusion perception system for networked automatic driving, which comprises:

[0132] A network relationship graph construction module is configured to randomly select a target networked vehicle, establish a vehicle-roadside multi-agent network relationship graph within the communication range of the target networked vehicle, and preliminarily share metadata information of each agent in the vehicle-roadside multi-agent network relationship graph; the agents include the target networked vehicle and non-target agents; the non-target agents include non-target networked vehicles and roadside facilities; and the metadata information includes pose data, sensor extrinsic matrix and agent type.

[0133] A pixel-point cloud feature fusion module is configured to extract and fuse image pixel features and point cloud features by using a multi-modal feature extraction network for the metadata information of each agent, to obtain multi-modal fusion features.

[0134] a data compression and transmission module configured to apply a convolutional autoencoder to the multi-modal fusion feature of each non-target intelligent agent for data compression, send compressed data to the target connected vehicle and decompress the compressed data; the multi-modal fusion feature is a pixel-point cloud fusion feature.

[0135] a feature aggregation module configured to, after the target connected vehicle receives and decompresses each compressed data, perform an attention weight estimation operation, a key point information transmission operation and a feature linear aggregation operation on the multi-modal fusion feature corresponding to each intelligent agent to obtain an aggregated feature.

[0136] a feature fusion module configured to apply a multi-scale window attention method to the aggregated feature corresponding to each intelligent agent to obtain a fused self-attention feature.

[0137] a target prediction module configured to, for the fused self-attention feature of each intelligent agent, use a dual-branch decoupled perception head to predict the environmental target position and category within the visual range of each intelligent agent.

[0138] Embodiment three

[0139] The embodiment provides an electronic device, including a memory and a processor, the memory is used for storing a computer program, and the processor runs the computer program to make the electronic device execute the multi-source data coordination and fusion perception method for connected automatic driving of embodiment one.

[0140] Optionally, the electronic device can be a server.

[0141] In addition, the embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the multi-source data coordination and fusion perception method for connected automatic driving of embodiment one.

[0142] Embodiments of the present application can be provided as a method, system, or computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0144] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0145] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0146] The same or similar parts among various embodiments can be mutually referred to. For the system disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant part can be referred to the method part.

[0147] The principles and implementation manners of the present application are described by using specific examples in the present application. The above embodiment description is only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, according to the idea of the present application, the specific implementation manner and application range can be changed. In conclusion, the content of the present application should not be understood as the limitation of the present application.

Claims

1. A multi-source data collaborative and fusion perception method for connected autonomous driving, characterized in that, The method includes: A target connected vehicle is randomly selected, and a vehicle-roadside multi-agent network relationship diagram is established within the communication range of the target connected vehicle. Metadata information of each agent in the vehicle-roadside multi-agent network relationship diagram is initially shared. The agents include the target connected vehicle and non-target agents. The non-target agents include non-target connected vehicles and roadside facilities. The metadata information includes pose data, sensor extrinsic parameter matrix, and agent type. For the metadata information of each intelligent agent, a multimodal feature extraction network is used to extract and fuse image pixel features and point cloud features to obtain multimodal fusion features; For each non-target intelligent agent, the multimodal fusion feature is compressed using a convolutional auto-decoder, and the compressed data is sent to the target connected vehicle and decompressed; the multimodal fusion feature is a pixel-point cloud fusion feature; After the target connected vehicle receives and decompresses each of the compressed data, it performs attention weight estimation, key point information transmission, and feature linear aggregation operations on the multimodal fusion features corresponding to each intelligent agent to obtain the aggregated features. A multi-scale window attention method is applied to the aggregated features corresponding to each agent to obtain fused self-attention features. For the fused self-attention features of each agent, a dual-branch decoupled sensor head is used to predict the location and category of environmental targets within the visual range of each agent. Specifically, for each agent, the multimodal fusion features are subjected to attention weight estimation, key point information transfer, and feature linear aggregation operations to obtain aggregated features, including: Construct a directed graph representation of the vehicle-roadside network based on the aforementioned vehicle-roadside multi-agent network relationship graph; For each pair of associated nodes in the vehicle-roadside directed graph representation, the attention weight value of each pair of associated nodes is calculated using the attention weight estimation operation based on the multimodal fusion features of the associated nodes; each pair of associated nodes refers to two nodes with an edge connection relationship; the two nodes are respectively denoted as the source node and the target node; the information of the source node is passed to the target node; Based on the multimodal fusion features of the target node, the attention features of the edge relationship between each pair of associated nodes are calculated using the key point information transmission operation to obtain the edge relationship information transmission result; The attention weight values ​​of each pair of associated nodes and the information transmission results of the corresponding edge relationships are linearly aggregated to obtain the aggregated features of each node in the vehicle-roadside directed graph representation.

2. The method according to claim 1, characterized in that, The metadata information of each agent in the vehicle-roadside multi-agent network relationship graph is initially shared, specifically including: In the vehicle-roadside multi-agent network relationship diagram, each non-target agent receives the metadata information of the target connected vehicle. At the same time, each non-target agent projects its own metadata information onto the coordinate system of the target connected vehicle through spatial transformation, so as to achieve the initial sharing of the metadata information among the agents.

3. The method according to claim 1, characterized in that, For the metadata information of each intelligent agent, a multimodal feature extraction network is used to extract and fuse image pixel features and point cloud features to obtain multimodal fused features, specifically including: A residual convolutional neural network is applied to the RGB image acquired by each of the intelligent agents to obtain the RGB image feature map of each of the intelligent agents; The point cloud data in the point cloud image collected by each of the aforementioned intelligent agents is transformed into a series of columnar tensors and aggregated and compressed into a 2D pseudo-image; The 2D pseudo-image is input into a columnar neural network structure to extract point cloud feature maps for each agent; the RGB image feature maps and the point cloud feature maps have the same spatial dimension. For each of the aforementioned intelligent agents, each point in the point cloud feature map is projected onto the RGB image feature map according to the sensor extrinsic parameter matrix, and the pixel corresponding to each point is determined. The neighboring region features of each pixel are aggregated using bilinear interpolation, and the aggregated neighboring features corresponding to each pixel are projected onto the point cloud feature plane to obtain the pixel-point cloud fusion features corresponding to each agent; the neighboring region features of each pixel are the region features within a preset range of each pixel.

4. The method according to claim 1, characterized in that, Before applying a convolutional auto-decoder to compress the multimodal fusion features of each non-target agent, the following steps are included: Based on the spatial location information of each non-target intelligent agent, the pose offset coordinates of each non-target intelligent agent during the data transmission and reception delay are calculated using a spatiotemporal correction method; the data transmission and reception delay refers to the time difference between the moment when the non-target intelligent agent sends information to the target connected vehicle and the moment when the target connected vehicle receives the information. The change in pose offset coordinates of the non-target intelligent agent during the data transmission delay is determined based on the number of feature channels of the multimodal fusion feature of each non-target intelligent agent and the data transmission delay value; The multimodal fusion features of each non-target agent are compensated based on the pose offset coordinates and the change in the pose offset coordinates to obtain the time-delay compensated multimodal fusion features.

5. The method according to claim 4, characterized in that, The expression for the pose offset coordinates is: ； In the formula, This represents the pose offset coordinates of a non-target intelligent agent during the data transmission and reception delay. The pose coordinates of the non-target agent were not collected at the time the information was sent. R represents the affine transformation matrix of rotation and translation; 11 R 12 R 21 R 22 The sensor representing the non-target agent rotates in four directions; and The sensor representing the non-target agent translates along the X and Y directions; The expression for the change in pose offset coordinates is: ； in, ; In the formula, This indicates the change in pose offset coordinates; Represents a linear mapping layer; represents the data transmission and reception latency; c represents the number of feature channels in the multimodal fusion feature; C represents the maximum number of feature channels in the multimodal fusion feature; This indicates that c is an even value; This indicates that c is an odd number.

6. The method according to claim 4, characterized in that, For each non-target agent, the multimodal fusion features are compressed using a convolutional auto-decoder. The compressed data is then sent to the target connected vehicle and decompressed. Specifically, this includes: An encoder employing a cascaded convolutional structure compresses the multimodal fusion feature along the feature channel dimension; the current multimodal fusion feature is the delay-compensated multimodal fusion feature. The compressed data is sent to the target connected vehicle; After the target connected vehicle receives each of the compressed data, a decoder with a deconvolution structure restores the feature channel dimensions of the compressed data before compression.

7. The method according to claim 1, characterized in that, The expression for the attention weight estimation operation is: ； in, ; ； ； In the formula, This represents the attention weight value of the associated node; i and j are the numbers of the two nodes in a pair of associated nodes; ( ) represents the activation function; () indicates an aggregation operation; The attention weights of the associated nodes obtained by the attention head h are represented; m represents the number of attention heads; MLP() represents a multilayer perceptron, i.e., a linear mapping layer; F i F represents the multimodal fusion feature corresponding to node i in the associated nodes; j This represents the multimodal fusion feature corresponding to node j in the associated nodes; Represents the learnable linear layer parameters; The edge relationship between node i and node j is represented by T; T represents transpose; C represents the maximum value of the feature fusion count of the multimodal fusion feature. The expression for the key point information transmission operation is: ； in, ; In the formula, Attention features representing the edge relationship between node i and node j; The attention feature represents the edge relationship obtained from the attention head h.

8. The method according to claim 1, characterized in that, Applying a multi-scale window attention method to the aggregated features corresponding to each agent yields fused self-attention features, specifically including: For each of the intelligent agents, the aggregated features are divided into multiple windows to obtain multiple window features. For each window, the attention features of each attention head are derived by combining the window features with the visual multi-head self-attention mechanism. For each attention head, the attention features of each attention head in each window are combined to obtain the multi-window attention features of each attention head; The multi-window attention features of each attention head are combined to obtain the multi-head attention features of each attention branch; The multi-head attention features of each attention branch are combined using a separate attention structure to obtain multi-branch attention features, i.e., the fused self-attention features.

Images