Information fusion method and device for vehicle infrastructure cooperation end-to-end system

Abstract

The invention provides an information fusion method and device for a vehicle-road cooperation end-to-end system, and relates to the technical field of automatic driving, and the method comprises the steps: extracting a second target level feature, a second lane level feature, a second scene level feature and a second track level feature through the vehicle-end end-to-end system; after space-time alignment is carried out on the first target level feature and the second target level feature, the first lane level feature and the second lane level feature, the first scene level feature and the second scene level feature, and the first track level feature and the second track level feature, cross-view track feature fusion is carried out; and finally, path planning is carried out based on all the finally obtained fusion features and the second aerial view features. According to the method, space-time alignment and fusion of the target level features are considered, and space-time alignment and fusion of the lane level features, the scene level features and the track level features are also considered, so that the accuracy of final path planning is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to an information fusion method and device of a vehicle-road cooperative end-to-end system. BACKGROUND

[0002] As an important part of the future intelligent transportation architecture, the vehicle-road cooperative end-to-end system is expected to significantly improve the performance of automatic driving by fusing the data of the perception units of the vehicle-side device and the road-side device. However, the field of vehicle-road cooperative automatic driving still lacks interpretable end-to-end system research.

[0003] In related technologies, when the vehicle-road cooperative end-to-end system performs information fusion, the frame data of the vehicle-side device and the frame data of the road-side device are usually target cooperatively fused in the perception module, and then path planning is performed based on the fusion result.

[0004] However, in the above related technology, the target cooperative fusion of the frame data of the vehicle-side device and the road-side device is only performed in the perception module, which causes the loss of other information in addition to the target and the accumulation of errors, thereby reducing the accuracy of the final path planning. SUMMARY

[0005] The present application provides an information fusion method and device of a vehicle-road cooperative end-to-end system to solve the defect of reducing the accuracy of the final path planning in the prior art.

[0006] The present application provides an information fusion method of a vehicle-road cooperative end-to-end system, comprising the following steps.

[0007] Receiving feature information sent by a road-side device, the feature information comprising a first target level feature, a first lane level feature, a first scene level feature and a first trajectory level feature, the feature information being obtained by feature extraction on a first bird's eye view feature corresponding to the road-side device by a road-side end-to-end system, the first bird's eye view feature being obtained based on first frame data collected by the road-side device; Obtaining second frame data collected by a vehicle-side device, and inputting the second frame data, the first target level feature, the first lane level feature, the first scene level feature and the first trajectory level feature into a vehicle-side end-to-end system, obtaining a second bird's eye view feature corresponding to the vehicle-side device based on the second frame data by the vehicle-side end-to-end system, and extracting a second target level feature, a second lane level feature, a second scene level feature and a second trajectory level feature corresponding to the vehicle-side device based on the second bird's eye view feature; After spatiotemporal alignment of the first target level feature and the second target level feature, cross-view target feature fusion is performed to obtain target fusion features; After the first lane level feature and the second lane level feature are spatio-temporally aligned, cross-view lane feature fusion is performed to obtain a lane fusion feature; After the first scene level feature and the second scene level feature are spatio-temporally aligned, cross-view scene feature fusion is performed to obtain a scene fusion feature; After the first trajectory level feature and the second trajectory level feature are spatio-temporally aligned, cross-view trajectory feature fusion is performed to obtain a trajectory fusion feature; The target fusion feature, the lane fusion feature, the scene fusion feature, the trajectory fusion feature and the second bird's eye view feature are used for path planning.

[0008] According to the information fusion method of the vehicle-road cooperative end-to-end system provided by the application, the second target level feature corresponding to the vehicle end device is extracted based on the second bird's eye view feature, including: Through the time sequence perception module in the vehicle end-to-end system, a target level query is generated based on the second bird's eye view feature by using an attention mechanism, and the target level query is used to represent the second target level feature.

[0009] According to the information fusion method of the vehicle-road cooperative end-to-end system provided by the application, the second lane level feature corresponding to the vehicle end device is extracted based on the second bird's eye view feature, including: Through the online mapping module in the vehicle end-to-end system, a lane level query is generated based on the second bird's eye view feature by using an attention mechanism, and the lane level query is used to represent the second lane level feature.

[0010] According to the information fusion method of the vehicle-road cooperative end-to-end system provided by the application, the second scene level feature corresponding to the vehicle end device is extracted based on the second bird's eye view feature, including: Through the grid occupancy module in the vehicle end-to-end system, an occupancy probability map is generated based on the second bird's eye view feature by using an attention mechanism, and the occupancy probability map is used to represent the second scene level feature.

[0011] According to the information fusion method of the vehicle-road cooperative end-to-end system provided by the application, the second trajectory level feature corresponding to the vehicle end device is extracted based on the second bird's eye view feature, including: Through the trajectory prediction module in the vehicle end-to-end system, a trajectory level query is generated based on the second target level feature and the second lane level feature by using an attention mechanism, and the trajectory level query is used to represent the second trajectory level feature.

[0012] According to the information fusion method of the vehicle-road cooperative end-to-end system provided by the application, after the first trajectory level feature and the second trajectory level feature are spatiotemporally aligned, cross-view trajectory feature fusion is performed to obtain trajectory fusion features, comprising: The time sequence information of the same first trajectory level feature is encoded through a self-attention mechanism to obtain encoding information of each first trajectory level feature, and the time sequence information of the same second trajectory level feature is encoded to obtain encoding information of each second trajectory level feature; The first trajectory level feature and the second trajectory level feature are cross-encoded through a cross-attention mechanism to obtain cross-encoding information; The encoding information of the first trajectory level feature and the cross-encoding information are fused through a multi-layer neural network to obtain first fusion encoding information corresponding to the first trajectory level feature, and the encoding information of the second trajectory level feature and the cross-encoding information are fused to obtain second fusion encoding information corresponding to the second trajectory level feature; Based on the position information of the trajectory at the current time, the first fusion encoding information and the second fusion encoding information, Hungarian matching is performed on each first trajectory corresponding to the first trajectory level feature and each second trajectory corresponding to the second trajectory level feature to obtain a spatiotemporally aligned trajectory matching pair; The information of the two trajectories across the view in the trajectory matching pair is fused through a multi-layer neural network to obtain the trajectory fusion features.

[0013] According to the information fusion method of the vehicle-road cooperative end-to-end system provided by the application, the feature information sent by the road end device is received, comprising: The first target level feature sent by the time sequence perception module in the road end-to-end system is received; The first lane level feature sent by the online mapping module in the road end-to-end system is received; The first scene level feature sent by the grid occupancy module in the road end-to-end system is received, the first scene level feature being obtained after the target occupancy probability graph is sparsely encoded and compressed by the grid occupancy module in the road end-to-end system, the target occupancy probability graph being obtained based on the first bird's eye view feature using an attention mechanism; The first trajectory level feature sent by the trajectory prediction module in the road end-to-end system is received.

[0014] According to the information fusion method of the vehicle-road cooperative end-to-end system provided by the application, the vehicle end-to-end system is a vehicle end-to-end system based on a deep learning model, and the vehicle end-to-end system based on the deep learning model is obtained based on the following method: acquire a first sample target level feature, a first sample lane level feature, a first sample scene level feature and a first sample trajectory level feature of a sample road end device; acquire a second sample target level feature, a second sample lane level feature, a second sample scene level feature and a second sample trajectory level feature of a sample vehicle end device; input the first sample target level feature, the first sample lane level feature, the first sample scene level feature, the first sample trajectory level feature, the second sample target level feature, the second sample lane level feature, the second sample scene level feature and the second sample trajectory level feature into an initial deep learning model to obtain sample target fusion features, sample lane fusion features, sample scene fusion features and sample trajectory fusion features output by the initial deep learning model in a cross-view spatio-temporal alignment; construct a target loss based on the sample target fusion features and a target true value, construct a lane loss based on the sample lane fusion features and a lane true value, construct a scene loss based on the sample scene fusion features and a scene true value, construct a trajectory loss based on the sample trajectory fusion features and a trajectory true value, and construct a path planning loss based on a final path planning result and a planning true value; determine a joint loss based on the target loss, the lane loss, the scene loss, the trajectory loss and the path planning loss; iteratively update network parameters of the initial deep learning model based on the joint loss until a convergence condition is reached to obtain a vehicle end deep learning model based end-to-end system.

[0015] According to the information fusion method of the vehicle-road cooperative end-to-end system provided by the application, in the training process, the feature information of the road end device and the feature information of the vehicle end device are aligned in the time dimension through query stream learning, and the feature information of the road end device and the feature information of the vehicle end device are aligned in the space dimension based on learning of frame level cooperative conversion matrix parameters.

[0016] The application further provides an information fusion device of a vehicle-road cooperative end-to-end system, comprising: a receiving unit configured to receive feature information sent by a road end device, wherein the feature information comprises a first target level feature, a first lane level feature, a first scene level feature and a first trajectory level feature, the feature information is obtained by performing feature extraction on a first bird's eye view feature corresponding to the road end device in a road end end-to-end system, and the first bird's eye view feature is obtained based on first frame data collected by the road end device; an acquisition unit configured to acquire second frame data collected by a vehicle end device; The feature extraction unit is configured to input the second frame data, the first target level feature, the first lane level feature, the first scene level feature and the first track level feature into a vehicle end-to-end system, obtain a second bird's eye view feature corresponding to the vehicle end device based on the second frame data through the vehicle end-to-end system, and extract a second target level feature, a second lane level feature, a second scene level feature and a second track level feature corresponding to the vehicle end device based on the second bird's eye view feature. The first fusion unit is configured to perform spatio-temporal alignment on the first target level feature and the second target level feature, and then perform cross-view target feature fusion to obtain a target fusion feature. The second fusion unit is configured to perform spatio-temporal alignment on the first lane level feature and the second lane level feature, and then perform cross-view lane feature fusion to obtain a lane fusion feature. The third fusion unit is configured to perform spatio-temporal alignment on the first scene level feature and the second scene level feature, and then perform cross-view scene feature fusion to obtain a scene fusion feature. The fourth fusion unit is configured to perform spatio-temporal alignment on the first track level feature and the second track level feature, and then perform cross-view track feature fusion to obtain a track fusion feature. The target fusion feature, the lane fusion feature, the scene fusion feature, the track fusion feature and the second bird's eye view feature are used for path planning.

[0017] The application further provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the information fusion method of the vehicle-road cooperative end-to-end system according to any one of the above.

[0018] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the information fusion method of the vehicle-road cooperative end-to-end system according to any one of the above.

[0019] The application further provides a computer program product, which includes a computer program, and the computer program is executable on a processor to implement the information fusion method of the vehicle-road cooperative end-to-end system according to any one of the above.

[0020] The application provides a vehicle-road cooperative end-to-end system information fusion method and device, which receives first target level features, first lane level features, first scene level features and first trajectory level features sent by a road end device, acquires second frame data collected by a vehicle end device, and inputs the second frame data, the first target level features, the first lane level features, the first scene level features and the first trajectory level features into a vehicle end end-to-end system, obtains second bird's eye view features corresponding to the vehicle end device based on the second frame data through the vehicle end end-to-end system, and extracts second target level features, second lane level features, second scene level features and second trajectory level features corresponding to the vehicle end device based on the second bird's eye view features; after the first target level features and the second target level features are spatio-temporally aligned, cross-view target feature fusion is performed to obtain target fusion features; after the first lane level features and the second lane level features are spatio-temporally aligned, cross-view lane feature fusion is performed to obtain lane fusion features; after the first scene level features and the second scene level features are spatio-temporally aligned, cross-view scene feature fusion is performed to obtain scene fusion features; after the first trajectory level features and the second trajectory level features are spatio-temporally aligned, cross-view trajectory feature fusion is performed to obtain trajectory fusion features, and finally, path planning is performed based on the target fusion features, the lane fusion features, the scene fusion features, the trajectory fusion features and the second bird's eye view features. It can be known that, in addition to considering spatio-temporal alignment and fusion of target level features, the application also considers spatio-temporal alignment and fusion of lane level features, scene level features and trajectory level features, so that the accuracy of final path planning is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0022] Figure 1 is one of the flowcharts of the vehicle-road cooperative end-to-end system information fusion method provided by the embodiments of the application.

[0023] Figure 2 is the second flowchart of the vehicle-road cooperative end-to-end system information fusion method provided by the embodiments of the application.

[0024] Figure 3 is the training flowchart of the vehicle end end-to-end system based on the deep learning model provided by the embodiments of the application.

[0025] Figure 4 is the overall framework diagram of the vehicle-road cooperative end-to-end system information fusion method provided by the embodiments of the application.

[0026] Figure 5 This is a schematic diagram of the information fusion device of the vehicle-road cooperative end-to-end system provided in an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The following is combined with Figures 1-4 This invention describes an information fusion method for a vehicle-road cooperative end-to-end system. The execution entity of this information fusion method can be an electronic device such as a computer, tablet computer, terminal, or server, or it can be an information fusion device for the vehicle-road cooperative end-to-end system installed in such an electronic device. This information fusion device can be implemented through software, hardware, or a combination of both.

[0030] Figure 1 This is one of the flowcharts illustrating the information fusion method for a vehicle-road cooperative end-to-end system provided in this embodiment of the invention, such as... Figure 1 As shown, the information fusion method of this vehicle-road cooperative end-to-end system includes the following steps: Step 101: Receive feature information sent by the roadside device. The feature information includes a first target level feature, a first lane level feature, a first scene level feature, and a first trajectory level feature. The feature information is obtained by the roadside end-to-end system extracting features from the first bird's-eye view feature corresponding to the roadside device. The first bird's-eye view feature is obtained based on the first frame of data collected by the roadside device.

[0031] For example, the road end-to-end system is a road end-to-end system based on a deep learning model. The deep learning model can be a Transformer architecture. The first frame data collected by the road end device is input into the road end-to-end system based on the deep learning model. The first Bird's Eye View (BEV) feature is generated based on the first frame data by the road end-to-end system based on the deep learning model. The first BEV feature is extracted to obtain the first target level feature, the first lane level feature, the first scene level feature, and the first trajectory level feature output by the road end-to-end system based on the deep learning model. The first target level feature, the first lane level feature, the first scene level feature, and the first trajectory level feature are sent to the vehicle end device. The road end device can be a device installed along the road, such as a camera, radar, roadside sensor, etc. for collecting road conditions, surrounding environment information, and vehicle driving conditions, etc. The first frame data refers to the data collected by the road end device at a certain time. The first frame data can be an image collected by a camera installed on the roadside, or a group of data collected by a sensor or radar installed on the roadside. The first target level feature includes target type, target appearance feature, and target size, etc. The first lane level feature includes lane type, lane width, lane boundary line, and lane marking, etc. The first scene level feature includes all surrounding scene information, including vehicles, personnel, buildings, etc. The first trajectory level feature includes trajectory information of vehicles or personnel around the vehicle end device collected by the road section device, etc.

[0032] It should be noted that the road end-to-end system based on the deep learning model is obtained by training in the following manner: obtaining first sample frame data collected by a sample road end device, inputting the first sample frame data into an initial deep learning model, obtaining first sample BEV features by the initial deep learning model, and outputting first predicted target level features, first predicted lane level features, first predicted scene level features, and first predicted trajectory level features based on the first sample BEV features. A loss function is constructed based on the errors between the first predicted target level features and the target true value, the errors between the first predicted lane level features and the lane true value, the errors between the first predicted scene level features and the scene true value, and the errors between the first predicted trajectory level features and the trajectory true value. The network parameters of the initial deep learning model are iteratively updated based on the loss function until a convergence condition is reached to obtain the road end-to-end system based on the deep learning model.

[0033] Step 102, acquiring second frame data collected by the vehicle-side device, and inputting the second frame data, the first target-level feature, the first lane-level feature, the first scene-level feature, and the first trajectory-level feature into the vehicle-side end-to-end system, obtaining the second bird's eye view feature corresponding to the vehicle-side device based on the second frame data through the vehicle-side end-to-end system, and extracting the second target-level feature, the second lane-level feature, the second scene-level feature, and the second trajectory-level feature corresponding to the vehicle-side device based on the second bird's eye view feature.

[0034] The vehicle-side device can be a sensor, a camera, a radar, or the like installed on a vehicle, used to collect information about the running state of the vehicle and the environment around the vehicle. The second frame data refers to data collected by the vehicle-side device at a certain time, which can be an image collected by a camera or a group of data collected by a sensor or a radar.

[0035] For example, the vehicle-side end-to-end system is a vehicle-side end-to-end system based on a deep learning model. The deep learning model can adopt a Transformer architecture or other architectures capable of extracting query. The second frame data collected by the vehicle-side device is input into the vehicle-side end-to-end system based on the deep learning model. The second bird's eye view feature is generated based on the second frame data through the vehicle-side end-to-end system based on the deep learning model, and the second bird's eye view feature is extracted to obtain the second target-level feature, the second lane-level feature, the second scene-level feature, and the second trajectory-level feature output by the vehicle-side end-to-end system based on the deep learning model.

[0036] Step 103, after spatio-temporal alignment of the first target-level feature and the second target-level feature, cross-view target feature fusion is performed to obtain target fusion features.

[0037] For example, the first target-level feature also needs to be input into the vehicle-side end-to-end system. After spatio-temporal alignment of each first target-level feature and each second target-level feature through the vehicle-side end-to-end system, cross-view target feature fusion is performed to obtain target fusion features.

[0038] Step 104, after spatio-temporal alignment of the first lane-level feature and the second lane-level feature, cross-view lane feature fusion is performed to obtain lane fusion features.

[0039] For example, the first lane-level feature also needs to be input into the vehicle-side end-to-end system. After spatio-temporal alignment of each first lane-level feature and each second lane-level feature through the vehicle-side end-to-end system, cross-view lane feature fusion is performed to obtain lane fusion features.

[0040] Step 105, after the spatio-temporal alignment of the first scene-level features and the second scene-level features, cross-view scene feature fusion is performed to obtain scene fusion features.

[0041] For example, the first scene-level features are also needed to be input into the vehicle-side end-to-end system, and after the spatio-temporal alignment of each first scene-level feature and each second scene-level feature by the vehicle-side end-to-end system, cross-view scene feature fusion is performed to obtain scene fusion features.

[0042] Step 106, after the spatio-temporal alignment of the first trajectory-level features and the second trajectory-level features, cross-view trajectory feature fusion is performed to obtain trajectory fusion features; wherein the target fusion features, the lane fusion features, the scene fusion features, the trajectory fusion features and the second bird's eye view features are used for path planning.

[0043] For example, the first trajectory-level features are also needed to be input into the vehicle-side end-to-end system, and after the spatio-temporal alignment of each first trajectory-level feature and each second trajectory-level feature by the vehicle-side end-to-end system, cross-view trajectory feature fusion is performed to obtain trajectory fusion features. After obtaining the target fusion features, the lane fusion features, the scene fusion features and the trajectory fusion features, comprehensive analysis is performed on the target fusion features, the lane fusion features, the scene fusion features, the trajectory fusion features and the second bird's eye view features corresponding to the vehicle-side device to obtain path planning information.

[0044] The information fusion method of the vehicle-road cooperative end-to-end system provided by the application receives first target level features, first lane level features, first scene level features and first trajectory level features sent by a road end device, acquires second frame data collected by a vehicle end device, and inputs the second frame data, the first target level features, the first lane level features, the first scene level features and the first trajectory level features into a vehicle end end-to-end system, obtains second bird's eye view features corresponding to the vehicle end device based on the second frame data through the vehicle end end-to-end system, and extracts second target level features, second lane level features, second scene level features and second trajectory level features corresponding to the vehicle end device based on the second bird's eye view features; after the first target level features and the second target level features are spatiotemporally aligned, cross-view target feature fusion is performed, and target fusion features are obtained; after the first lane level features and the second lane level features are spatiotemporally aligned, cross-view lane feature fusion is performed, and lane fusion features are obtained; after the first scene level features and the second scene level features are spatiotemporally aligned, cross-view scene feature fusion is performed, and scene fusion features are obtained; after the first trajectory level features and the second trajectory level features are spatiotemporally aligned, cross-view trajectory feature fusion is performed, and trajectory fusion features are obtained, and finally, path planning is performed based on the target fusion features, the lane fusion features, the scene fusion features, the trajectory fusion features and the second bird's eye view features. It can be known that, in addition to considering the spatiotemporal alignment and fusion of target level features, the application also considers the spatiotemporal alignment and fusion of lane level features, scene level features and trajectory level features, so that the accuracy of final path planning is improved.

[0045] In an embodiment, the step of extracting the second target level features corresponding to the vehicle end device based on the second bird's eye view features in the step 102 can be implemented in the following manner: The time sequence perception module in the vehicle end end-to-end system generates target level queries based on the second bird's eye view features by using an attention mechanism, and the target level queries are used to represent the second target level features.

[0046] For example, in a vehicle end end-to-end system based on a deep learning model, the time sequence perception module utilizes the second bird's eye view features, which describe the overhead view information of the environment around the vehicle of the vehicle end device, analyzes the second bird's eye view features by applying an attention mechanism, generates target level queries (object queries), represents the second target level features through the object queries, and can capture and highlight key targets in the environment around the vehicle, such as other vehicles, pedestrians or obstacles, so as to provide detailed and dynamic feature descriptions of the surrounding targets for the vehicle.

[0047] In this embodiment, the attention mechanism is used to generate a lane level query for representing the second lane level feature based on the second bird's eye view feature. The attention mechanism can dynamically focus on the most important part of the second bird's eye view feature, so that the generated second lane level feature is more accurate.

[0048] In an embodiment, the second lane level feature corresponding to the vehicle terminal device is extracted based on the second bird's eye view feature in step 102, which can be implemented in the following way: The lane level query is generated by the online mapping module in the vehicle terminal end-to-end system based on the second bird's eye view feature using the attention mechanism, and the lane level query is used to represent the second lane level feature.

[0049] For example, in a vehicle terminal end-to-end system based on a deep learning model, the online mapping module uses second bird's eye view features, which describe the overhead view information of the environment around the vehicle where the vehicle terminal device is located. By applying the attention mechanism, it can identify and focus on the features that are crucial for lane recognition and mapping, and generate a lane level query (lane query). Through the lane query, the second lane level features such as the geometric properties, topological structure and dynamic changes of the lane can be accurately represented, so as to realize accurate modeling and understanding of the vehicle driving environment.

[0050] In this embodiment, the attention mechanism is used to generate a lane level query for representing the second lane level feature based on the second bird's eye view feature. The attention mechanism can focus on the features that are crucial for lane recognition and mapping, so that the generated second lane level feature is more accurate.

[0051] In an embodiment, the second lane level feature corresponding to the vehicle terminal device is extracted based on the second bird's eye view feature in step 102, which can be implemented in the following way: The occupied probability map is generated by the grid occupancy module in the vehicle terminal end-to-end system based on the second bird's eye view feature using the attention mechanism, and the occupied probability map is used to represent the second scene level feature.

[0052] For example, in a vehicle terminal end-to-end system based on a deep learning model, the grid occupancy module uses second bird's eye view features, which describe the overhead view information of the environment around the vehicle where the vehicle terminal device is located. By applying the attention mechanism, it can identify and emphasize key areas, and generate an occupied probability map (occupied probability map).

[0053] In this embodiment, the attention mechanism is adopted to generate an occupancy probability map for representing the second scene-level feature based on the second bird's eye view feature, the occupancy probability map quantifies the possibility of each grid being occupied by an object, comprehensively describes the traffic environment and possible obstacle distribution where the vehicle is located, so that the generated second scene-level feature is more accurate.

[0054] In an embodiment, the second trajectory-level feature corresponding to the vehicle terminal device is extracted based on the second bird's eye view feature in step 102, which can be realized by the following way: The trajectory prediction module in the vehicle terminal end-to-end system generates trajectory-level queries for representing the second trajectory-level feature based on the second target-level feature and the second lane-level feature by adopting the attention mechanism.

[0055] For example, in the vehicle terminal end-to-end system based on the deep learning model, the trajectory prediction module comprehensively utilizes the second target-level feature obtained by the time-series perception module and the second lane-level feature obtained by the online mapping module, and generates trajectory-level queries by applying the attention mechanism to identify and focus on the information most critical to trajectory prediction. These trajectory-level queries can accurately represent the possible movement trajectories of the vehicle and surrounding targets in a future period of time, i.e., the second trajectory-level feature.

[0056] In this embodiment, the attention mechanism is adopted to generate trajectory-level queries for representing the second trajectory-level feature based on the second target-level feature and the second lane-level feature, and these trajectory-level queries can accurately represent the possible movement trajectories of the surrounding targets or surrounding vehicles in a future period of time, thereby providing accurate trajectory prediction for the vehicle of autonomous driving, helping the vehicle to effectively plan the path and avoid obstacles, and ensuring the safety and efficiency of driving.

[0057] In an embodiment, Figure 2 is a flowchart of the information fusion method of the vehicle-road cooperative end-to-end system provided by the embodiment of the present application, as shown in Figure 2 After the spatio-temporal alignment of the first trajectory-level feature and the second trajectory-level feature in step 103, cross-view trajectory feature fusion is performed to obtain trajectory fusion features, which can be realized by the following way: Step 201: encode the time-series information of the same first trajectory-level feature by the self-attention mechanism to obtain the encoding information of each first trajectory-level feature, and encode the time-series information of the same second trajectory-level feature to obtain the encoding information of each second trajectory-level feature.

[0058] For example, the time sequence information of the same first trajectory level feature is encoded by using the self-attention mechanism, that is, the information of each time point in the same first trajectory level feature is analyzed and processed by using the self-attention mechanism, the relationship and dependency between different time points in the same first trajectory level feature are captured, and the encoding information of each first trajectory level feature is obtained. Similarly, the time sequence information of the same second trajectory level feature is encoded by using the self-attention mechanism, that is, the information of each time point in the same second trajectory level feature is analyzed and processed by using the self-attention mechanism, the relationship and dependency between different time points in the same second trajectory level feature are captured, and the encoding information of each second trajectory level feature is obtained.

[0059] In step 202, the first trajectory level feature and the second trajectory level feature are cross-encoded by using the cross-attention mechanism to obtain cross-encoding information.

[0060] For example, the first trajectory level feature collected by the vehicle-side device and the second trajectory level feature collected by the road-side device are cross-encoded by using the cross-attention mechanism, that is, the cross-attention mechanism is used to simultaneously consider the two types of trajectory information, analyze and capture the mutual relationship and interaction between the first trajectory level feature collected by the vehicle-side device and the second trajectory level feature collected by the road-side device, and cross-encoding information fused with the perspectives of the vehicle-side device and the road-side device is obtained.

[0061] In step 203, the encoding information of the first trajectory level feature and the cross-encoding information are fused by using the multi-layer neural network to obtain the first fusion encoding information corresponding to the first trajectory level feature, and the encoding information of the second trajectory level feature and the cross-encoding information are fused to obtain the second fusion encoding information corresponding to the second trajectory level feature.

[0062] For example, the encoding information and the cross-encoding information of the first trajectory level feature collected by the vehicle-side device are further fused by using the multi-layer neural network, the multi-layer structure of the multi-layer neural network can learn the complex relationship and interaction mode between the encoding information, and thus the first fusion encoding information corresponding to the first trajectory level feature is obtained. Meanwhile, the encoding information and the cross-encoding information of the second trajectory level feature collected by the road-side device are further fused by using the multi-layer neural network, the multi-layer structure of the multi-layer neural network can learn the complex relationship and interaction mode between the encoding information, and thus the second fusion encoding information corresponding to the second trajectory level feature is obtained.

[0063] In step 204, based on the position information of the current time of the trajectory, the first fusion encoding information and the second fusion encoding information, the first trajectory corresponding to each first trajectory level feature and the second trajectory corresponding to each second trajectory level feature are subjected to the Hungarian matching to obtain the spatio-temporal aligned trajectory matching pair.

[0064] For example, based on the position information of the trajectories at the current time, the first trajectories corresponding to each first trajectory-level feature and the second trajectories corresponding to each second trajectory-level feature are optimally matched based on the position information of the trajectories at the current time, the first fusion encoding information and the second fusion encoding information by using a Hungarian matching algorithm. The Hungarian matching algorithm finds a matching scheme that minimizes the total matching cost by calculating and comparing the similarities or distances between different fusion encoding information, thereby obtaining the spatio-temporally aligned trajectory matching pairs, i.e., ensuring that each first trajectory corresponding to each first trajectory-level feature can be accurately matched with a second trajectory corresponding to a corresponding second trajectory-level feature, and achieving time and space synchronization between trajectories.

[0065] Step 205: fusing, by a multi-layer neural network, the two trajectory information across the perspectives in the trajectory matching pair to obtain the trajectory fusion feature.

[0066] For example, the two trajectory information from different perspectives (vehicle-end device and road-end device) in the trajectory matching pair are deeply fused by a multi-layer neural network. The multi-layer neural network can learn and extract complementary features and complex correlations in the two trajectory information across the perspectives through its multi-layer structure, thereby obtaining a trajectory fusion feature that integrates the vehicle-end and road-end perspectives, and inputting the trajectory fusion feature into a trajectory prediction module for trajectory prediction. The trajectory fusion feature not only contains the geometric and dynamic information of a single perspective trajectory, but also fuses the supplementary information from another perspective. Therefore, the trajectory fusion feature can more comprehensively and accurately represent the motion state and interaction relationship of the trajectory, and provide rich and reliable feature support for environmental perception, trajectory prediction and decision planning of an autonomous vehicle.

[0067] In an embodiment, the step 101 receives the feature information sent by the road-end device, which can be implemented in the following manner: receiving the first target-level feature sent by the time-series perception module in the road-end end-to-end system; receiving the first lane-level feature sent by the online mapping module in the road-end end-to-end system; receiving the first scene-level feature sent by the grid occupancy module in the road-end end-to-end system, the first scene-level feature being obtained by sparse encoding and compression of a target occupancy probability map by the grid occupancy module in the road-end end-to-end system, the target occupancy probability map being generated based on the first bird's-eye view feature using an attention mechanism; and receiving the first trajectory-level feature sent by the trajectory prediction module in the road-end end-to-end system.

[0068] For example, in the hybrid transmission mode, the vehicle end-to-end system receives different types of information sent by multiple modules of the road end-to-end system, including the first target level feature sent by the timing perception module, which reflects the identification and tracking results of the target by the road end-to-end system; the first lane level feature sent by the online mapping module, which provides lane information constructed by the road end-to-end system; the first scene level feature sent by the grid occupancy module, which is the result of sparse coding and compression of the target occupancy probability map generated based on the first bird's eye view feature through the attention mechanism, and is used to represent the spatial distribution of the target in the scene; and the first trajectory level feature sent by the trajectory prediction module, which depicts the predicted trajectory data of the road end-to-end system.

[0069] In this embodiment, in the hybrid transmission mode, the timing perception module transmits the target level feature, the online mapping module transmits the lane level feature, and the trajectory prediction module transmits the trajectory level feature, which are more sparse than the common bird's eye view feature; the grid occupancy module transmits the scene level feature, which guarantees the accuracy of the feature under a certain sparsity. This hybrid transmission mode of sparsity and density greatly reduces the data transmission amount on the basis of retaining effective information, and the transmitted information also has interpretability, thereby reducing the communication cost while guaranteeing the accuracy of the feature.

[0070] In an embodiment, the vehicle end-to-end system is a deep learning model-based vehicle end-to-end system, Figure 3 is a training process schematic diagram of the deep learning model-based vehicle end-to-end system provided by the embodiment of the present application, as Figure 3 shown, the deep learning model-based vehicle end-to-end system is trained based on the following way: Step 301, obtaining the first sample target level feature, the first sample lane level feature, the first sample scene level feature and the first sample trajectory level feature of the sample road end device.

[0071] For example, the deep learning model-based road end-to-end system generates the first sample bird's eye view feature based on the first sample frame data collected by the sample road end device, and extracts features from the first sample bird's eye view feature to obtain the first sample target level feature, the first sample lane level feature, the first sample scene level feature and the first sample trajectory level feature output by the deep learning model-based road end-to-end system, and sends the first sample target level feature, the first sample lane level feature, the first sample scene level feature and the first sample trajectory level feature to the sample vehicle end device.

[0072] Step 302, obtaining the second sample target level feature, the second sample lane level feature, the second sample scene level feature and the second sample trajectory level feature of the sample vehicle end device.

[0073] For example, the second sample bird's eye view feature is generated based on the second sample frame data collected by the sample vehicle terminal device by the vehicle terminal end-to-end system based on the deep learning model, and feature extraction is performed on the second sample bird's eye view feature to obtain the second sample target level feature, the second sample lane level feature, the second sample scene level feature and the second sample trajectory level feature output by the vehicle terminal end-to-end system based on the deep learning model.

[0074] Step 303, input the first sample target level feature, the first sample lane level feature, the first sample scene level feature, the first sample trajectory level feature, the second sample target level feature, the second sample lane level feature, the second sample scene level feature and the second sample trajectory level feature into the initial deep learning model to obtain the sample target fusion feature, the sample lane fusion feature, the sample scene fusion feature and the sample trajectory fusion feature output by the initial deep learning model.

[0075] For example, the first sample target level feature, the first sample lane level feature, the first sample scene level feature, the first sample trajectory level feature, the second sample target level feature, the second sample lane level feature, the second sample scene level feature and the second sample trajectory level feature are input into the initial deep learning model, and the initial deep learning model utilizes its feature extraction and fusion capabilities to perform deep analysis and integration on these cross-view spatio-temporal data, and finally outputs the sample target fusion feature, the sample lane fusion feature, the sample scene fusion feature and the sample trajectory fusion feature.

[0076] Step 304, based on the sample target fusion feature and the target true value, a target loss is constructed, based on the sample lane fusion feature and the lane true value, a lane loss is constructed, based on the sample scene fusion feature and the scene true value, a scene loss is constructed, based on the sample trajectory fusion feature and the trajectory true value, a trajectory loss is constructed, and based on the final path planning result and the planning true value, a path planning loss is constructed.

[0077] For example, in the process of training the deep learning model, in order to optimize the fusion ability of the deep learning model for different dimensional features such as targets, lanes, scenes and trajectories, based on the differences between the sample target fusion features output from the initial deep learning model and the actual target ground truth, the sample lane fusion features and the actual lane ground truth, the sample scene fusion features and the actual scene ground truth, the sample trajectory fusion features and the actual trajectory ground truth, and the final path planning result and the planning ground truth, target loss, lane loss, scene loss, trajectory loss and path planning loss are respectively constructed. These loss functions measure the deviation between the model prediction and the actual situation. By minimizing these losses during training, the model can learn to more accurately fuse and represent feature information from different perspectives, thereby improving the model's performance in target detection, lane recognition, scene understanding, trajectory prediction and path planning tasks.

[0078] Step 305, based on the target loss, the lane loss, the scene loss, the trajectory loss and the path planning loss, determine the joint loss.

[0079] For example, in the process of training the deep learning model, in order to comprehensively evaluate and optimize the performance of the model on multiple tasks, the target loss, lane loss, scene loss, trajectory loss and path planning loss, which are five loss functions respectively for target detection, lane recognition, scene understanding, trajectory prediction and path planning tasks, are weighted and summed or fused to determine a joint loss. This joint loss reflects the overall performance of the model on all related tasks. By minimizing the joint loss during training, the model can simultaneously learn to effectively fuse and represent multi-dimensional features such as targets, lanes, scenes, trajectories and path planning, thereby improving performance on multiple related tasks.

[0080] Step 306, based on the joint loss, iteratively update the network parameters of the initial deep learning model until the convergence condition is reached, to obtain the end-to-end system based on the deep learning model.

[0081] For example, in the process of training the deep learning model, based on the calculated joint loss, the network parameters of the initial deep learning model are iteratively updated using optimization algorithms such as gradient descent, i.e. adjusting parameter values according to the gradient of the joint loss with respect to the parameters to reduce the loss. Through multiple rounds of iterative training, the network parameters of the initial deep learning model are continuously optimized until the performance of the initial deep learning model no longer significantly improves or reaches a preset convergence condition, such as the loss function value no longer significantly decreases or reaches a certain number of iterations. Finally, the end-to-end system based on the deep learning model is obtained.

[0082] It should be noted that the vehicle end-to-end system based on the deep learning model needs to be trained based on the second sample frame data collected by the sample vehicle end device before being trained based on the joint loss. The specific process is as follows: the second sample frame data is input into the initial deep learning model, the second sample bird's eye view feature is obtained through the initial deep learning model, and the second predicted target level feature, the second predicted lane level feature, the second predicted scene level feature and the second predicted trajectory level feature are output based on the second sample bird's eye view feature. The loss function is constructed based on the error between the second predicted target level feature and the target true value, the error between the second predicted lane level feature and the lane true value, the error between the second predicted scene level feature and the scene true value, and the error between the second predicted trajectory level feature and the trajectory true value. The network parameters of the initial deep learning model are iteratively updated based on the loss function until the convergence condition is reached, and the vehicle end-to-end system based on the deep learning model is obtained.

[0083] It should be noted that when the network parameters of the initial deep learning model are iteratively updated based on the loss function, the network parameters related to the network layer of the second sample bird's eye view feature can be fixed, and only the network parameters of the network layer related to the second predicted target level feature, the second predicted lane level feature, the second predicted scene level feature and the second predicted trajectory level feature are iteratively updated.

[0084] In this embodiment, the network parameters of the initial deep learning model are iteratively updated based on the joint loss in the shared representation space, with the global optimum as the optimization target, to train the vehicle end-to-end system based on the deep learning model, so as to achieve the optimal overall performance of the vehicle end-to-end system based on the deep learning model, and to comprehensively utilize the advantages of multi-source data and multi-task learning to achieve accurate, efficient and robust perception, prediction and decision-making in complex traffic environments.

[0085] In an embodiment, in the training process, the feature information of the road end device and the feature information of the vehicle end device are aligned in the time dimension through query stream learning, and are aligned in the space dimension based on the learning of the frame-level collaborative conversion matrix parameters.

[0086] By way of example, the present application designs a cache scheduling mechanism supporting asynchronous / delay feature alignment, introduces a buffering and compensation mechanism to address the communication delay asynchronization and spatial misalignment between the vehicle-side device and the road-side device, and realizes cross-frame and cross-view data synchronization and fusion. In the training process of the deep learning model, the Query flow learning method is used to realize the alignment of the feature information from the road-side device and the vehicle-side device in the time dimension, i.e., the time synchronization technology is used to ensure the consistency of the two types of feature information in the time dimension, so as to accurately reflect the traffic environment state at the same time; at the same time, based on the learning of the frame-level collaborative conversion matrix parameters, the alignment of the two types of feature information in the spatial dimension is further realized, i.e., a conversion matrix is learned to align the positions of the feature information collected by different devices in the spatial coordinate system, so as to accurately reflect the traffic environment features at the same spatial position. By aligning the feature information of the road-side device and the vehicle-side device in both the time and spatial dimensions, the vehicle-side end-to-end system based on the deep learning model can more accurately fuse and utilize the data from different devices, and improve the perception and understanding ability of the vehicle-side end-to-end system based on the deep learning model for the traffic environment.

[0087] Figure 4 is the overall framework diagram of the information fusion method of the vehicle-road cooperative end-to-end system provided by the embodiment of the present application, as shown in Figure 4 the road-side end-to-end system and the vehicle-side end-to-end system, wherein the road-side end-to-end system and the vehicle-side end-to-end system each include a time sequence perception module, an online mapping module, a grid occupancy module and a trajectory prediction module, and the vehicle-side end-to-end system further includes a planning module. The time sequence perception module in the road-side end-to-end system sends the first target-level feature obtained based on the first bird's eye view feature to the vehicle-side end-to-end system, the online mapping module in the road-side end-to-end system sends the first lane-level feature obtained based on the first bird's eye view feature to the vehicle-side end-to-end system, the grid occupancy module in the road-side end-to-end system sends the first scene-level feature obtained based on the first bird's eye view feature and compressed by sparse coding to the vehicle-side end-to-end system, and the trajectory prediction module in the road-side end-to-end system sends the first trajectory-level feature obtained based on the first target-level feature and the first lane-level feature to the vehicle-side end-to-end system.

[0088] The timing awareness module in the vehicle-end-to-end system obtains second target level features based on the second bird's eye view features, and after spatio-temporal alignment of the first target level features and the second target level features, cross-view target feature fusion is performed to obtain target fusion features; the online mapping module in the vehicle-end-to-end system obtains second lane level features based on the second bird's eye view features, and after spatio-temporal alignment of the first lane level features and the second lane level features, cross-view lane feature fusion is performed to obtain lane fusion features; the grid occupancy module in the vehicle-end-to-end system obtains second scene level features based on the second bird's eye view features, and after spatio-temporal alignment of the first scene level features and the second scene level features, cross-view scene feature fusion is performed to obtain scene fusion features; the trajectory prediction module in the vehicle-end-to-end system obtains second trajectory level features based on the second target level features and the second lane level features, and after spatio-temporal alignment of the first trajectory level features and the second trajectory level features, cross-view trajectory feature fusion is performed to obtain trajectory fusion features; the planning module performs path planning based on the target fusion features, the lane fusion features, the scene fusion features, the trajectory fusion features, and the second bird's eye view features.

[0089] It should be noted that the information fusion method of the vehicle-road cooperative end-to-end system provided by the present application can also be applied in the vehicle-vehicle cooperative field, and the present application does not limit this.

[0090] The embodiment of the present application provides an information fusion method of a vehicle-road cooperative end-to-end system, designs a sparse-dense hybrid transmission mode, greatly reduces the data transmission amount on the basis of retaining effective information, and the transmitted information also has interpretability. Meanwhile, in the information fusion, spatio-temporal alignment learning is performed on the features of the road-end device and the features of the vehicle-end device, cross-view data information effective fusion is realized, and finally the vehicle-road cooperative end-to-end system is realized.

[0091] The information fusion device of the vehicle-road cooperative end-to-end system provided by the present application is described below, and the information fusion device of the vehicle-road cooperative end-to-end system described below can be correspondingly referred to the information fusion method of the vehicle-road cooperative end-to-end system described above.

[0092] Figure 5 is a structural schematic diagram of the information fusion device of the vehicle-road cooperative end-to-end system provided by the embodiment of the present application, as Figure 5 shown, the information fusion device 500 of the vehicle-road cooperative end-to-end system includes a receiving unit 501, an obtaining unit 502, a feature extraction unit 503, a first fusion unit 504, a second fusion unit 505, a third fusion unit 506, and a fourth fusion unit 507; The receiving unit 501 is configured to receive characteristic information sent by the road terminal device, wherein the characteristic information comprises a first target level feature, a first lane level feature, a first scene level feature and a first trajectory level feature, the characteristic information is obtained by performing feature extraction on a first bird's eye view feature corresponding to the road terminal device in a road terminal end-to-end system, and the first bird's eye view feature is obtained based on first frame data collected by the road terminal device; The acquisition unit 502 is configured to acquire second frame data collected by the vehicle terminal device. The feature extraction unit 503 is configured to input the second frame data, the first target level feature, the first lane level feature, the first scene level feature and the first trajectory level feature into a vehicle terminal end-to-end system, obtain a second bird's eye view feature corresponding to the vehicle terminal device based on the second frame data by the vehicle terminal end-to-end system, and extract a second target level feature, a second lane level feature, a second scene level feature and a second trajectory level feature corresponding to the vehicle terminal device based on the second bird's eye view feature. The first fusion unit 504 is configured to perform spatio-temporal alignment on the first target level feature and the second target level feature, and then perform cross-view target feature fusion to obtain target fusion features. The second fusion unit 505 is configured to perform spatio-temporal alignment on the first lane level feature and the second lane level feature, and then perform cross-view lane feature fusion to obtain lane fusion features. The third fusion unit 506 is configured to perform spatio-temporal alignment on the first scene level feature and the second scene level feature, and then perform cross-view scene feature fusion to obtain scene fusion features. The fourth fusion unit 507 is configured to perform spatio-temporal alignment on the first trajectory level feature and the second trajectory level feature, and then perform cross-view trajectory feature fusion to obtain trajectory fusion features. The target fusion features, the lane fusion features, the scene fusion features, the trajectory fusion features and the second bird's eye view feature are used for path planning.

[0093] The information fusion device of the vehicle-road cooperative end-to-end system provided by the application receives first target level features, first lane level features, first scene level features and first trajectory level features sent by a road end device, acquires second frame data collected by a vehicle end device, and inputs the second frame data, the first target level features, the first lane level features, the first scene level features and the first trajectory level features into a vehicle end end-to-end system, obtains second bird's eye view features corresponding to the vehicle end device based on the second frame data through the vehicle end end-to-end system, and extracts second target level features, second lane level features, second scene level features and second trajectory level features corresponding to the vehicle end device based on the second bird's eye view features; after the first target level features and the second target level features are spatio-temporally aligned, cross-view target feature fusion is performed to obtain target fusion features; after the first lane level features and the second lane level features are spatio-temporally aligned, cross-view lane feature fusion is performed to obtain lane fusion features; after the first scene level features and the second scene level features are spatio-temporally aligned, cross-view scene feature fusion is performed to obtain scene fusion features; after the first trajectory level features and the second trajectory level features are spatio-temporally aligned, cross-view trajectory feature fusion is performed to obtain trajectory fusion features, and finally, path planning is performed based on the target fusion features, the lane fusion features, the scene fusion features, the trajectory fusion features and the second bird's eye view features. It can be known that, in addition to considering the spatio-temporal alignment and fusion of target level features, the application also considers the spatio-temporal alignment and fusion of lane level features, scene level features and trajectory level features, thereby improving the accuracy of final path planning.

[0094] Based on any of the above embodiments, the feature extraction unit 503 is specifically configured to: extracting, based on the second bird's eye view features, second target level features corresponding to the vehicle end device, includes: generating, by a time sequence perception module in the vehicle end end-to-end system, based on the second bird's eye view features, target level queries using an attention mechanism, the target level queries being used to represent the second target level features.

[0095] Based on any of the above embodiments, the feature extraction unit 503 is specifically configured to: generating, by an online mapping module in the vehicle end end-to-end system, based on the second bird's eye view features, lane level queries using an attention mechanism, the lane level queries being used to represent the second lane level features.

[0096] Based on any of the above embodiments, the feature extraction unit 503 is specifically configured to: generating, by a grid occupancy module in the vehicle end end-to-end system, based on the second bird's eye view features, occupancy probability maps using an attention mechanism, the occupancy probability maps being used to represent the second scene level features.

[0097] According to any one of the above embodiments, the feature extraction unit 503 is specifically configured to: The trajectory prediction module in the vehicle end-to-end system generates a trajectory level query based on the second target level feature and the second lane level feature by using an attention mechanism, and the trajectory level query is used to represent the second trajectory level feature.

[0098] According to any one of the above embodiments, the fourth fusion unit 507 is specifically configured to: The self-attention mechanism is used to encode the time sequence information of the same first trajectory level feature to obtain the encoding information of each first trajectory level feature, and the time sequence information of the same second trajectory level feature is encoded to obtain the encoding information of each second trajectory level feature; The cross-attention mechanism is used to cross-encode the first trajectory level feature and the second trajectory level feature to obtain cross-encoding information; The multi-layer neural network is used to fuse the encoding information of the first trajectory level feature and the cross-encoding information to obtain the first fusion encoding information corresponding to the first trajectory level feature, and fuse the encoding information of the second trajectory level feature and the cross-encoding information to obtain the second fusion encoding information corresponding to the second trajectory level feature; Based on the position information of the current time of the trajectory, the first fusion encoding information and the second fusion encoding information, the first trajectory corresponding to each first trajectory level feature and the second trajectory corresponding to each second trajectory level feature are subjected to Hungarian matching to obtain a spatiotemporally aligned trajectory matching pair; The multi-layer neural network is used to fuse the information of the two trajectories across the view angle in the trajectory matching pair to obtain the trajectory fusion feature.

[0099] According to any one of the above embodiments, the receiving unit 501 is specifically configured to: Receive the first target level feature sent by the time sequence perception module in the road end-to-end system; Receive the first lane level feature sent by the online mapping module in the road end-to-end system; Receive the first scene level feature sent by the grid occupancy module in the road end-to-end system, wherein the first scene level feature is obtained by sparsely encoding and compressing a target occupancy probability map by the grid occupancy module in the road end-to-end system, and the target occupancy probability map is obtained by using an attention mechanism based on the first bird's eye view feature; Receive the first trajectory level feature sent by the trajectory prediction module in the road end-to-end system.

[0100] Based on any of the above embodiments, the vehicle end-to-end system is a deep learning model-based vehicle end-to-end system, which is trained based on the following manner: Obtain the first sample target level feature, the first sample lane level feature, the first sample scene level feature and the first sample trajectory level feature of the sample road end device; Obtain the second sample target level feature, the second sample lane level feature, the second sample scene level feature and the second sample trajectory level feature of the sample vehicle end device; Input the first sample target level feature, the first sample lane level feature, the first sample scene level feature, the first sample trajectory level feature, the second sample target level feature, the second sample lane level feature, the second sample scene level feature and the second sample trajectory level feature into the initial deep learning model to obtain the sample target fusion feature, the sample lane fusion feature, the sample scene fusion feature and the sample trajectory fusion feature output by the initial deep learning model. Based on the sample target fusion feature and the target true value, construct a target loss, based on the sample lane fusion feature and the lane true value, construct a lane loss, based on the sample scene fusion feature and the scene true value, construct a scene loss, based on the sample trajectory fusion feature and the trajectory true value, construct a trajectory loss, and based on the final path planning result and the planning true value, construct a path planning loss. Based on the target loss, the lane loss, the scene loss, the trajectory loss and the path planning loss, determine a joint loss. Based on the joint loss, iteratively update the network parameters of the initial deep learning model until the convergence condition is reached, to obtain the deep learning model-based vehicle end-to-end system.

[0101] Based on any of the above embodiments, in the training process, the feature information of the road end device and the feature information of the vehicle end device are aligned in time dimension through query stream learning, and are aligned in space dimension based on learning of frame level collaborative conversion matrix parameters.

[0102] Figure 6 is the entity structure schematic diagram of the electronic device provided by the embodiment of the present application, like Figure 6As shown, the electronic device can include a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 complete mutual communication through the communications bus 640. The processor 610 can invoke a logical instruction in the memory 630 to execute an information fusion method of a vehicle-road cooperative end-to-end system, the method comprising: receiving characteristic information sent by a road-side device, the characteristic information comprising a first target-level feature, a first lane-level feature, a first scene-level feature, and a first trajectory-level feature, the characteristic information being obtained by feature extraction on a first bird's-eye view feature corresponding to the road-side device by a road-side end-to-end system, the first bird's-eye view feature being obtained based on first frame data collected by the road-side device; obtaining second frame data collected by a vehicle-side device, and inputting the second frame data, the first target-level feature, the first lane-level feature, the first scene-level feature, and the first trajectory-level feature into a vehicle-side end-to-end system, obtaining a second bird's-eye view feature corresponding to the vehicle-side device based on the second frame data by the vehicle-side end-to-end system, and extracting a second target-level feature, a second lane-level feature, a second scene-level feature, and a second trajectory-level feature corresponding to the vehicle-side device based on the second bird's-eye view feature; performing cross-view target feature fusion after spatiotemporal alignment of the first target-level feature and the second target-level feature, to obtain a target fusion feature; performing cross-view lane feature fusion after spatiotemporal alignment of the first lane-level feature and the second lane-level feature, to obtain a lane fusion feature; performing cross-view scene feature fusion after spatiotemporal alignment of the first scene-level feature and the second scene-level feature, to obtain a scene fusion feature; performing cross-view trajectory feature fusion after spatiotemporal alignment of the first trajectory-level feature and the second trajectory-level feature, to obtain a trajectory fusion feature; The target fusion feature, the lane fusion feature, the scene fusion feature, the trajectory fusion feature, and the second bird's-eye view feature are used for path planning.

[0103] Moreover, the logic instructions in the memory 630 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0104] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor, so that the computer can execute the information fusion method of the vehicle-road cooperative end-to-end system provided by the above-mentioned method, the method comprises: receiving the characteristic information sent by the road end device, the characteristic information comprises a first target level feature, a first lane level feature, a first scene level feature and a first trajectory level feature, the characteristic information is obtained by feature extraction on the first bird's eye view feature corresponding to the road end device of the road end end-to-end system, and the first bird's eye view feature is obtained based on the first frame data collected by the road end device; obtaining the second frame data collected by the vehicle end device, and inputting the second frame data, the first target level feature, the first lane level feature, the first scene level feature and the first trajectory level feature into the vehicle end end-to-end system, obtaining the second bird's eye view feature corresponding to the vehicle end device based on the second frame data through the vehicle end end-to-end system, and extracting the second target level feature, the second lane level feature, the second scene level feature and the second trajectory level feature corresponding to the vehicle end device based on the second bird's eye view feature; After the spatio-temporal alignment of the first target level feature and the second target level feature, cross-view target feature fusion is performed to obtain target fusion feature; After the spatio-temporal alignment of the first lane level feature and the second lane level feature, cross-view lane feature fusion is performed to obtain lane fusion feature; After the spatio-temporal alignment of the first scene level feature and the second scene level feature, cross-view scene feature fusion is performed to obtain scene fusion feature; After the spatio-temporal alignment of the first trajectory level feature and the second trajectory level feature, cross-view trajectory feature fusion is performed to obtain a trajectory fusion feature; The target fusion feature, the lane fusion feature, the scene fusion feature, the trajectory fusion feature, and the second bird's eye view feature are used for path planning.

[0105] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the information fusion method of the vehicle-road cooperative end-to-end system provided by the above method, the method comprising: receiving feature information sent by a road end device, the feature information comprising a first target level feature, a first lane level feature, a first scene level feature, and a first trajectory level feature, the feature information being obtained by feature extraction on a first bird's eye view feature corresponding to the road end device by a road end end-to-end system, the first bird's eye view feature being obtained based on first frame data collected by the road end device; Second frame data collected by a vehicle end device is obtained, and the second frame data, the first target level feature, the first lane level feature, the first scene level feature, and the first trajectory level feature are input into a vehicle end end-to-end system, the second bird's eye view feature corresponding to the vehicle end device is obtained based on the second frame data by the vehicle end end-to-end system, and the second target level feature, the second lane level feature, the second scene level feature, and the second trajectory level feature corresponding to the vehicle end device are extracted based on the second bird's eye view feature; After spatio-temporal alignment of the first target level feature and the second target level feature, cross-view target feature fusion is performed to obtain a target fusion feature; After spatio-temporal alignment of the first lane level feature and the second lane level feature, cross-view lane feature fusion is performed to obtain a lane fusion feature; After spatio-temporal alignment of the first scene level feature and the second scene level feature, cross-view scene feature fusion is performed to obtain a scene fusion feature; After spatio-temporal alignment of the first trajectory level feature and the second trajectory level feature, cross-view trajectory feature fusion is performed to obtain a trajectory fusion feature; The target fusion feature, the lane fusion feature, the scene fusion feature, the trajectory fusion feature, and the second bird's eye view feature are used for path planning.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0107] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An information fusion method of a vehicle infrastructure integration end-to-end system, characterized by, The method comprises the following steps: receiving feature information sent by a road end device, the feature information comprising a first target level feature, a first lane level feature, a first scene level feature and a first trajectory level feature, the feature information being obtained by feature extraction on a first bird's eye view feature corresponding to the road end device by a road end to end system, the first bird's eye view feature being obtained based on first frame data collected by the road end device; obtaining second frame data collected by a vehicle end device, and inputting the second frame data, the first target level feature, the first lane level feature, the first scene level feature and the first trajectory level feature into a vehicle end to end system, obtaining a second bird's eye view feature corresponding to the vehicle end device based on the second frame data by the vehicle end to end system, and extracting a second target level feature, a second lane level feature, a second scene level feature and a second trajectory level feature corresponding to the vehicle end device based on the second bird's eye view feature; spatially and temporally aligning the first target level feature and the second target level feature, and then performing cross-view target feature fusion to obtain a target fusion feature; spatially and temporally aligning the first lane level feature and the second lane level feature, and then performing cross-view lane feature fusion to obtain a lane fusion feature; spatially and temporally aligning the first scene level feature and the second scene level feature, and then performing cross-view scene feature fusion to obtain a scene fusion feature; spatially and temporally aligning the first trajectory level feature and the second trajectory level feature, and then performing cross-view trajectory feature fusion to obtain a trajectory fusion feature; wherein the target fusion feature, the lane fusion feature, the scene fusion feature, the trajectory fusion feature and the second bird's eye view feature are used for path planning. 2.The information fusion method of the vehicle infrastructure cooperative end-to-end system according to claim 1, characterized in that, extracting the second target level feature corresponding to the vehicle end device based on the second bird's eye view feature comprises: generating a target level query based on the second bird's eye view feature by a time sequence perception module in the vehicle end to end system, the target level query being used to represent the second target level feature. 3.The information fusion method of the vehicle infrastructure cooperative end-to-end system according to claim 2, characterized in that, extracting the second lane level feature corresponding to the vehicle end device based on the second bird's eye view feature comprises: generating a lane level query based on the second bird's eye view feature by an online mapping module in the vehicle end to end system, the lane level query being used to represent the second lane level feature. 4.The information fusion method of the vehicle infrastructure cooperative end-to-end system according to claim 3, characterized in that, extracting the second scene level feature corresponding to the vehicle end device based on the second bird's eye view feature comprises: generating an occupancy probability map based on the second bird's eye view feature by a grid occupancy module in the vehicle end to end system, the occupancy probability map being used to represent the second scene level feature. 5.The information fusion method of a vehicle infrastructure integration end-to-end system according to claim 3, wherein, extracting the second trajectory level feature corresponding to the vehicle end device based on the second bird's eye view feature comprises: The trajectory prediction module in the vehicle end-to-end system generates a trajectory-level query based on the second target-level feature and the second lane-level feature using an attention mechanism, where the trajectory-level query is used to represent the second trajectory-level feature. 6.The information fusion method of a vehicle infrastructure integration end-to-end system according to claim 1, wherein, After the spatiotemporal alignment of the first trajectory-level feature and the second trajectory-level feature, cross-view trajectory feature fusion is performed to obtain trajectory fusion features, including: The temporal information of the same first trajectory-level feature is encoded by a self-attention mechanism to obtain encoding information of each first trajectory-level feature, and the temporal information of the same second trajectory-level feature is encoded to obtain encoding information of each second trajectory-level feature; The first trajectory-level feature and the second trajectory-level feature are cross-encoded by a cross-attention mechanism to obtain cross-encoding information; The encoding information of the first trajectory-level feature and the cross-encoding information are fused by a multi-layer neural network to obtain first fusion encoding information corresponding to the first trajectory-level feature, and the encoding information of the second trajectory-level feature and the cross-encoding information are fused to obtain second fusion encoding information corresponding to the second trajectory-level feature; Based on the position information of the current time of the trajectory, the first fusion encoding information, and the second fusion encoding information, a Hungarian matching is performed on each first trajectory corresponding to the first trajectory-level feature and each second trajectory corresponding to the second trajectory-level feature to obtain a spatiotemporally aligned trajectory matching pair; The trajectory fusion features are obtained by fusing the information of the two trajectories across the view in the trajectory matching pair by a multi-layer neural network. 7.The information fusion method of a vehicle infrastructure integration end-to-end system according to claim 1, wherein, The feature information sent by the road end device includes: The first target-level feature sent by the timing perception module in the road end-to-end system is received; The first lane-level feature sent by the online mapping module in the road end-to-end system is received; The first scene-level feature sent by the grid occupancy module in the road end-to-end system is received, which is obtained by sparse coding and compression of the target occupancy probability map by the grid occupancy module in the road end-to-end system, and the target occupancy probability map is obtained by generating an attention mechanism based on the first bird's eye view feature; The first trajectory-level feature sent by the trajectory prediction module in the road end-to-end system is received. 8.The information fusion method of a vehicle infrastructure integration end-to-end system according to any one of claims 1-7, characterized in that, The vehicle end-to-end system is a deep learning model-based vehicle end-to-end system, which is trained based on the following method: The first sample target-level feature, the first sample lane-level feature, the first sample scene-level feature, and the first sample trajectory-level feature of the sample road end device are obtained; The second sample target-level feature, the second sample lane-level feature, the second sample scene-level feature, and the second sample trajectory-level feature of the sample vehicle end device are obtained; input the first sample target level feature, the first sample lane level feature, the first sample scene level feature, the first sample track level feature, the second sample target level feature, the second sample lane level feature, the second sample scene level feature and the second sample track level feature into an initial deep learning model to obtain sample target fusion features, sample lane fusion features, sample scene fusion features and sample track fusion features output by the initial deep learning model in cross-view spatio-temporal alignment; construct a target loss based on the sample target fusion features and a target true value, construct a lane loss based on the sample lane fusion features and a lane true value, construct a scene loss based on the sample scene fusion features and a scene true value, construct a track loss based on the sample track fusion features and a track true value, and construct a path planning loss based on a final path planning result and a planning true value; determine a joint loss based on the target loss, the lane loss, the scene loss, the track loss and the path planning loss; iteratively update network parameters of the initial deep learning model based on the joint loss until a convergence condition is reached to obtain a vehicle end-to-end system based on a deep learning model. 9.The information fusion method of the vehicle infrastructure cooperative end-to-end system according to claim 8, characterized in that, In the training process, the feature information of the road end device and the feature information of the vehicle end device are aligned in the time dimension through query stream learning, and are aligned in the space dimension based on learning of frame level collaborative conversion matrix parameters.

10. An information fusion device of a vehicle infrastructure integration end-to-end system, characterized by, Comprise: a receiving unit configured to receive feature information sent by a road end device, the feature information comprising first target level features, first lane level features, first scene level features and first track level features, the feature information being obtained by feature extraction on first bird's eye view features corresponding to the road end device by a road end end-to-end system, the first bird's eye view features being obtained based on first frame data collected by the road end device; an obtaining unit configured to obtain second frame data collected by a vehicle end device; a feature extraction unit configured to input the second frame data, the first target level features, the first lane level features, the first scene level features and the first track level features into a vehicle end end-to-end system, obtain second bird's eye view features corresponding to the vehicle end device based on the second frame data by the vehicle end end-to-end system, and extract second target level features, second lane level features, second scene level features and second track level features corresponding to the vehicle end device based on the second bird's eye view features; a first fusion unit configured to perform spatio-temporal alignment on the first target level features and the second target level features, and then perform cross-view target feature fusion to obtain target fusion features; a second fusion unit configured to perform spatio-temporal alignment on the first lane level features and the second lane level features, and then perform cross-view lane feature fusion to obtain lane fusion features; a third fusion unit configured to perform cross-view scene feature fusion on the first scene-level feature and the second scene-level feature after spatiotemporal alignment, to obtain a scene fusion feature; a fourth fusion unit configured to perform cross-view trajectory feature fusion on the first trajectory-level feature and the second trajectory-level feature after spatiotemporal alignment, to obtain a trajectory fusion feature; wherein the target fusion feature, the lane fusion feature, the scene fusion feature, the trajectory fusion feature, and the second bird's eye view feature are used for path planning.

Images