Feature-level multi-source multi-modal cloud perception fusion system and method

Abstract

The application relates to a feature-level multi-source multi-modal cloud perception fusion system and method, which comprises a sensor, a networked intelligent vehicle, a road terminal device, a wireless communication unit and a cloud computing platform; the method comprises the following steps: a vehicle terminal and a road terminal respectively acquire multi-source multi-modal perception data; BEV perception features under different modes are coded according to the acquired multi-source multi-modal perception data; the vehicle terminal and the road terminal transmit the BEV perception features to the cloud for spatial alignment processing, and then sequentially perform multi-modal branch collaborative feature fusion and cross-modal feature dynamic fusion to output cloud fusion BEV features; the cloud fusion BEV features are decoded and processed to output dynamic target information. Compared with the prior art, the application can improve the robustness, accuracy and scalability of cloud perception fusion, and can reliably and accurately generate dynamic target information in the cloud control detection range by combining multi-source multi-modal perception information of the vehicle terminal and the road terminal.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a feature-level multi-source multimodal cloud perception fusion system and method. Background Technology

[0002] Currently, intelligent vehicles, represented by autonomous driving, are at a critical stage of development. Among them, the single-vehicle intelligence approach is the mainstream implementation method in the current intelligent driving field. This approach relies on the intelligent vehicle's own sensors and computing platform to complete tasks such as perception, decision-making, and control. However, single-vehicle perception has limitations, such as susceptibility to occlusion and beyond-line-of-sight effects, making it difficult to meet the higher levels of autonomous driving requirements in complex traffic environments. Therefore, the collaborative intelligence approach is considered the future for achieving higher-level intelligent driving. However, traditional vehicle-to-vehicle and vehicle-to-infrastructure (V2I) collaboration emphasizes information interaction between vehicles or infrastructural systems, aiming to assist single-vehicle perception and decision-making behavior, lacking consideration for regional collaborative behavior. Recently, the proposed integrated vehicle-road-cloud fusion control system provides a more global collaborative intelligence approach. This system considers full-domain perception within the cloud control range, thereby providing intelligent vehicles with global information and enabling collaborative decision-making and planning for intelligent connected vehicles within the cloud control range. In the integrated vehicle-road-cloud fusion control system, the cloud-based perception fusion module is responsible for integrating multi-source perception information from the vehicle and roadside, generating dynamic target mappings within the cloud control range, and periodically providing reliable and accurate target detection information to the integrated vehicle-road-cloud system.

[0003] However, research on cloud-based perception fusion within the industry is still in the exploratory stage. Existing solutions all adopt target-level fusion to achieve cloud-based perception fusion. This type of method results in the cloud receiving the least amount of individual perception information, relying on the results of individual perceptions, and is less robust to individual perception noise than feature-level fusion methods. Feature-level collaborative fusion provides richer perception information within acceptable transmission bandwidth requirements, thus exhibiting better detection results than target-level fusion. However, in the field of collaborative perception, feature-level fusion schemes often default to a single perception modality, such as LiDAR. This is detrimental to the scalability of the vehicle-road-cloud system and may lead to cloud-based perception failure due to the failure of a single-modality sensor, which is detrimental to the perception robustness of the vehicle-road-cloud system. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a feature-level multi-source multimodal cloud-based perception fusion system and method, which can improve the robustness, accuracy and scalability of cloud-based perception fusion. By combining multi-source multimodal perception information from vehicle and roadside, dynamic target information within the cloud control detection range can be reliably and accurately generated.

[0005] The objective of this invention can be achieved through the following technical solution: a feature-level multi-source multimodal cloud-based perception fusion system, comprising sensors, connected intelligent vehicles, roadside equipment, wireless communication units, and a cloud computing platform, wherein the sensors are used to collect raw perception data and their own pose information;

[0006] The connected intelligent vehicle is a vehicle node (VN) with sensing and computing capabilities;

[0007] The roadside equipment is a roadside node (RN) with the ability to communicate with the cloud computing platform. Different edge scenarios are configured with specific roadside equipment according to the traffic environment. Different scenarios are configured with different numbers of roadside equipment and different perception modalities.

[0008] The wireless communication unit is installed on the connected intelligent vehicle;

[0009] The cloud computing platform is used to receive VN perception information from connected intelligent vehicles and RN information from roadside equipment, and perform fusion processing to output dynamic target information.

[0010] Furthermore, the roadside equipment is specifically a Central Roadside Node (CRN), which is coupled to a cloud computing platform to provide stable sensing output for the cloud computing platform.

[0011] Furthermore, the sensors include perception-related sensors and positioning-related sensors, wherein the perception-related sensors include an RGB camera and a LiDAR, which are responsible for collecting raw perception data;

[0012] Positioning-related sensors include GPS and IMU, which are responsible for acquiring their own pose information.

[0013] A feature-level multi-source multimodal cloud-based sensing fusion method includes the following steps:

[0014] S1. Vehicle-side and road-side data are acquired from multiple sources and multiple modes of perception.

[0015] S2. Based on the acquired multi-source multimodal perception data, encode the BEV (Bird's Eye View) perception features under different modalities;

[0016] S3: The vehicle and roadside transmit the BEV perception features to the cloud for spatial alignment processing. Then, multimodal branch collaborative feature fusion and cross-modal feature dynamic fusion are performed sequentially to output the cloud-fused BEV features.

[0017] S4. Decode the cloud-integrated BEV features and output dynamic target information.

[0018] Furthermore, the specific process of step S1 is as follows:

[0019] The vehicle-side node (VN) acquires local perception data of the vehicle side based on surround-view RGB cameras or LiDAR, and obtains the vehicle's position and pose through GPS and IMU.

[0020] The roadside node (RN) acquires roadside perception data based on LiDAR.

[0021] Furthermore, step S2 specifically involves obtaining BEV perception features in point cloud mode or image mode based on encoding different perception models.

[0022] Furthermore, the specific process of encoding and acquiring BEV perception features in point cloud modality in step S2 is as follows:

[0023] For the original laser point cloud, point cloud preprocessing is performed. First, the point cloud is filtered based on the set detection range, and laser points reflected on themselves are removed to obtain the laser point set.

[0024] Then, a point cloud pseudo-image is generated. By dividing the detection range, point cloud pillars are obtained. The features of each laser point are encoded, and the features of each point cloud pillar are obtained based on the max pooling operation, thus obtaining the pseudo-image features.

[0025] The pseudo-image of the point cloud is input into a 2D convolutional backbone network to encode the BEV perception features in the point cloud mode.

[0026] Furthermore, the specific process of encoding and acquiring BEV perception features in image modality in step S2 is as follows:

[0027] The Resent network is used to extract 2D image features from the view-around RGB image;

[0028] Based on the set detection range, a three-dimensional voxel grid is divided with the origin of the vehicle coordinate system as the center. Then, based on the intrinsic and extrinsic parameter matrices of each camera, the three-dimensional center points of each grid are projected onto the visual feature map of each view. The image features of the projection points on each view are obtained based on bilinear interpolation. The panoramic image features of each projection point are obtained along the viewing angle and filled into the three-dimensional grid.

[0029] Then, based on the 2D convolutional layer, the three-dimensional raster features are compressed along the height direction, and then the BEV perception features in the image modality are obtained by encoding through the 2D convolutional backbone network.

[0030] Furthermore, the specific process of multimodal branch collaborative feature fusion in step S3 is as follows:

[0031] I. Query Vector Generation

[0032] For the i-th vehicle-end BEV feature F after spatial alignment ′i i = 1, ..., N v , where N v To determine the number of vehicle-side nodes, we first encode the BEV features of each vehicle-side node using a 2D convolutional layer with a kernel size of 1*1 and a sigmoid function, thus obtaining the information map of each vehicle-side node.

[0033] Each grid position takes a value between 0 and 1. The closer the value is to 1, the more local location information is centered on that grid position. Then, the cloud performs N... v The information graph of each vehicle node is subjected to max pooling operation based on the grid position to obtain the overall information graph. This represents the overall information distribution of the sensing features provided by the dynamic vehicle-end node at this time. If some grid positions contain a lot of information, the cloud should take them into consideration. Conversely, if some grid positions contain a little information, they should be avoided to reduce the introduction of additional cooperative noise.

[0034] The cloud-based system guides query vector generation based on the overall information map. Using a greedy approach, it sorts the overall information map from highest to lowest evaluated information content, then obtains the top K grid positions, and uses these to index the roadside BEV features to obtain a preliminary query vector set.

[0035] II. Value Vector Generation

[0036] The roadside BEV features are filled into the corresponding blank grid positions, and then modality-related value feature mapping is used to obtain the value features for attention operations:

[0037]

[0038] in, This refers to the linear mapping function of the value vector under the corresponding modal branch;

[0039] Each vehicle-side BEV feature is flattened after value mapping encoding to obtain a value vector set. Where K v =H×W×N v , indicating the number of value vectors;

[0040] III. Collaborative Feature Fusion Based on Deformable Cross-Attention

[0041] First, global max pooling and average pooling operations are performed on the BEV features of each vehicle to obtain the following results: and

[0042] Next, with the obtained query features Perform broadcast-based element summation to obtain query features with vehicle-end node awareness.

[0043] Then based on two linear mapping layers (Linear) offset and Linear weight Output the sampling bias of the corresponding query vector and weight Where n p It is the preset number of sampling points for each query vector under each vehicle-end node, and A is the number of sampling points along N based on the linear mapping. v ×n p The sampling weights obtained by performing a softmax operation on each point are now the basis for each query vector. All can be based on their sampling bias and sampling weights Complete the set of logarithmic vectors Perform deformable sampling and weighted summation:

[0044]

[0045] Where DCA(q) represents performing a deformable cross-attention operation on the query vector q. W represents the feature of the sampled values ​​at the corresponding vehicle-side node based on the sampling bias. o It is a learnable parameter matrix used for output mapping;

[0046] After deformable attention operations, the query features, which incorporate dynamic vehicle-side perception features, are then filled into the loop-side BEV features to obtain the cloud-based collaborative fusion features under the corresponding modal branch. in, These are point cloud modal collaborative fusion features and image modal collaborative fusion features, respectively.

[0047] Furthermore, the specific process of cross-modal feature dynamic fusion in step S3 is as follows:

[0048] I. Static splicing and fusion

[0049] The point cloud modal collaborative fusion features and the image modal collaborative fusion features are initially fused to obtain static fusion features.

[0050]

[0051] Where Concate(·) represents concatenation along the feature channels, and σ(·) is the ReLU activation function;

[0052] II. Dynamic Correction and Fusion

[0053] The statically fused features are subjected to global average pooling along the space, then encoded by a linear layer and a ReLU activation layer, and finally output by a sigmoid function to represent a correction factor for dynamic channel adjustment. Finally, the dynamic correction factor is combined with the static fusion feature to calculate the Hadamard product, resulting in the cloud-based fusion feature.

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] This invention comprises sensors, connected intelligent vehicles, roadside equipment, wireless communication units, and a cloud computing platform. Different connected intelligent vehicles can use different perception modalities, and different numbers and different perception modalities of roadside equipment can be configured in different scenarios. The cloud computing platform is responsible for integrating VN and RN perception information and outputting the final dynamic target information. This effectively combines multi-source and multi-modal perception information from the vehicle and roadside to generate dynamic target information within the cloud control detection range. On the one hand, it helps intelligent connected vehicles obtain more comprehensive and effective information; on the other hand, it supports the full-domain collaborative decision-making and planning of the vehicle-road-cloud integrated system.

[0056] This invention uses multi-source sensing information from both vehicle and roadside units as input to achieve dynamic target detection within the cloud control range. First, the vehicle and roadside units acquire sensing data in different modalities based on their respective sensors. Then, each sensing node on both the vehicle and roadside units initially encodes BEV sensing features in different modalities. Next, the vehicle and roadside units transmit the BEV sensing features and other auxiliary information (positioning information, timestamps, etc.) to the cloud for spatial alignment of valid BEV sensing features based on the same spatial coordinate system. Then, multi-modal branch collaborative feature fusion (performing collaborative fusion of individual features in image and point cloud modalities respectively to obtain cloud-based collaborative fusion features in different modal branches) and cross-modal feature dynamic fusion (further cross-modal fusion of collaborative fusion features from different modalities to obtain the final cloud-based fused BEV features) are performed sequentially. Finally, dynamic target information, including location, confidence level, and size, is decoded and output based on the cloud-based fused BEV features. This overcomes the problem of insufficient single-vehicle sensing and uses a feature-level fusion method to improve the robustness and accuracy of cloud-based fused sensing.

[0057] To more fully extract collaborative perception information across different modalities, this invention designs a hierarchical approach to progressively achieve multimodal collaborative fusion. First, collaborative feature fusion is performed across different modal branches to extract as much contextual information as possible related to the dynamic target. Since the BEV features transmitted by vehicle-end nodes inevitably have time delays, and the poses of vehicle-end nodes often have errors, the BEV features received by the cloud are spatially offset. Furthermore, different vehicle / roadside nodes provide differentiated perception information at different times and locations, and the information provided by different perception nodes should be adaptively considered at different grid positions. Therefore, to overcome these problems, this invention uses Deformable Cross-Attention to achieve collaborative feature fusion across different modal branches. It consists of three parts: query vector generation, value vector generation, and collaborative feature fusion based on Deformable Cross-Attention. This addresses the lack of consideration for multimodal individual perception in existing feature-level collaborative perception systems, further improving the scalability and robustness of the cloud-based perception fusion system. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0059] Figure 2 Here is a flowchart of the feature extraction algorithm for point cloud modal single entity BEV;

[0060] Figure 3 This is a schematic diagram of a 2D convolutional backbone coding network.

[0061] Figure 4 Here is a flowchart of the image modality BEV feature extraction algorithm;

[0062] Figure 5 Flowchart of cloud-based collaborative feature fusion algorithm under single-modal branch;

[0063] Figure 6 This is a flowchart of a cloud-based cross-modal collaborative feature fusion algorithm. Detailed Implementation

[0064] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0065] Example

[0066] This embodiment provides a feature-level multi-source multimodal cloud-based perception fusion system, including sensors, connected intelligent vehicles, roadside equipment, wireless communication units, and a cloud computing platform. The sensors are used to collect raw perception data and their own pose information.

[0067] Connected intelligent vehicles are vehicle-side nodes (VNs) with sensing and computing capabilities;

[0068] Roadside equipment is a roadside node (RN) with the ability to communicate with a cloud computing platform. Different edge scenarios are configured with specific roadside equipment according to the traffic environment. Different scenarios are configured with different numbers of roadside equipment and different perception modalities.

[0069] The wireless communication unit is installed on the connected intelligent vehicle;

[0070] The cloud computing platform is used to receive VN perception information from connected intelligent vehicles and RN information from roadside equipment, and perform fusion processing to output dynamic target information.

[0071] Specifically, the sensors mainly include perception-related sensors and positioning-related sensors. The former includes RGB cameras and LiDAR, which are responsible for collecting raw perception data; the latter includes GPS and IMU, which are responsible for acquiring their own pose information.

[0072] Connected intelligent vehicles refer to vehicle-to-everything (VN) nodes equipped with onboard wireless communication units and possessing sensing and computing capabilities. In this system, different connected intelligent vehicles can use different sensing modes.

[0073] Roadside equipment refers to roadside nodes (RNs) equipped with sensing sensors and capable of communicating with the cloud platform. Different edge scenarios require specific roadside equipment configurations based on the traffic environment. Different numbers and sensing modalities of roadside equipment can be configured in different scenarios. Since connected intelligent vehicles are dynamic, to form a stable cloud-based sensing mapping, this system sets up central roadside nodes (CRNs) coupled to the cloud. Multiple CRNs can be present, directly connected to the cloud computing platform via fiber optic cables. Through reasonable arrangement, they can basically cover the considered area, thus providing a relatively stable sensing output to the cloud. The advantage of this arrangement is that when there are no connected intelligent vehicles within the cloud control range (where the penetration rate of connected intelligent vehicles is low), the system can still obtain reliable sensing results based on roadside sensing. When connected intelligent vehicles are present, the cloud simultaneously receives vehicle-side sensing information and fuses it with CRN sensing information. The cloud computing platform is responsible for integrating VN and RN sensing information and outputting the final dynamic target information.

[0074] This embodiment only considers deploying a single RN equipped with a lidar sensor, which is a CRN. However, in practical applications, this system can be expanded to include multiple RNs. Furthermore, the system has a total of N vehicle-side nodes, where N... img A visual image modality perception, Npcd A laser point cloud-based modal perception system.

[0075] Based on the above system, a feature-level multi-source multimodal cloud-based perception fusion method is implemented, such as... Figure 1 As shown, it includes:

[0076] 1) Multi-source single-unit sensing data acquisition: Vehicle-side and road-side sensors acquire sensing data in different modalities based on their respective sensors, such as RGB images and LiDAR point clouds;

[0077] 2) Multimodal BEV (Bird's Eye View) feature encoding: Initial encoding of BEV perception features under different modes at each perception node at the vehicle end and road end;

[0078] 3) BEV perception feature transmission and alignment: The vehicle and roadside transmit BEV perception features and other auxiliary information (location information, timestamps, etc.) to the cloud server, and perform spatial alignment on valid BEV perception features based on the same spatial coordinate system.

[0079] 4) Multimodal branch collaborative feature fusion: Perform collaborative fusion on individual features in RGB image mode and Lidar point cloud mode respectively to obtain cloud collaborative fusion features under different modal branches;

[0080] 5) Dynamic fusion of cross-modal features: The collaborative fusion features of different modalities obtained above are further fused across modalities to obtain the final cloud-fused BEV features;

[0081] 6) Fusion Feature Decoding: Based on the aforementioned cloud-based fusion BEV feature decoding, dynamic target information is output, including information such as location, confidence level, and size.

[0082] This embodiment applies the above technical solution, and its main contents include:

[0083] I. Acquisition of Multi-Source Single-Unit Sensing Data

[0084] The vehicle-side node (VN) acquires local perception data from the vehicle side using surround-view RGB cameras or LiDAR, namely surround-view RGB images or LiDAR point clouds (LiDAR point information includes the 3D coordinates and reflection intensity of each point). Additionally, the VN obtains its own vehicle pose via GPS and IMU. The central roadside node (CRN) acquires roadside perception data using LiDAR. Both vehicle-side and roadside sensors undergo thorough parameter calibration. The vehicle-side node projects the LiDAR point cloud onto its coordinate system based on LiDAR sensor extrinsic parameters, while the roadside node directly projects the LiDAR point cloud onto the global coordinate system within the cloud control range, reducing subsequent repeated coordinate transformations.

[0085] II. Multimodal BEV Feature Encoding

[0086] In this solution, the vehicle-side / roadside nodes have multiple perception modalities. Therefore, it is necessary to first obtain BEV features under different modalities based on different perception models. In the point cloud modality, this solution uses the PointPillars algorithm to extract BEV features. The algorithm flow is as follows: Figure 2 As shown. For the laser point cloud acquired by node i at time t, first based on the considered detection range (X... min ,X max ,Y min ,Y max Z min Z max The point cloud is filtered, and laser points reflected from themselves are removed to obtain a laser point set. (N p (This represents the total number of laser points). Then, based on the predefined BEV grid size (v... x ,v y The aforementioned detection range is divided into H×W BEV grids, with each grid having a height of Z. max -Z min The divided grid is also known as a point cloud pillar. Then, a linear mapping layer is used to embed and encode the original features of the laser points. The input feature dimension of the laser points is 4, i.e., three-dimensional coordinates and reflection intensity. After encoding, the feature dimension of the laser points is D. It is important to note that since the number of laser points in different point cloud pillars varies, the number M of laser points considered in each point cloud pillar needs to be pre-set. If the number of points in a point cloud pillar exceeds M, it is downsampled to M points based on Farthest Point Sampling (FPS); if the number of points in a point cloud pillar is less than M, zero-value feature points are filled in. After encoding the features of each laser point, the features of each point cloud pillar are obtained based on max pooling, thus obtaining pseudo-image features with dimensions D×H×W. The pseudo-image is further processed by a 2D convolutional backbone network to extract spatial information. The 2D convolutional backbone network used in this scheme is structurally similar to a feature pyramid, such as... Figure 3 As shown, the BEV features in the point cloud modality are obtained after encoding by a 2D convolutional backbone network. Where C represents the BEV feature dimension. The BEV feature encoding process is the same for both vehicle-side and road-end cloud modes, but the final output BEV feature size will differ, with the road-end node outputting a larger BEV feature space size.

[0087] In the image modality, this scheme employs a top-down approach to extract BEV features, and the overall process is as follows: Figure 4 As shown. For node i at time t, the loop view image... Where S represents the number of surround-view cameras, and H0 and W0 are the height and width of the RGB image, respectively. This scheme first uses a ResNet network to perform feature encoding on the 2D image, obtaining visual feature maps from different viewpoints. Then, based on the local detection range (X...) considered by the vehicle... min ,X max ,Y min ,Y max Z min Z max A three-dimensional voxel grid is divided with the origin of the vehicle coordinate system as the center. The size of each voxel grid is (v x ,v y ,v z This process yields a 3D grid of size Z×H×W, where Z represents the height direction, H represents the Y direction, and W represents the X direction. Then, based on the intrinsic and extrinsic parameter matrices of each camera, the pre-defined grid center points are projected onto each view. It's important to note that this process determines whether the grid center point is associated with a view: if the projected pixel is within the corresponding view's feature map, the grid is considered associated with that view. For grids with associated views, their projection points are used to obtain their visual features under the corresponding view using bilinear interpolation. Since the same grid may have multiple associated views, this scheme performs averaging along the viewing angle on the projection features of each grid, resulting in a 3D voxel feature of size D×Z×H×W, where D is the visual feature dimension. To obtain BEV features, this scheme applies a 2D convolutional layer with a 1*1 kernel to compress the aforementioned features along the height direction. Then, based on the feature pyramid-shaped backbone network of the laser point cloud modality, feature encoding is performed to obtain spatial correlation, ultimately yielding the BEV features of the image modality. It should be noted that the backbone coding network used here is only structurally the same as that in the point cloud mode, but the network parameters are different.

[0088] III. BEV Sensing Feature Transmission and Alignment

[0089] After individual sensing feature encoding, vehicle-side node i uses the vehicle communication unit to transmit BEV features. Self-driving position and timestamp t i Information is wirelessly transmitted to the cloud computing platform. Since the central roadside node e is directly coupled to the cloud platform, the cloud directly acquires the roadside BEV characteristics. For the BEV characteristics transmitted by the vehicle-side nodes, the cloud platform will first calculate the relative time difference tt. iOnly when the relative time difference is less than a predetermined threshold τ is the transmitted BEV feature considered valid and taken into account. Since the BEV feature of the vehicle-side node is completed in the vehicle's coordinate system, it needs to be transformed to the cloud's global coordinate system. As shown in Equation 1, the transmitted vehicle pose... This includes the 2D coordinates and yaw angle of the vehicle-end node i in the cloud's global coordinate system from a bird's-eye view. The cloud platform utilizes this pose, based on a 2D spatial affine transformation Γ. i→global Obtain the BEV features of this node in the cloud global coordinate system. Specifically, the transformation process uses bilinear interpolation to obtain the corresponding raster features. For rasteres that do not have corresponding original features, they are filled with 0 values. Therefore, the mask is obtained simultaneously during the transformation process. When the raster feature is valid, the corresponding mask raster value is 1; otherwise, it is 0.

[0090]

[0091] IV. Multimodal Branch Collaborative Feature Fusion

[0092] To more fully extract collaborative perception information from different modalities, this scheme implements multimodal collaborative fusion step by step in a hierarchical manner. First, collaborative feature fusion is performed under different modal branches to extract as much contextual information related to the dynamic target as possible. Since the BEV features transmitted by vehicle-end nodes inevitably have time delays, and the pose of vehicle-end nodes often has errors, the BEV features received from the cloud actually have spatial distribution offsets. Furthermore, different vehicle-end / road-end nodes provide differentiated perception information at different times and locations, and the information provided by different perception nodes should be adaptively considered at different grid positions. Therefore, to overcome the above problems, this scheme uses Deformable Cross-Attention to complete the collaborative feature fusion under different modal branches. For ease of explanation, this section will not distinguish between image modality and point cloud modality features, and will ignore time superscripts. This scheme uses the central road-end node (the static perception subject in the cloud) as the query subject and adaptively fuses the perception information of dynamic vehicle-end nodes. The collaborative feature fusion algorithm flow under a single modal branch is as follows: Figure 5 As shown, it can be mainly divided into three parts: query vector generation, value vector generation, and collaborative feature fusion based on deformable cross attention, which will be explained in detail below.

[0093] The first step is query vector generation. The vehicle-side BEV features received from the cloud are often sparse, meaning many grid locations contain little information relevant to the dynamic target. This not only introduces noise into cloud-based perception fusion but also increases computational overhead. Therefore, a cloud-based query generation method guided by vehicle-side information evaluation is designed. First, as shown in Equation 2, for the spatially aligned i-th vehicle-side BEV feature F... ′i i = 1, ..., N v , where N v To determine the number of vehicle-side nodes, we first encode the BEV features of each vehicle-side node using a 2D convolutional layer with a kernel size of 1*1 and a sigmoid function, thus obtaining the information map of each vehicle-side node.

[0094]

[0095] Each grid position takes a value from 0 to 1, with values ​​closer to 1 indicating a greater amount of local location information centered on that grid position. Then, the cloud performs N... v The information graph of each vehicle node is subjected to max pooling operation based on the grid position to obtain the overall information graph. This represents the overall information distribution of the perceived features provided by the dynamic vehicle-end nodes at this time. If some grid positions contain a lot of information, the cloud should consider them; conversely, if some grid positions contain little information, they should be avoided to reduce the introduction of additional cooperative noise. Based on the above, the cloud guides the generation of query vectors based on the overall information map. Specifically, a greedy approach is adopted: the obtained overall information map is sorted from high to low according to the evaluated information content, and then the top K grid positions are obtained. These are used to index the features of the central road-end node to obtain a preliminary query vector set.

[0096] Secondly, there is the value vector generation. The vehicle-side BEV features received from the cloud are local, resulting in many blank grid positions after spatial alignment. This solution fills the corresponding blank grid positions with the central roadside BEV features, and then obtains the value features for attention operations through modality-related value feature mapping. The calculation process is shown in Equation 3.

[0097]

[0098] in This refers to the linear mapping function of the value vector under the corresponding modal branch. This value vector generation method facilitates subsequent attention operations and ensures that the perceived features of each dynamic vehicle node have a global awareness in the cloud. After value mapping encoding, the BEV features of each vehicle are flattened to obtain the value vector set. Where K v =H×W×N v , which represents the number of value vectors.

[0099] Next, we perform BEV feature fusion based on deformable attention. First, we need to generate sampling biases and sampling weights for each query vector, which are used for value vector sampling and weighted summation in subsequent deformable attention operations. Conventional deformable attention directly generates a fixed number of sampling biases and sampling weights based on the aforementioned query feature Q, which is a static generation method. However, in this scheme, the number of connected intelligent vehicles is dynamically changing, and the biases and weights under different vehicle-end nodes are often different. To enable the model to dynamically generate matching sampling biases and weights according to the vehicle-end nodes, a generation method with stronger coupling to the vehicle-end nodes is designed, as shown in Equations 4-8.

[0100]

[0101]

[0102]

[0103] Δ = Linear offset (Q a (7)

[0104] A = Softmax(Linear) weight (Q a (8)

[0105] First, global max pooling and average pooling operations are performed on the BEV features of each vehicle to obtain the following results: and Next, the query features obtained above Perform broadcast-based element summation to obtain query features with vehicle-end node awareness. Then based on two linear mapping layers (Linear) offset and Linear weight Output the sampling bias of the corresponding query vector and weight Where n p This is the preset number of sampling points for each query vector at each vehicle-end node. In the formula, A is the number of sampling points along N based on the linear mapping. v ×n p The sampling weights are obtained by performing a softmax operation on each point. At this point, each query vector... Based on its sampling bias and sampling weights value vector set Deformable sampling and weighted summation are performed, as shown in Equation 9.

[0106]

[0107] In the formula, DCA(q) represents performing a deformable cross-attention operation on the query vector q. W represents the feature of the sampled values ​​at the corresponding vehicle-side node based on the sampling bias. o This is a learnable parameter matrix used for output mapping. After deformable attention operations, the query features, which incorporate dynamic vehicle-end perception features, are then filled back into the central roadside BEV features to obtain the cloud-based collaborative fusion features under the corresponding modal branch. Compared to standard attention mechanisms, the deformable attention mechanism used in this scheme can significantly reduce computational overhead and minimize the impact of irrelevant noise points.

[0108] V. Dynamic Fusion of Cross-Modal Features

[0109] The aforementioned steps have preliminarily yielded cloud-to-cloud fusion characteristics considering different vehicle-end node modes. This section further performs cross-modal fusion, and adaptively adjusts the weights of the fused feature channels in a dynamic manner. The overall cross-modal fusion process is as follows: Figure 6 As shown, it mainly consists of two stages: static stitching and fusion, and dynamic correction and fusion. First, static stitching and fusion is shown in Equation 10:

[0110]

[0111] Where Concate(·) represents concatenation along the feature channels, and σ(·) is the ReLU activation function. The preliminary cross-modal fusion features are obtained. The fusion is performed from a static perspective, and the differences between channels are not well distinguished, which may lead to confusion of information perceived by different modalities. Therefore, this patent further performs dynamic correction fusion, as shown in Equations 11-12.

[0112] A dynamic =Sigmoid(σ(Linear(AvgPool)) global (F static (11)

[0113]

[0114] First, the statically fused features obtained above are subjected to global average pooling along the space, then encoded by a linear layer and a ReLU activation layer, and finally output by a Sigmoid function to represent a correction factor for dynamic channel adjustment. Finally, the dynamic correction factor is multiplied by the static fusion feature to obtain the final cloud-based fusion feature.

[0115] VI. Cloud-based Feature Decoding

[0116] This embodiment uses an anchor-frame-based approach to predict the target information. First, an anchor-frame feature map is generated in the cloud. Where n a This indicates the number of anchor frames in each grid cell, and 7 represents the characteristics of each anchor frame, namely, its 3D centroid coordinates, 3D dimensions, and yaw angle. Next, the aforementioned cloud-based fusion features are... The inputs are fed into the regression head reg_head and the classification head cls_head, respectively. Both are 2D convolutional neural networks, which output the pose deviation feature map at each grid position. and confidence plot Based on the predicted pose deviation and the pre-generated anchor box feature map, the true pose information of the predicted bounding box can be recovered. Then, FP detection boxes are removed based on the confidence threshold. Next, the cloud uses the Non-Maximum Suppression (NMS) algorithm to remove duplicate detection boxes, and finally obtains the detection target boxes within the cloud control range.

[0117] VII. Training Setup

[0118] In this embodiment, the overall training loss includes a regression loss term. and classification loss items The regression loss uses the smoothl1 loss function, as shown in Equation 13.

[0119]

[0120] Where Δx represents the difference between the predicted pose deviation and the actual pose deviation.

[0121] The classification loss uses the Focal loss function, as shown in Equation 14.

[0122]

[0123] Where y represents the sample classification label. This represents the confidence value of the positive sample predicted by the system. α and γ are hyperparameters used to alleviate the imbalance between positive and negative samples and the problem of difficult-to-classify samples, respectively. In this embodiment, their values ​​are 0.25 and 2.0, respectively.

[0124] To preserve the collaborative perception information extraction capability of the model under different modal branches and to promote the effectiveness of multimodal fusion, this scheme adopts a phased training strategy. First, without considering the multimodal settings of the vehicle-end node, the cloud collaborative fusion model under point cloud modality and image modality is trained separately. After ensuring that the model has collaborative perception capability under different modal branches, the parameters of the collaborative fusion module of different modal branches are fixed, and the dynamic cross-modal fusion module and decoding head are trained.

[0125] In summary, this solution proposes a cloud-based perception fusion system and method. This system uses multi-source perception information from both vehicle and roadside sources as input to achieve dynamic target detection within the cloud control range, providing target information at the edge cloud level for the vehicle-road-cloud system and overcoming the problem of insufficient single-vehicle perception. Addressing the issues of limited input perception information and susceptibility to individual perception noise in target-level cloud-based perception fusion, this patent employs a feature-level fusion approach to improve the robustness and accuracy of cloud-based fused perception. Furthermore, addressing the lack of consideration for multimodal individual perception in existing feature-level collaborative perception, this patent designs a hierarchical multimodal collaborative perception fusion method based on deformable attention, further improving the scalability and robustness of the cloud-based perception fusion system.

Claims

1. A feature-level multi-source multimodal cloud-based sensing fusion method, applied to a feature-level multi-source multimodal cloud-based sensing fusion system, characterized in that, This feature-level multi-source multimodal cloud-based perception fusion system includes sensors, connected intelligent vehicles, roadside equipment, wireless communication units, and a cloud computing platform. The sensors are used to collect raw perception data and their own pose information. The connected intelligent vehicle is a vehicle-side node (VN) with sensing and computing capabilities; The roadside equipment is a roadside node (RN) with the ability to communicate with the cloud computing platform. Different edge scenarios are configured with specific roadside equipment according to the traffic environment. Different scenarios are configured with different numbers and different perception modalities of roadside equipment. The wireless communication unit is installed on the connected intelligent vehicle; The cloud computing platform is used to receive VN perception information from connected intelligent vehicles and RN information from roadside equipment, and perform fusion processing to output dynamic target information. The feature-level multi-source multimodal cloud-based perception fusion method includes the following steps: S1. Vehicle-side and road-side data are acquired from multiple sources and multiple modes of perception. S2. Based on the acquired multi-source multimodal sensing data, encode the BEV sensing features under different modalities; S3: The vehicle and roadside transmit the BEV perception features to the cloud for spatial alignment processing. Then, multimodal branch collaborative feature fusion and cross-modal feature dynamic fusion are performed sequentially to output the cloud-fused BEV features. S4. Decode the cloud-integrated BEV features and output dynamic target information; The specific process of multimodal branch collaborative feature fusion in S3 is as follows: I. Query Vector Generation For the spatially aligned first BEV characteristics at the vehicle end ,in To determine the number of nodes on the vehicle side, first use a convolutional kernel with a size of 1. A 1-dimensional 2D convolutional layer and a sigmoid function are used to encode the BEV features of each vehicle to obtain the node information map of each vehicle. , ; Each grid position takes a value between 0 and 1. The closer the value is to 1, the more local location information is centered on that grid position. Then, the cloud performs... The information graph of each vehicle node is subjected to max pooling operation based on the grid position to obtain the overall information graph. This represents the overall information distribution of the sensing features provided by the dynamic vehicle-end node at this time. If some grid positions contain a lot of information, the cloud should take them into consideration. Conversely, if some grid positions contain a little information, they should be avoided to reduce the introduction of additional cooperative noise. The cloud-based system guides query vector generation based on the overall information graph. Using a greedy approach, it sorts the overall information graph from highest to lowest based on the amount of information evaluated, and then retrieves the top-ranked queries. The grid positions of each element are used to index the BEV (Build-Engineer Vehicle) features at the road end, in order to obtain a preliminary query vector set. ; II. Value Vector Generation The roadside BEV features are filled into the corresponding blank grid positions, and then modality-related value feature mapping is used to obtain the value features for attention operations: in, This refers to the linear mapping function of the value vector under the corresponding modal branch; Each vehicle-side BEV feature is flattened after value mapping encoding to obtain a value vector set. ,in , which represents the number of value vectors; III. Collaborative Feature Fusion Based on Deformable Cross-Attention First, global max pooling and average pooling operations are performed on the BEV features of each vehicle to obtain the following results: and ; Next, with the obtained query features Perform broadcast-based element summation to obtain query features with vehicle-end node awareness. ; Then based on two linear mapping layers and Output the sampling bias of the corresponding query vector and weight ,in This is the preset number of sampling points for each query vector under each vehicle-end node. Based on linear mapping along Execution at each point The sampling weights obtained from the operation are now used for each query vector. All can be based on their sampling bias and sampling weights Complete the set of value vectors Perform deformable sampling and weighted summation: in, Indicates the query vector Perform deformable cross-attention operations. This represents the feature of the sampled values ​​at the corresponding vehicle-side node based on the sampling bias. It is a learnable parameter matrix used for output mapping; After deformable attention operations, the query features, which incorporate dynamic vehicle-side perception features, are then filled into the loop-side BEV features to obtain the cloud-based collaborative fusion features under the corresponding modal branch. ,in, These are point cloud modal collaborative fusion features and image modal collaborative fusion features, respectively.

2. The feature-level multi-source multimodal cloud-based sensing fusion method according to claim 1, characterized in that, The roadside equipment is specifically a central roadside node (CRN), which is coupled to a cloud computing platform to provide stable sensing output for the cloud computing platform.

3. The feature-level multi-source multimodal cloud-based sensing fusion method according to claim 1, characterized in that, The sensors include perception-related sensors and positioning-related sensors, wherein the perception-related sensors include an RGB camera and a LiDAR, which are responsible for collecting raw perception data; Positioning-related sensors include GPS and IMU, which are responsible for acquiring their own pose information.

4. The feature-level multi-source multimodal cloud-based perception fusion method according to claim 1, characterized in that, The specific process of step S1 is as follows: The vehicle-side node (VN) acquires local perception data of the vehicle side based on surround-view RGB cameras or LiDAR, and obtains the vehicle's position and pose through GPS and IMU. The roadside node (RN) acquires roadside perception data based on LiDAR.

5. The feature-level multi-source multimodal cloud-based sensing fusion method according to claim 1, characterized in that, Step S2 specifically involves obtaining BEV perception features in point cloud mode or image mode based on encoding different perception models.

6. The feature-level multi-source multimodal cloud-based perception fusion method according to claim 5, characterized in that, The specific process of encoding and obtaining BEV perception features in point cloud modality in step S2 is as follows: For the original laser point cloud, point cloud preprocessing is performed. First, the point cloud is filtered based on the set detection range, and laser points reflected on themselves are removed to obtain the laser point set. Then, a point cloud pseudo-image is generated. By dividing the detection range, point cloud pillars are obtained. The features of each laser point are encoded, and the features of each point cloud pillar are obtained based on the max pooling operation, thus obtaining the pseudo-image features. The pseudo-image of the point cloud is input into a 2D convolutional backbone network to encode the BEV perception features in the point cloud mode.

7. The feature-level multi-source multimodal cloud-based perception fusion method according to claim 5, characterized in that, The specific process of encoding and obtaining BEV perception features in the image modality in step S2 is as follows: The Resent network is used to extract 2D image features from the view-around RGB image; Based on the set detection range, a three-dimensional voxel grid is divided with the origin of the vehicle coordinate system as the center. Then, based on the intrinsic and extrinsic parameter matrices of each camera, the three-dimensional center points of each grid are projected onto the visual feature map of each view. The image features of the projection points on each view are obtained based on bilinear interpolation. The panoramic image features of each projection point are obtained along the viewing angle and filled into the three-dimensional grid. Then, based on the 2D convolutional layer, the three-dimensional raster features are compressed along the height direction, and then the BEV perception features in the image modality are obtained by encoding through the 2D convolutional backbone network.

8. The feature-level multi-source multimodal cloud-based perception fusion method according to claim 1, characterized in that, The specific process of cross-modal feature dynamic fusion in step S3 is as follows: I. Static splicing and fusion The point cloud modal collaborative fusion features and the image modal collaborative fusion features are initially fused to obtain static fusion features. : in, This indicates splicing along the feature channel. It is the ReLU activation function; II. Dynamic Correction and Fusion The statically fused features are subjected to global average pooling along the space, then encoded by a linear layer and a ReLU activation layer, and finally output by a Sigmoid function to represent a correction factor for dynamic channel adjustment. Finally, the dynamic correction factor is combined with the static fusion feature to calculate the Hadamard product, thus obtaining the cloud-based fusion feature. .

Images