Automatic driving cooperative sensing method and system based on 3D Gaussian sputtering

Abstract

The invention relates to an automatic driving cooperative sensing method and system based on 3D Gaussian sputtering. The system is composed of a sender vehicle and a receiver vehicle. The sender vehicle extracts 2D features from the multi-view image, discretizes the space into a voxel grid, and converts the 2D features into 3D features through projection; performing feature decoding on the 3D voxel features, and performing optimization adjustment on a preset 3D Gaussian in each voxel grid; the sender screens the 3D gausses containing the semantic information and transmits the 3D gausses to the receiver; and the receiver vehicle aligns and aggregates the received 3D gauss from the sender with the own three-dimensional coordinate system, performs Gaussian rasterization processing on the aggregated 3D gauss, and maps the Gaussian representation to the view angle of the aerial view through the 3D Gaussian splashing process to obtain a BEV semantic segmentation result. According to the method, Gaussian representation can be promoted to be used for cooperative sensing to remarkably improve the sensing precision, and the optimal sensing precision is realized on the premise of ensuring the real-time performance.

Description

Technical Field

[0001] This invention belongs to the field of vehicle-to-everything (V2X) wireless communication technology, specifically relating to an autonomous driving cooperative perception method and system based on 3D Gaussian sputtering. Background Technology

[0002] For autonomous vehicles to operate safely on the road, real-time and accurate perception of surrounding traffic conditions is crucial. Currently, mainstream autonomous vehicles primarily rely on onboard sensors such as cameras, millimeter-wave radar, and lidar to perceive their environment. However, no matter how advanced a single sensor is, it is difficult to avoid perception errors caused by factors such as damage, obstructed vision, limited detection range, or inclement weather. Therefore, the perception capabilities of a single vehicle are far from sufficient to meet the stringent safety requirements of autonomous driving. In light of this, with the advancement of wireless communication technology, the concept of "collaborative perception" has been proposed. The core of this technology is to utilize V2X (vehicle-to-everything) wireless communication to achieve the sharing of perception data between vehicles, thereby effectively expanding the perception boundaries of vehicles.

[0003] Currently, mainstream collaboration awareness solutions are divided into three main categories based on the types of data shared:

[0004] (1) Collaborative perception based on raw data: Vehicles directly share unprocessed raw sensor data. This can preserve information to the greatest extent, but the amount of raw data is huge, which will put a lot of pressure on the wireless communication channel;

[0005] (2) Collaborative perception based on result data: Vehicles only share the detection results processed by the target detection model. This results in a very small amount of data, but it is highly dependent on the perception capabilities of each vehicle and has very limited improvement on the perception effect of the receiver.

[0006] (3) Feature-based collaborative perception shares partially processed feature data, achieving a good balance between data volume and perception effect. Currently, the mainstream intermediate feature form is BEV (Browser-Earned Object) features, which align the feature space to a unified BEV perspective, suitable for various tasks. However, BEV features are dense, and in actual communication environments, transmitting such BEV features completely remains difficult. In addition, BEV features lack height information, making it difficult to losslessly aggregate information about objects at different heights at the same location in collaborative perception systems. Therefore, in complex scenarios, using BEV features for sharing cannot achieve both high performance and high efficiency, affecting the effectiveness of collaborative perception. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide an autonomous driving cooperative perception method and system based on 3D Gaussian sputtering. Addressing the problem that existing cooperative perception methods cannot achieve high performance and high efficiency, this invention focuses on a key application in the perception system—BEV (bird's-eye view) semantic segmentation. It aims to expand the vehicle perception range and improve the accuracy of BEV semantic segmentation while flexibly adapting to the dynamic changes in wireless bandwidth, thereby effectively ensuring the real-time performance of BEV semantic segmentation.

[0008] The technical problem solved by this invention is achieved through the following technical solution:

[0009] A collaborative perception method for autonomous driving based on 3D Gaussian sputtering, the steps of which are as follows:

[0010] S1, The sending vehicle and the receiving vehicle are viewed from multiple perspective images I1,...,I N Extracting 2D features from images The image 2D features include the edges, corners, or unique textures of objects; ε(·) represents the image 2D feature extraction function, which uses a pre-trained ResNet34 / 50;

[0011] S2. Convert the 2D features of the image into 3D features of the image. Including depth estimation and precise coordinate transformation, the two-dimensional image information is accurately mapped to the three-dimensional voxel space V, where T(·) represents the 2D to 3D feature transformation function. This function uses ray projection to uniformly distribute each 2D pixel feature under each camera viewpoint to the 3D space along the camera ray, as shown in Equation (1):

[0012]

[0013] Where (u, v) are pixel coordinates, and (x, y, z) are equidistant sampling points along the camera light rays. Multi-view feature fusion adopts a simple average, as shown in Equation (2):

[0014]

[0015] Wherein: T i Let N be the extrinsic transformation matrix of the i-th camera, and N be the number of cameras;

[0016] S3. In the three-dimensional voxel space V, 3D voxel features are decoded by 3D convolution, discretizing the continuous 3D space into a series of regular voxel grids. At the same time, for each voxel v j Preset Gaussian distribution g init Perform attribute updates; the updated Gaussian representation Each Gaussian is defined as: g k (μk ,Σ k ,c k ,α k ), from the mean μ k Covariance Σ k Opacity α k and semantic information c k The updated Gaussian model more accurately captures and represents detailed information in 3D scenes.

[0017] S4. The sending vehicle performs semantic Gaussian filtering to remove redundant or unimportant information. The filtering process is based on the importance score of the Gaussian, where the importance score is calculated using the Gaussian algorithm. k The calculation method for j is as shown in equation (3):

[0018]

[0019] Where: h k Let α be the height value after Gaussian normalization. weight h weight λ is the weight of the corresponding coefficient, C is the total number of semantic categories, and the screening process is defined as shown in equation (4):

[0020] G filtered ={g k ∈G∣score(g k (4)

[0021] Where: τ is the threshold corresponding to the percentage calculated based on the current network bandwidth, and G filtered These are the Gaussians remaining after filtering;

[0022] Subsequently, the filtered Gaussian data was compressed. compressed =C(G filtered To reduce the amount of data, the compression function C(·) employs quantization and codebook techniques;

[0023] The optimized Gaussian data is transmitted to the receiving vehicle via wireless communication.

[0024] S5. After the receiving vehicle receives the sensing data, it needs to register the received sensing data. Using the GPS and IMU data of the sending and receiving vehicles, the rotation matrix and translation vector are calculated. The coordinate system of the sending vehicle and the coordinate system of the receiving vehicle are unified through the above transformation. The rotation matrix R is calculated by equation (5):

[0025] R = R z (θ yaw )R y (θ pitch )R x (θ roll(5)

[0026] Where: θ yaw ,θ pitch ,θ roll These are the differences between the yaw angle, pitch angle, and roll angle, respectively.

[0027] The coordinates of the sending vehicle data are calibrated for steering and displacement, and the translation vector is calculated as shown in equation (6):

[0028]

[0029] Where: (X) s ,Y s Z s ) and (X' s ,Y' s ,Z' s ( ) represent the coordinate systems of the sender data before and after registration, respectively; (Δd x ,Δd y ,Δd z () indicates the displacement difference between the two vehicles;

[0030] S6. The calibrated 3D Gaussian data of the sending vehicle is fused with the original 3D Gaussian data of the receiving vehicle in S3. The fused 3D Gaussian data is expressed as Equation (7):

[0031] G f =g r ∪g' s (7)

[0032] Among them: G f g r g' s These represent the fused global 3D Gaussian data, the receiver vehicle's 3D Gaussian data, and the registered sender vehicle's 3D Gaussian data, respectively.

[0033] S7. Finally, the fused 3D Gaussian data G needs to be processed. f Rasterization is performed and sputtered into the camera's view from the BEV perspective to obtain the final BEV semantic segmentation result. The specific process is described in formula (8).

[0034]

[0035] Where: C(u,v) represents the final semantic output at BEV pixel (u,v), P represents the total number of all relevant 3D Gaussians, and T k (u,v) represents the cumulative opacity of all Gaussians up to the last Gaussian along the pixel's line of sight. The two-dimensional coordinates μ of the BEV pixel k ' represents the three-dimensional mean μ of Gaussian kk The two-dimensional mean projected onto the BEV plane, Σ k 'Represents the three-dimensional covariance Σ of Gaussian k k Two-dimensional covariance projected onto the BEV plane.

[0036] An autonomous driving cooperative perception system based on 3D Gaussian sputtering, the system comprising:

[0037] The image 2D feature extraction module extracts high-dimensional abstract features from input images from multiple camera perspectives for downstream tasks. This module outputs a 2D feature set.

[0038] The image 2D feature conversion module utilizes camera parameters to map 2D features into a discretized 3D space, and fuses multiple camera views into a unified 3D space, outputting 3D voxel features downstream.

[0039] The 3D voxel feature Gaussian update module optimizes Gaussian attributes, including position, covariance, and semantic information, by decoding 3D voxel features in the voxel mesh, thereby improving the fit between scene geometry and semantics and outputting the Gaussian set corresponding to the scene.

[0040] The semantic Gaussian filtering and compression module, through the 3D Gaussian data registration module, requires the receiver to register the Gaussian data after receiving it, transforming it from the coordinate system of other vehicles to its own coordinate system. This module outputs a set of Gaussian data after coordinate transformation from other vehicles.

[0041] The 3D Gaussian data fusion module aggregates the Gaussian data sets of different vehicles and outputs the complete aggregated Gaussian data set.

[0042] The Gaussian splash raster processing module splashes Gaussian rasterization onto the specified camera BEV viewpoint and processes the semantic attributes of the Gaussian rasterization, ultimately outputting the BEV semantic segmentation result.

[0043] The advantages and beneficial effects of this invention are as follows:

[0044] 1. This invention proposes an end-to-end autonomous driving collaborative perception framework that supports the generation of 3D Gaussian representations and the generation of BEV semantic segmentation results, aiming to promote the use of Gaussian representations for collaborative perception to significantly improve perception accuracy.

[0045] 2. This invention proposes a bandwidth-adaptive data filtering and compression scheme. This scheme aims to intelligently filter and compress 3D Gaussian sensing data to adapt to different network bandwidth conditions, thereby achieving optimal sensing accuracy while ensuring real-time performance.

[0046] 3. This invention can be applied to various BEV semantic segmentation models and supports intelligent connected vehicles with different computing capabilities. Attached Figure Description

[0047] Figure 1 This is a flowchart of the image-based BEV semantic segmentation process of the present invention;

[0048] Figure 2 This is a flowchart of the collaborative perception process based on 3D Gaussian in this invention;

[0049] Figure 3 This is a schematic diagram of the sensing data registration of the present invention. Detailed Implementation

[0050] The present invention will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present invention.

[0051] This invention proposes a cooperative perception system based on 3D Gaussian sputtering, specifically designed for semantic segmentation tasks in BEVs (Battery Electric Vehicles) for autonomous driving. Figure 1 The BEV semantic segmentation task shown is as follows: Figure 2 The system configuration is shown. The system consists of a sending vehicle and a receiving vehicle, which transmit sensing data using V2V wireless communication technologies such as DSRC and LTE-4G.

[0052] The sending vehicle first extracts 2D features, such as edges, corners, and textures, from multi-view images. These features capture local information about the images. Subsequently, these 2D features are converted into 3D features, a process involving depth estimation and coordinate transformation, mapping the image information into a three-dimensional voxel space. Next, the system performs feature decoding on these 3D voxel features, discretizing the continuous 3D space into a voxel grid and optimizing a pre-defined Gaussian distribution within each voxel. Based on this, a Gaussian representation of the space can be generated, accurately describing objects and structures in the space. Finally, the sending vehicle filters these semantic Gaussians, removing redundant or unimportant information and compressing them to optimize data transmission efficiency and storage space.

[0053] After receiving perception data from the sender, the receiving vehicle undergoes a similar processing flow. First, it extracts 2D features from multi-view images and converts them into 3D Gaussian representations, ensuring data consistency. Next, unlike the sender, the receiving vehicle aligns the received 3D Gaussian representation from the sender with its own 3D coordinate system to obtain a more comprehensive global 3D Gaussian representation of the scene. Subsequently, Gaussian rasterization is performed, mapping the Gaussian representation to the BEV's perspective through a 3D Gaussian splashing process. This ultimately yields the BEV (bird's-eye view) semantic segmentation result, which categorizes objects and regions in the 3D scene into different semantic categories, providing crucial perception information for subsequent decision-making and planning.

[0054] An autonomous driving collaborative perception method based on 3D Gaussian sputtering, the innovation of which lies in the following steps:

[0055] S1, The sending vehicle and the receiving vehicle are viewed from multiple perspective images I1,...,I N Extracting 2D features from images The image 2D features include the edges, corners, or unique textures of objects; ε(·) represents the image 2D feature extraction function, which uses a pre-trained ResNet34 / 50;

[0056] S2. Convert the 2D features of the image into 3D features of the image. Including depth estimation and precise coordinate transformation, the two-dimensional image information is accurately mapped to the three-dimensional voxel space V, where T(·) represents the 2D to 3D feature transformation function. This function uses ray projection to uniformly distribute each 2D pixel feature under each camera viewpoint to the 3D space along the camera ray, as shown in Equation (1):

[0057]

[0058] Where (u, v) are pixel coordinates, and (x, y, z) are equidistant sampling points along the camera light rays. Multi-view feature fusion adopts a simple average, as shown in Equation (2):

[0059]

[0060] Wherein: T i Let N be the extrinsic transformation matrix of the i-th camera, and N be the number of cameras;

[0061] S3. In the three-dimensional voxel space V, 3D voxel features are decoded by 3D convolution, discretizing the continuous 3D space into a series of regular voxel grids. At the same time, for each voxel v j Preset Gaussian distribution g initPerform attribute updates; the updated Gaussian representation Each Gaussian is defined as: g k (μ k ,Σ k ,c k ,α k ), from the mean μ k Covariance Σ k Opacity α k and semantic information c k The updated Gaussian model more accurately captures and represents detailed information in 3D scenes.

[0062] S4. The sending vehicle performs semantic Gaussian filtering to remove redundant or unimportant information. The filtering process is based on the importance score of the Gaussian, where the importance score is calculated using the Gaussian algorithm. k The calculation method for j is as shown in equation (3):

[0063]

[0064] Where: h k Let α be the height value after Gaussian normalization. weight h weight λ is the weight of the corresponding coefficient, C is the total number of semantic categories, and the screening process is defined as shown in equation (4):

[0065] G filtered ={g k ∈G∣score(g k (4)

[0066] Where: τ is the threshold corresponding to the percentage calculated based on the current network bandwidth, and G filtered These are the Gaussians remaining after filtering;

[0067] Subsequently, the filtered Gaussian data was compressed. compressed =C(G filtered To reduce the amount of data, the compression function C(·) employs quantization and codebook techniques;

[0068] The optimized Gaussian data is transmitted to the receiving vehicle via wireless communication.

[0069] S5, such as Figure 3 As shown, after the receiving vehicle receives the sensing data, it needs to register the received sensing data first. Using the GPS and IMU data of the sending and receiving vehicles, the rotation matrix and translation vector are calculated. Through the above transformation, the coordinate system of the sending vehicle and the coordinate system of the receiving vehicle are unified. The rotation matrix R is calculated by equation (5):

[0070] R = R z (θyaw )R y (θ pitch )R x (θ roll (5)

[0071] Where: θ yaw ,θ pitch ,θ roll These are the differences between the yaw angle, pitch angle, and roll angle, respectively.

[0072] The coordinates of the sending vehicle data are calibrated for steering and displacement, and the translation vector is calculated as shown in equation (6):

[0073]

[0074] Where: (X) s ,Y s Z s ) and (X' s ,Y' s ,Z' s ( ) represent the coordinate systems of the sender data before and after registration, respectively; (Δd x ,Δd y ,Δd z () indicates the displacement difference between the two vehicles;

[0075] S6. The calibrated 3D Gaussian data of the sending vehicle is fused with the original 3D Gaussian data of the receiving vehicle in S3. The fused 3D Gaussian data is expressed as Equation (7):

[0076] G f =g r ∪g' s (7)

[0077] Among them: G f g r g' s These represent the fused global 3D Gaussian data, the receiver vehicle's 3D Gaussian data, and the registered sender vehicle's 3D Gaussian data, respectively.

[0078] S7. Finally, the fused 3D Gaussian data G needs to be processed. f Rasterization is performed and sputtered into the camera's view from the BEV perspective to obtain the final BEV semantic segmentation result. The specific process is described in formula (8).

[0079]

[0080] Where: C(u,v) represents the final semantic output at BEV pixel (u,v), P represents the total number of all relevant 3D Gaussians, and T k(u,v) represents the cumulative opacity of all Gaussians up to the last Gaussian along the pixel's line of sight. The two-dimensional coordinates μ of the BEV pixel k ' represents the three-dimensional mean μ of Gaussian k k The two-dimensional mean projected onto the BEV plane, Σ k 'Represents the three-dimensional covariance Σ of Gaussian k k Two-dimensional covariance projected onto the BEV plane.

[0081] An autonomous driving cooperative perception system based on 3D Gaussian sputtering, the innovation of which is that the system includes:

[0082] The image 2D feature extraction module extracts high-dimensional abstract features from input images from multiple camera perspectives for downstream tasks. This module outputs a 2D feature set.

[0083] The image 2D feature conversion module utilizes camera parameters to map 2D features into a discretized 3D space, and fuses multiple camera views into a unified 3D space, outputting 3D voxel features downstream.

[0084] The 3D voxel feature Gaussian update module optimizes Gaussian attributes, including position, covariance, and semantic information, by decoding 3D voxel features in the voxel mesh, thereby improving the fit between scene geometry and semantics and outputting the Gaussian set corresponding to the scene.

[0085] The semantic Gaussian filtering and compression module, through the 3D Gaussian data registration module, requires the receiver to register the Gaussian data after receiving it, transforming it from the coordinate system of other vehicles to its own coordinate system. This module outputs a set of Gaussian data after coordinate transformation from other vehicles.

[0086] The 3D Gaussian data fusion module aggregates the Gaussian data sets of different vehicles and outputs the complete aggregated Gaussian data set.

[0087] The Gaussian splash raster processing module splashes Gaussian rasterization onto the specified camera BEV viewpoint and processes the semantic attributes of the Gaussian rasterization, ultimately outputting the BEV semantic segmentation result.

[0088] Although embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A cooperative perception method for autonomous driving based on 3D Gaussian sputtering, characterized in that: The steps of the method are as follows: S1, The sending vehicle and the receiving vehicle are viewed from multiple perspective images I1,...,I N Extracting 2D features from images The image 2D features include the edges, corners, or unique textures of objects; ε(·) represents the image 2D feature extraction function, which uses a pre-trained ResNet34 / 50; S2. Convert the 2D features of the image into 3D features of the image. Including depth estimation and precise coordinate transformation, the two-dimensional image information is accurately mapped to the three-dimensional voxel space V, where T(·) represents the 2D to 3D feature transformation function. This function uses ray projection to uniformly distribute each 2D pixel feature under each camera viewpoint to the 3D space along the camera ray, as shown in Equation (1): Where (u, v) are pixel coordinates, and (x, y, z) are equidistant sampling points along the camera light rays. Multi-view feature fusion adopts a simple average, as shown in Equation (2): Wherein: T i Let N be the extrinsic transformation matrix of the i-th camera, and N be the number of cameras; S3. In the three-dimensional voxel space V, 3D voxel features are decoded by 3D convolution, discretizing the continuous 3D space into a series of regular voxel grids. At the same time, for each voxel v j Preset Gaussian distribution g init Perform attribute updates; the updated Gaussian representation Each Gaussian is defined as: g k (μ k ,Σ k ,c k ,α k ), from the mean μ k Covariance Σ k Opacity α k and semantic information c k The updated Gaussian model more accurately captures and represents detailed information in 3D scenes. S4. The sending vehicle performs semantic Gaussian filtering to remove redundant or unimportant information. The filtering process is based on the importance score of the Gaussian, where the importance score is calculated using the Gaussian algorithm. k The calculation method for j is as shown in equation (3): Where: h k Let α be the height value after Gaussian normalization. weight h weight λ is the weight of the corresponding coefficient, C is the total number of semantic categories, and the screening process is defined as shown in equation (4): G filtered ={g k ∈G∣score(g k )>τ} (4) Where: τ is the threshold corresponding to the percentage calculated based on the current network bandwidth, and G filtered These are the Gaussians remaining after filtering; Subsequently, the filtered Gaussian data was compressed. compressed =C(G filtered To reduce the amount of data, the compression function C(·) employs quantization and codebook techniques; The optimized Gaussian data is transmitted to the receiving vehicle via wireless communication. S5. After the receiving vehicle receives the sensing data, it needs to register the received sensing data. Using the GPS and IMU data of the sending and receiving vehicles, the rotation matrix and translation vector are calculated. The coordinate system of the sending vehicle and the coordinate system of the receiving vehicle are unified through the above transformation. The rotation matrix R is calculated by equation (5): R=R z (i yaw )R y (i pitch )R x (i roll ); (5) Where: θ yaw ,θ pitch ,θ roll These are the differences between the yaw angle, pitch angle, and roll angle, respectively. The coordinates of the sending vehicle data are calibrated for steering and displacement, and the translation vector is calculated as shown in equation (6): Where: (X) s ,Y s Z s ) and (X' s ,Y' s ,Z' s ( ) represent the coordinate systems of the sender data before and after registration, respectively; (Δd x ,Δd y ,Δd z () indicates the displacement difference between the two vehicles; S6. The calibrated 3D Gaussian data of the sending vehicle is fused with the original 3D Gaussian data of the receiving vehicle in S3. The fused 3D Gaussian data is expressed as Equation (7): G f =g r ∪g' s ; (7) Among them: G f g r g' s These represent the fused global 3D Gaussian data, the receiver vehicle's 3D Gaussian data, and the registered sender vehicle's 3D Gaussian data, respectively. S7. Finally, the fused 3D Gaussian data G needs to be processed. f Rasterization is performed and sputtered into the camera's view from the BEV perspective to obtain the final BEV semantic segmentation result. The specific process is described in formula (8). Where: C(u,v) represents the final semantic output at BEV pixel (u,v), P represents the total number of all relevant 3D Gaussians, and T k (u,v) represents the cumulative opacity of all Gaussians up to the last Gaussian along the pixel's line of sight. The two-dimensional coordinates μ of the BEV pixel k ' represents the three-dimensional mean μ of Gaussian k k The two-dimensional mean projected onto the BEV plane, Σ k 'Represents the three-dimensional covariance Σ of Gaussian k k Two-dimensional covariance projected onto the BEV plane.

2. An autonomous driving cooperative perception system based on 3D Gaussian sputtering, characterized in that: The system includes: The image 2D feature extraction module extracts high-dimensional abstract features from input images from multiple camera perspectives for downstream tasks. This module outputs a 2D feature set. The image 2D feature conversion module utilizes camera parameters to map 2D features into a discretized 3D space, and fuses multiple camera views into a unified 3D space, outputting 3D voxel features downstream. The 3D voxel feature Gaussian update module optimizes Gaussian attributes, including position, covariance, and semantic information, by decoding 3D voxel features in the voxel mesh, thereby improving the fit between scene geometry and semantics and outputting the Gaussian set corresponding to the scene. The semantic Gaussian filtering and compression module, through the 3D Gaussian data registration module, requires the receiver to register the Gaussian data after receiving it, transforming it from the coordinate system of other vehicles to its own coordinate system. This module outputs a set of Gaussian data after coordinate transformation from other vehicles. The 3D Gaussian data fusion module aggregates the Gaussian data sets of different vehicles and outputs the complete aggregated Gaussian data set. The Gaussian splash raster processing module splashes Gaussian rasterization onto the specified camera BEV viewpoint and processes the semantic attributes of the Gaussian rasterization, ultimately outputting the BEV semantic segmentation result.

Images