Data processing method, device and equipment and computer readable storage medium

Abstract

A data processing method, apparatus and device, and a computer readable storage medium relate to the technical field of target detection, and comprise the steps of obtaining millimeter-wave radar point voxel data corresponding to millimeter-wave radar data; encoding the millimeter wave radar point voxel data to generate target voxel sequence features; performing multi-layer linear calculation on the target voxel sequence features to generate query sequence features, key sequence features and value sequence features; calculating the query sequence features and the key sequence features based on an attention mechanism to obtain an autocorrelation matrix; and coding the autocorrelation matrix and the value sequence features through a multi-head self-attention mechanism, and performing residual connection on a coding result and the target voxel sequence features to generate millimeter wave radar autocorrelation enhancement sequence features. According to the method and the device, the sparsity problem of the Radar data can be effectively improved, so that the utilization efficiency of the Radar data is improved.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a data processing method, device, equipment and computer-readable storage medium. Background Art

[0002] At present, in order to improve driving safety and convenience and enhance the performance of intelligent driving systems, vehicles are often equipped with a variety of sensors such as LiDAR, millimeter-wave radar and cameras. By fusing data from different types of sensors, more comprehensive and accurate information about the vehicle's surrounding environment is obtained to provide a data basis for target detection, thereby better supporting assisted driving functions such as automatic following, adaptive cruise control, lane keeping, and automatic parking.

[0003] Among them, Radar data is often sparse due to the limitation of millimeter wave radar resolution, environmental interference, multipath effect, and the shape, size, material and surface roughness of the detected target. However, when realizing multi-sensor data fusion, the relevant technology usually ignores the sparsity problem of Radar data, which makes it easy to mismatch during the data fusion process, resulting in poor target detection accuracy. It can be seen that how to improve the sparsity problem of Radar data and improve the utilization efficiency of Radar data is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a data processing method, apparatus, device and computer-readable storage medium, which can effectively improve the sparsity problem of Radar data to improve the utilization efficiency of Radar data.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the data processing method comprising:

[0006] Obtaining millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data;

[0007] Encoding millimeter wave radar point voxel data to generate target voxel sequence features;

[0008] Performing multi-layer linear calculations on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features;

[0009] The query sequence features and key sequence features are calculated based on the attention mechanism to obtain the autocorrelation matrix;

[0010] The autocorrelation matrix and value sequence features are encoded through a multi-head self-attention mechanism, and the encoding results and the target voxel sequence features are residually connected to generate millimeter-wave radar autocorrelation enhanced sequence features.

[0011] In combination with the first aspect, in one implementation, after the step of generating the millimeter wave radar autocorrelation enhancement sequence feature, the method further includes:

[0012] Determine the index coordinates based on the voxel coordinates corresponding to the millimeter wave radar autocorrelation enhancement sequence characteristics and the perception range of the bird's eye view;

[0013] The millimeter-wave radar autocorrelation enhancement sequence features are mapped to millimeter-wave radar bird's-eye view features based on the index coordinates.

[0014] In combination with the first aspect, in one implementation, the calculation formula of the index coordinate is:

[0015] coords=r·vy+vx

[0016] Where coords represents the index coordinates, r represents the perceptual range of the bird's-eye view, vy represents the voxel vertical axis coordinate, and vx represents the voxel horizontal axis coordinate.

[0017] In combination with the first aspect, in one implementation, after the step of obtaining the millimeter wave radar bird's-eye view feature, the method further includes:

[0018] Obtaining lidar bird's eye view features corresponding to the lidar data and camera bird's eye view features corresponding to the camera image data;

[0019] The millimeter-wave radar bird's-eye view features, the lidar bird's-eye view features, and the camera bird's-eye view features are respectively transformed in feature dimension, and all the transformed bird's-eye view features are spliced ​​to obtain spliced ​​features;

[0020] Using different activation functions to perform feature activation operations on the concatenated features to obtain different learnable attention weights;

[0021] The learnable attention weights and their corresponding bird's-eye view features are weighted respectively to obtain the target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features and the target camera bird's-eye view features;

[0022] The target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features and the target camera bird's-eye view features are spliced ​​to generate fusion features.

[0023] In combination with the first aspect, in one implementation, the step of converting the feature dimensions of the millimeter wave radar bird's eye view features, the laser radar bird's eye view features, and the camera bird's eye view features respectively includes:

[0024] The millimeter-wave radar bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted millimeter-wave radar bird's-eye view features;

[0025] The lidar bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted lidar bird's-eye view features;

[0026] The camera bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted camera bird's-eye view features.

[0027] In combination with the first aspect, in one implementation, the calculation formula of the autocorrelation matrix is:

[0028]

[0029] Where M auto represents the autocorrelation matrix, Q represents the query sequence characteristics, K T represents the transpose of the key sequence characteristics, C K Feature channel dimension representing key sequence features.

[0030] In combination with the first aspect, in one implementation, the expression of the millimeter wave radar autocorrelation enhancement sequence feature is:

[0031]

[0032] In the formula, represents the millimeter-wave radar autocorrelation enhanced sequence feature, MultiHeadAttention represents the multi-head attention module, V represents the value sequence feature, head represents the number of heads in the multi-head attention module, and f radar Represents the target voxel sequence features.

[0033] In a second aspect, an embodiment of the present application provides a data processing device, wherein the data processing device includes an autocorrelation point enhancement module, which is used to:

[0034] Obtaining millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data;

[0035] Encoding millimeter wave radar point voxel data to generate target voxel sequence features;

[0036] Performing multi-layer linear calculations on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features;

[0037] The query sequence features and key sequence features are calculated based on the attention mechanism to obtain the autocorrelation matrix;

[0038] The autocorrelation matrix and value sequence features are encoded through a multi-head self-attention mechanism, and the encoding results and the target voxel sequence features are residually connected to generate millimeter-wave radar autocorrelation enhanced sequence features.

[0039] In conjunction with the second aspect, in one implementation, the apparatus further includes an autocorrelation Radar feature allocation module, which is used to:

[0040] Determine the index coordinates based on the voxel coordinates corresponding to the millimeter wave radar autocorrelation enhancement sequence characteristics and the perception range of the bird's eye view;

[0041] The millimeter-wave radar autocorrelation enhancement sequence features are mapped to millimeter-wave radar bird's-eye view features based on the index coordinates.

[0042] In conjunction with the second aspect, in one implementation, the calculation formula for the index coordinate is:

[0043] coords=r·vy+vx

[0044] Where coords represents the index coordinates, r represents the perceptual range of the bird's-eye view, vy represents the voxel vertical axis coordinate, and vx represents the voxel horizontal axis coordinate.

[0045] In conjunction with the second aspect, in one embodiment, the device further includes an adaptability module, which is used to:

[0046] Obtaining lidar bird's-eye view features corresponding to the lidar data and camera bird's-eye view features corresponding to the camera image data;

[0047] The millimeter-wave radar bird's-eye view features, the lidar bird's-eye view features and the camera bird's-eye view features are respectively transformed in feature dimension, and all the transformed bird's-eye view features are spliced ​​to obtain spliced ​​features;

[0048] Using different activation functions to perform feature activation operations on the concatenated features to obtain different learnable attention weights;

[0049] The learnable attention weights and their corresponding bird's-eye view features are weighted respectively to obtain the target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features and the target camera bird's-eye view features;

[0050] The target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features and the target camera bird's-eye view features are spliced ​​to generate fusion features.

[0051] In conjunction with the second aspect, in one implementation, the adaptability module is specifically used to:

[0052] The millimeter-wave radar bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted millimeter-wave radar bird's-eye view features;

[0053] The lidar bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted lidar bird's-eye view features;

[0054] The camera bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted camera bird's-eye view features.

[0055] In conjunction with the second aspect, in one implementation, the calculation formula of the autocorrelation matrix is:

[0056]

[0057] Where M auto represents the autocorrelation matrix, Q represents the query sequence characteristics, K T represents the transpose of the key sequence characteristics, C K Feature channel dimension representing key sequence features.

[0058] In conjunction with the second aspect, in one implementation, the millimeter wave radar autocorrelation enhancement sequence feature is expressed as:

[0059]

[0060] In the formula, represents the millimeter-wave radar autocorrelation enhanced sequence feature, MultiHeadAttention represents the multi-head attention module, V represents the value sequence feature, head represents the number of heads in the multi-head attention module, and f radar Represents the target voxel sequence features.

[0061] In a third aspect, an embodiment of the present application provides a data processing device, comprising a processor, a memory, and a data processing program stored in the memory and executable by the processor, wherein when the data processing program is executed by the processor, the steps of the data processing method as described above are implemented.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a data processing program is stored, wherein when the data processing program is executed by a processor, the steps of the aforementioned data processing method are implemented.

[0063] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0064] The target voxel sequence features are generated by encoding the Radar point voxel data corresponding to the millimeter-wave radar Radar data, so as to convert the Radar point information from a specific geometric position to a high-dimensional space; then, multi-layer linear calculations are performed on the target voxel sequence features to generate query sequence features, key sequence features and value sequence features to achieve high-dimensional encoding; then, in order to obtain the autocorrelation Radar point enhancement results, the correlation between the query sequence features and the key sequence features is calculated based on the attention mechanism to generate an autocorrelation matrix; finally, in order to ensure the intrinsic connection of the original Radar point sequence features, the autocorrelation matrix and the value sequence features are encoded using multi-head self-attention encoding, and the encoding results and the target voxel sequence features are residually connected to generate the Radar autocorrelation enhancement sequence features, thereby effectively improving the sparsity of Radar data and improving the utilization efficiency of Radar data. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flow chart of an embodiment of the data processing method of the present application;

[0066] Figure 2 A schematic diagram of the Pillar encoding method involved in the embodiment of the present application;

[0067] Figure 3 This is a schematic diagram of feature allocation involved in the embodiment of the present application;

[0068] Figure 4 This is a schematic diagram of the target detection process involved in the embodiment of the present application;

[0069] Figure 5 This is a schematic diagram of the feature fusion process involved in the embodiment of the present application;

[0070] Figure 6 This is a schematic diagram of the hardware structure of the data processing device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0072] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0073] In a first aspect, an embodiment of the present application provides a data processing method.

[0074] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the data processing method of this application. Figure 1 As shown, the data processing method includes:

[0075] Step S10: Acquire millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data.

[0076] Exemplarily, in this embodiment, the mathematical expression of the millimeter wave radar point Radar is first defined as: Where N represents the total number of Radar points collected by the millimeter-wave radar in a certain driving scene; D represents the information contained in a Radar point, such as the coordinates (x, y, z) of the Radar point, the reflection value of the Radar point (the reflection value corresponds to the confidence of the Radar point. If the reflection value is lower than a certain threshold, the Radar point will not be recorded) and the speed value of the Radar point, etc.; P Radar represents a point set of N×D dimensions. Then Radar Voxelization is performed, wherein in the voxelization process, the three-dimensional space is divided into a series of small cubes (i.e., voxels) according to a preset voxel size, and P Radar Each point in is assigned to a corresponding voxel to obtain Radar point voxel data; it should be noted that the specific method and principle of voxelization processing are common knowledge in the art, so for the sake of brevity of description, they will not be repeated here.

[0077] Step S20: Encode the millimeter-wave radar point voxel data to generate target voxel sequence features.

[0078] For example, it should be understood that, currently, after completing the voxelization of the Radar point, the Radar point voxel data is usually directly encoded as a BEV (Bird's Eye View) feature, so that the exploration of the relationship between points in the Radar is ignored, which leads to the generated BEV feature having the disadvantage of sparseness. Therefore, in order to solve the data sparsity problem, the present embodiment can preferably use Pillar encoding (which is an encoding method that divides the three-dimensional point cloud data into a series of two-dimensional grid Pillars evenly distributed on the xy plane) to encode the voxelized data (i.e., the Radar point voxel data) into a set of voxel features, that is, to obtain the target voxel sequence feature, which can be expressed in mathematical symbols as follows: Where N is the same as P above Radar N corresponds to, m represents P RadarThe dimension in which the D-dimensional information is encoded can be determined according to actual needs. For example, m = 16, then Table P Radar The D-dimensional information in is encoded into 16 dimensions.

[0079] It should be noted that when using Pillar coding to encode the Radar point voxel data into the target voxel sequence feature f radar When the voxel is divided, it is mainly all the voxel area points on each entire height (i.e. Figure 2 All Radar points contained in the dotted box in the figure are uniformly encoded instead of encoding each voxel block area to avoid generating invalid features. However, the specific encoding method and principle are common knowledge in the art, so for the sake of brevity, they will not be repeated here.

[0080] Step S30: performing multi-layer linear calculations on the target voxel sequence features to generate query sequence features, key sequence features and value sequence features.

[0081] Exemplarily, it can be understood that after being encoded by the aforementioned Pillar encoding method, P Radar Encoded as sequence features, so that the information of the Radar point is converted from a specific geometric position to a high-dimensional space. In order to improve the sparse characteristics of the Radar from a high-dimensional space, this embodiment proposes an autocorrelation point enhancement method based on an attention mechanism (such as a Transformer attention mechanism), that is, the point-to-point relationship in the Radar point is enhanced through an autocorrelation point enhancement module to obtain an improved Radar point feature.

[0082] Among them, the multi-layer linear computing unit MLP is used to calculate the target voxel sequence feature f radar High-dimensional encoding is performed to obtain the query sequence feature Q, key sequence feature K and value sequence feature V; specifically, the target voxel sequence feature f can be encoded based on MLP (16, 32, 64). radar The following calculation is performed so that the three sets of parameters Q, K and V are encoded into 64 dimensions, that is,

[0083]

[0084] Where W Q , W K , W V Represent different neural network learning parameters and W K =W V It should be noted that W Q , W K , W V The specific value of can be determined according to actual needs and is not limited here.

[0085] Step S40: Calculate the query sequence features and the key sequence features based on the attention mechanism to obtain an autocorrelation matrix; wherein the calculation formula of the autocorrelation matrix is:

[0086]

[0087] Where M auto represents the autocorrelation matrix, Q represents the query sequence characteristics, K T represents the transpose of the key sequence characteristics, C K Feature channel dimension representing key sequence features.

[0088] Exemplarily, in this embodiment, in order to obtain the autocorrelated Radar point enhancement result, the correlation between the query sequence feature Q and the key sequence feature K is calculated by using the attention mechanism; wherein, since the query sequence feature Q and the key sequence feature K parameters are both derived from the target voxel sequence feature f radar , so the autocorrelation matrix M can be obtained through the attention mechanism auto , the specific calculation is as follows:

[0089]

[0090] Among them, SelfAttention represents Softmax calculation, that is, it uses the Softmax activation function to calculate the correlation between the query sequence feature Q and the key sequence feature K:

[0091]

[0092] In the formula, C K The dimension of the feature channel representing the key sequence features, K T Represents the transpose of the key sequence feature K.

[0093] Therefore, based on equations (2) and (3), the autocorrelation matrix M can be obtained: auto The calculation formula is:

[0094]

[0095] Step S50: Encode the autocorrelation matrix and value sequence features through a multi-head self-attention mechanism, and perform a residual connection between the encoding result and the target voxel sequence feature to generate a millimeter-wave radar autocorrelation enhancement sequence feature. The expression of the millimeter-wave radar autocorrelation enhancement sequence feature is:

[0096]

[0097] In the formula, represents the millimeter-wave radar autocorrelation enhanced sequence feature, MultiHeadAttention represents the multi-head attention module, V represents the value sequence feature, head represents the number of heads in the multi-head attention module, and f radar Represents the target voxel sequence features.

[0098] For example, in this embodiment, in order to ensure the intrinsic connection of the original Radar point sequence features, a multi-head self-attention encoding method is used to encode the autocorrelation matrix M auto The sum value sequence feature V is encoded, and the encoding result and the target voxel sequence feature f are combined. radar Residual connection is performed to obtain enhanced Radar autocorrelation features, that is, to generate Radar autocorrelation enhanced sequence features, so as to effectively improve the sparsity of Radar data, thereby improving the utilization efficiency of Radar data. Specifically, the calculation formula of Radar autocorrelation enhanced sequence features is as follows:

[0099]

[0100] In the formula, Represents Radar autocorrelation enhanced sequence features, MultiHeadAttention represents the multi-head self-attention module, which requires the input of three parameters: the autocorrelation matrix M auto , value sequence feature V and the number of heads head corresponding to the multi-head attention module. It is worth noting that this embodiment can effectively stabilize the network training and ensure the convergence of the network by superimposing the original sequence features using residual connections. It can be understood that in view of the sparse characteristics of Radar data, this embodiment implements the autocorrelation point enhancement method through the aforementioned steps to enhance the perception ability of Radar features, and when performing data enhancement operations, it can be calculated with a lower computing load, thereby ensuring the efficiency of the network.

[0101] It can be seen that this embodiment generates a target voxel sequence feature by encoding the Radar point voxel data corresponding to the millimeter-wave radar Radar data, so as to convert the Radar point information from a specific geometric position to a high-dimensional space; then, multi-layer linear calculations are performed on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features to achieve high-dimensional encoding; then, in order to obtain the autocorrelation Radar point enhancement result, the correlation between the query sequence features and the key sequence features is calculated based on the attention mechanism to generate an autocorrelation matrix; finally, in order to ensure the intrinsic connection of the original Radar point sequence features, the autocorrelation matrix and the value sequence features are encoded using a multi-head self-attention encoding method, and the encoding result and the target voxel sequence features are residually connected to generate a Radar autocorrelation enhancement sequence feature, so as to effectively improve the sparsity of the Radar data, improve the utilization efficiency of the Radar data, and thus improve the efficiency and accuracy of data fusion.

[0102] Furthermore, in one embodiment, after the step of generating the millimeter wave radar autocorrelation enhancement sequence feature, the method further includes:

[0103] Determine the index coordinates based on the voxel coordinates corresponding to the millimeter wave radar autocorrelation enhancement sequence characteristics and the perception range of the bird's eye view;

[0104] The millimeter-wave radar autocorrelation enhancement sequence features are mapped to millimeter-wave radar bird's-eye view features based on the index coordinates.

[0105] For example, in this embodiment, after the enhancement of the correlation feature within the Radar point is completed, the Radar autocorrelation enhancement sequence feature can be obtained. But these features In a discrete state, in order to obtain the final feature distribution under the BEV view, this embodiment also designs an autocorrelation Radar feature allocation module to convert the discrete The features are assigned to the final BEV view space to obtain the final Radar BEV feature BEV R And BEV R The dimensions are H×W×C, where H represents BEV R length, W represents BEV R Width, C stands for BEV R The number of channels; H, W and BEV space have the same size, while C is the same as The dimensions are the same, for example Then C is 64.

[0106] It should be noted that the size of the BEV space can be defined according to the definition of the autonomous driving perception range commonly used in driving scene Radar; for example, assuming that the perception range of the driving scene is ±51.2m for the front and rear (y axis), ±51.2m for the left and right (x axis), and the height range (z axis) is -5m to +3m. Since this embodiment only cares about the perception range of the front and rear and left and right, the perception range of the point cloud can be determined according to the perception range corresponding to the front and rear axis and the left and right axis. (Right now ), and define the BEV perception grid size according to the voxel length h (for example, if h is 0.8m, the BEV perception grid size is also 0.8m), and then according to the perception range The perception range of the BEV view can be calculated by combining the BEV perception grid size Therefore, the size of the BEV space can be defined as 128×128, so both H and W are 128.

[0107] In this embodiment, when the autocorrelation Radar feature allocation module specifically implements feature allocation, it will determine the index coordinates of the Radar autocorrelation enhanced sequence feature according to the voxel coordinates corresponding to the voxel where the Radar autocorrelation enhanced sequence feature is located and the perception range of the BEV, that is, it will construct a conversion relationship between the voxel coordinates and the BEV perception grid coordinates, so as to determine the index coordinates of the Radar autocorrelation enhanced sequence feature through the conversion relationship; then, the mapping relationship between the Radar autocorrelation enhanced sequence feature and the BEV perception grid can be constructed through the index coordinates, that is, the Radar autocorrelation enhanced sequence feature allocation (for example) can be constructed based on the index coordinates. Figure 3 The rectangular block S1 in the figure) to the corresponding BEV sensing grid (e.g. Figure 3 A1) in the figure, and then we get the RadarBEV feature BEV R It should be noted that the voxel coordinates corresponding to each voxel will be predefined in advance. When performing feature allocation, the voxel coordinates can be directly obtained to participate in the index coordinate calculation.

[0108] Furthermore, in one embodiment, the calculation formula of the index coordinate is:

[0109] coords=r·vy+vx

[0110] In the formula, coords represents the index coordinates, and r represents the perceptual range of the bird's-eye view, that is, vy represents the voxel vertical axis coordinate, vx represents the voxel horizontal axis coordinate;

[0111] Exemplarily, in this embodiment, the voxel coordinates behind the Radar voxel are first defined as (vz, vy, vx) and the voxel size of each Radar is defined (for example, the voxel size is: voxel height d = 3-(-5) = 8m, voxel length h = 0.8m, voxel width w = 0.8m, which is consistent with the BEV perception grid size); it can be understood that each Radar point will be assigned to a voxel, so the Radar autocorrelation enhancement sequence feature obtained by encoding a voxel The index coordinates of can be calculated by the voxel coordinates vy, vx and the perception range r of the voxel, that is, each Radar autocorrelation enhanced sequence feature The index coordinates can be calculated by the following formula:

[0112] coords=r·vy+vx

[0113] Where coords represents the index coordinates. After allocating the Radar autocorrelation enhanced sequence features to the corresponding BEV perception grid based on the index coordinates, the Radar BEV feature BEV can be obtained by 2D convolution coding. R .

[0114] In this embodiment, after the autocorrelation Radar point feature allocation, the Radar autocorrelation enhanced sequence feature Assigned to the corresponding BEV view space, that is, by coordinate mapping Convert to RadarBEV FeaturesBEV R , for subsequent feature interaction. It can be seen that this embodiment improves the accuracy and range of perception through precise feature allocation and definition of BEV view space, so that the system can more accurately identify and locate the target.

[0115] Furthermore, in one embodiment, after the step of obtaining the millimeter wave radar bird's-eye view features, the method further includes:

[0116] Obtaining lidar bird's-eye view features corresponding to the lidar data and camera bird's-eye view features corresponding to the camera image data;

[0117] The millimeter-wave radar bird's-eye view features, the lidar bird's-eye view features and the camera bird's-eye view features are respectively transformed in feature dimension, and all the transformed bird's-eye view features are spliced ​​to obtain spliced ​​features;

[0118] Using different activation functions to perform feature activation operations on the concatenated features to obtain different learnable attention weights;

[0119] The learnable attention weights and their corresponding bird's-eye view features are weighted respectively to obtain the target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features and the target camera bird's-eye view features;

[0120] The target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features and the target camera bird's-eye view features are spliced ​​to generate fusion features.

[0121] For example, in this embodiment, the adaptive fusion of three modalities, namely, LiDAR, millimeter wave radar and camera data, will be realized. Figure 4 As shown, first obtain the RadarBEV feature BEV R , LiDAR BEV feature BEV corresponding to LiDAR data L and the camera BEV feature BEV corresponding to the camera image data I ; Among them, the LiDAR data can be voxelized and 3D sparse convolutional encoded in turn to obtain the LiDAR BEV feature BEV L At the same time, the camera image data is sequentially subjected to shallow feature extraction, deep feature extraction, image view conversion based on LSS (Lift-Splat-Shoot, a bottom-up BEV perception method) and 2D convolution encoding to obtain the camera BEV feature BEV R It should be noted that the LiDAR data and the camera image data are used to generate the LiDAR BEV feature BEV respectively. L and camera BEV features BEV I The method and principle are common knowledge in the art, so for the sake of brevity of description, they will not be described here in detail.

[0122] Secondly, for the BEV features from the three modalities, this embodiment designs a simple and efficient gated fusion module (i.e. Figure 5 The adaptive module AU shown in FIG. 4 is used to generate unified BEV features so that the model can adaptively fuse complementary modalities. For details, see Figure 5 As shown, for BEV R , BEV L and BEV I The feature dimension is transformed, and all the transformed BEV features are concatenated to obtain the concatenated features; then different activation functions Sigmoid are used to perform feature activation operations on the concatenated features to obtain three different sets of learnable attention weights (such as ω R ,ω L and ω I), it should be noted that the type and parameter setting of the Sigmoid function can be determined according to actual needs and are not limited here; then the corresponding BEV features are weighted by three sets of learnable attention weights, that is, the attention weighting method is used to weight ω R With BEV R ,ω L With BEV L And ω I With BEV I Multiply them respectively to get the new target RadarBEV feature BEV R ′、Target LiDAR BEV Features BEV L ′ and target camera BEV feature BEV I ′; then for BEV R ′、BEV L ′ and BEV I ′ can be concatenated to generate fusion features.

[0123] Finally, the fused features are sent to the detection head module to complete the final 3D object and other target detection tasks; it should be understood that the detection head module can use methods such as CenterPoint to achieve target detection, where the CenterPoint algorithm refers to a method for target detection and tracking based on the center point. It first converts the 3D point cloud data into a top-down feature map through a 3D encoder, and then uses the detection head to find the center point of the target and regress the target's 3D size, orientation, speed and other attributes. The specific implementation method and principle are common knowledge in the field and are not limited here for the sake of brevity of description.

[0124] Further, in one embodiment, the feature dimension conversion of the millimeter wave radar bird's eye view features, the laser radar bird's eye view features and the camera bird's eye view features respectively includes:

[0125] The millimeter-wave radar bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted millimeter-wave radar bird's-eye view features;

[0126] The lidar bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted lidar bird's-eye view features;

[0127] The camera bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted camera bird's-eye view features.

[0128] For example, see Figure 5As shown in the figure, the calculation of the adaptive module AU uses BEV features of three different modalities as input, and performs feature dimension conversion on the BEV features of the three different modalities, that is, the BEV features of the three different modalities are calculated in sequence through convolution, batch normalization and activation operations (that is, CBR represents the commonly used convolution Convolution+batch normalization BatchNormlization+activation function ReLU operation), and the converted Radar BEV features, the converted LiDAR BEV features and the converted camera BEV features are obtained respectively.

[0129] This embodiment optimizes the sparsity of millimeter-wave radar data through the autocorrelation point enhancement technology, utilizes the point-to-point relationship in the scene and significantly improves the feature expression ability of radar data by learning the point-to-point autocorrelation mechanism within the Radar point, thereby significantly improving the sparse disadvantage of the Radar BEV features obtained by the traditional method, reducing the dependence on the pre-selected box, thereby enhancing the accuracy of target detection and the overall robustness of the system; secondly, this embodiment effectively maps the enhanced radar point features to the BEV view space through the autocorrelation Radar point feature allocation module, ensuring the precise distribution of the features in space, improving the positioning accuracy of the perception system and the adaptability to complex environments; therefore, the adaptive fusion and autocorrelation enhancement can improve the adaptability of the model to environmental changes, thereby enhancing the robustness of the model; in addition, this embodiment adopts a full-modal adaptive fusion strategy, integrates BEV features from different sensors (such as cameras, lidars and millimeter-wave radars), optimizes the feature fusion process through the designed adaptive module, and realizes the efficient integration of multimodal data to improve the accuracy and robustness of detection, thereby improving the comprehensive performance and real-time performance of the detection system.

[0130] In a second aspect, an embodiment of the present application also provides a data processing device.

[0131] In one embodiment, the data processing device includes an autocorrelation point enhancement module, which is used to:

[0132] Obtaining millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data;

[0133] Encoding millimeter wave radar point voxel data to generate target voxel sequence features;

[0134] Performing multi-layer linear calculations on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features;

[0135] The query sequence features and key sequence features are calculated based on the attention mechanism to obtain the autocorrelation matrix;

[0136] The autocorrelation matrix and value sequence features are encoded through a multi-head self-attention mechanism, and the encoding results and the target voxel sequence features are residually connected to generate millimeter-wave radar autocorrelation enhanced sequence features.

[0137] Furthermore, in one embodiment, the apparatus further includes an autocorrelation Radar feature allocation module, which is used to:

[0138] Determine the index coordinates based on the voxel coordinates corresponding to the millimeter wave radar autocorrelation enhancement sequence characteristics and the perception range of the bird's eye view;

[0139] The millimeter-wave radar autocorrelation enhancement sequence features are mapped to millimeter-wave radar bird's-eye view features based on the index coordinates.

[0140] Furthermore, in one embodiment, the calculation formula of the index coordinate is:

[0141] coords=r·vy+vx

[0142] Where coords represents the index coordinates, r represents the perceptual range of the bird's-eye view, vy represents the voxel vertical axis coordinate, and vx represents the voxel horizontal axis coordinate.

[0143] Furthermore, in one embodiment, the device further comprises an adaptability module, which is used to:

[0144] Obtaining lidar bird's-eye view features corresponding to the lidar data and camera bird's-eye view features corresponding to the camera image data;

[0145] The millimeter-wave radar bird's-eye view features, the lidar bird's-eye view features and the camera bird's-eye view features are respectively transformed in feature dimension, and all the transformed bird's-eye view features are spliced ​​to obtain spliced ​​features;

[0146] Using different activation functions to perform feature activation operations on the concatenated features to obtain different learnable attention weights;

[0147] The learnable attention weights and their corresponding bird's-eye view features are weighted respectively to obtain the target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features and the target camera bird's-eye view features;

[0148] The target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features and the target camera bird's-eye view features are spliced ​​to generate fusion features.

[0149] Furthermore, in one embodiment, the adaptability module is specifically used for:

[0150] The millimeter-wave radar bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted millimeter-wave radar bird's-eye view features;

[0151] The lidar bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted lidar bird's-eye view features;

[0152] The camera bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted camera bird's-eye view features.

[0153] Furthermore, in one embodiment, the calculation formula of the autocorrelation matrix is:

[0154]

[0155] Where M auto represents the autocorrelation matrix, Q represents the query sequence characteristics, K T represents the transpose of the key sequence characteristics, C K Feature channel dimension representing key sequence features.

[0156] Furthermore, in one embodiment, the expression of the millimeter wave radar autocorrelation enhancement sequence feature is:

[0157]

[0158] In the formula, represents the millimeter-wave radar autocorrelation enhanced sequence feature, MultiHeadAttention represents the multi-head attention module, V represents the value sequence feature, head represents the number of heads in the multi-head attention module, and f radar Represents the target voxel sequence features.

[0159] Among them, the functional implementation of each module in the above-mentioned data processing device corresponds to each step in the above-mentioned data processing method embodiment, and its functions and implementation processes are no longer repeated here one by one.

[0160] In a third aspect, an embodiment of the present application provides a data processing device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0161] Reference Figure 6 , Figure 6 Schematic diagram of the hardware structure of the data processing device involved in the embodiment of the present application. In the embodiment of the present application, the data processing device may include a processor, a memory, a communication interface and a communication bus.

[0162] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0163] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces, which are used to interconnect devices within a data processing device, as well as interfaces used to interconnect a data processing device with other devices (such as other computing devices or user devices). Physical interfaces can be Ethernet interfaces, optical fiber interfaces, ATM interfaces, etc.; user devices can be display screens (Display), keyboards (Keyboard), etc.

[0164] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0165] The processor may be a general-purpose processor, and the general-purpose processor may call the data processing program stored in the memory and execute the data processing method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the data processing program is called may refer to the various embodiments of the data processing method of the present application, and will not be repeated here.

[0166] Those skilled in the art will understand that Figure 6 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0167] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0168] The readable storage medium of the present application stores a data processing program, wherein when the data processing program is executed by a processor, the steps of the data processing method as described above are implemented.

[0169] Among them, the method implemented when the data processing program is executed can refer to the various embodiments of the data processing method of the present application, and will not be repeated here.

[0170] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0171] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0172] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0173] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0174] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0175] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0176] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Obtaining millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data; Encoding millimeter wave radar point voxel data to generate target voxel sequence features; Performing multi-layer linear calculations on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features; The query sequence features and key sequence features are calculated based on the attention mechanism to obtain the autocorrelation matrix; The autocorrelation matrix and value sequence features are encoded through a multi-head self-attention mechanism, and the encoding results and the target voxel sequence features are residually connected to generate millimeter-wave radar autocorrelation enhanced sequence features.

2. The data processing method according to claim 1, characterized in that: After the step of generating the millimeter wave radar autocorrelation enhancement sequence feature, the method further includes: Determine the index coordinates based on the voxel coordinates corresponding to the millimeter wave radar autocorrelation enhancement sequence characteristics and the perception range of the bird's eye view; The millimeter-wave radar autocorrelation enhancement sequence features are mapped to millimeter-wave radar bird's-eye view features based on the index coordinates.

3. The data processing method according to claim 2, characterized in that: The calculation formula of the index coordinates is: coords=r·vy+vx Where coords represents the index coordinates, r represents the perceptual range of the bird's-eye view, vy represents the voxel vertical axis coordinate, and vx represents the voxel horizontal axis coordinate.

4. The data processing method according to claim 2 or 3, characterized in that: After the step of obtaining the millimeter wave radar bird's-eye view features, the method further includes: Obtaining lidar bird's-eye view features corresponding to the lidar data and camera bird's-eye view features corresponding to the camera image data; The millimeter-wave radar bird's-eye view features, the lidar bird's-eye view features, and the camera bird's-eye view features are respectively transformed in feature dimension, and all the transformed bird's-eye view features are spliced ​​to obtain spliced ​​features; Using different activation functions to perform feature activation operations on the concatenated features to obtain different learnable attention weights; The learnable attention weights and their corresponding bird's-eye view features are weighted respectively to obtain the target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features and the target camera bird's-eye view features; The target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features and the target camera bird's-eye view features are spliced ​​to generate fusion features.

5. The data processing method according to claim 4, characterized in that: The feature dimension conversion of the millimeter wave radar bird's eye view features, the laser radar bird's eye view features and the camera bird's eye view features respectively includes: The millimeter-wave radar bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted millimeter-wave radar bird's-eye view features; The lidar bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted lidar bird's-eye view features; The camera bird's-eye view features are sequentially convolved, batch normalized, and activated to obtain the converted camera bird's-eye view features.

6. The data processing method according to claim 1, characterized in that: The calculation formula of the autocorrelation matrix is: Where M auto represents the autocorrelation matrix, Q represents the query sequence characteristics, K T represents the transpose of the key sequence characteristics, C K Feature channel dimension representing key sequence features.

7. The data processing method according to claim 1, characterized in that: The expression of the millimeter wave radar autocorrelation enhancement sequence characteristics is: In the formula, represents the millimeter-wave radar autocorrelation enhanced sequence feature, MultiHeadAttention represents the multi-head attention module, V represents the value sequence feature, head represents the number of heads in the multi-head attention module, and f radar Represents the target voxel sequence features.

8. A data processing device, characterized in that: The data processing device comprises an autocorrelation point enhancement module, which is used for: Obtaining millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data; Encoding millimeter wave radar point voxel data to generate target voxel sequence features; Performing multi-layer linear calculations on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features; The query sequence features and key sequence features are calculated based on the attention mechanism to obtain the autocorrelation matrix; The autocorrelation matrix and value sequence features are encoded through a multi-head self-attention mechanism, and the encoding results and the target voxel sequence features are residually connected to generate millimeter-wave radar autocorrelation enhanced sequence features.

9. A data processing device, characterized in that: The data processing device comprises a processor, a memory, and a data processing program stored in the memory and executable by the processor, wherein when the data processing program is executed by the processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data processing program, wherein when the data processing program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images