Automatic driving perception method and device, electronic equipment and readable storage medium

Abstract

The disclosure provides an automatic driving perception method and device, electronic equipment and readable storage medium, relates to the technical field of artificial intelligence, specifically to the technical field of computer vision, virtual reality, deep learning, large model and the like, and can be applied to scenarios such as automatic driving. The specific implementation scheme is: obtaining an aerial view feature, the aerial view feature is extracted based on environment perception data of at least one mode, and the environment perception data is collected for a target space where a target object is located; the target space is divided into a plurality of voxels, and an initial voxel feature corresponding to each voxel is constructed based on the position of each voxel; target voxel features are extracted based on the initial voxel features and the aerial view features; and perception processing is performed based on the target voxel features to obtain a perception result of the target space, wherein the perception processing includes occupancy prediction and at least one related perception processing related to the occupancy prediction.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, virtual reality, deep learning, large model, etc., and can be applied to scenarios such as automatic driving. Specifically, the present disclosure relates to an automatic driving perception method and device, an electronic device, and a readable storage medium. BACKGROUND

[0002] Automatic driving perception is to perceive environmental information in the process of automatic driving to provide support for automatic driving decision-making.

[0003] Bird's Eye View (BEV) is a perspective from which an object or scene is viewed from above, as if a bird were looking down at the ground. A bird's eye view feature is a feature of a bird's eye view. In the field of automatic driving, a bird's eye view feature can be used to process some automatic driving perception tasks.

[0004] When a bird's eye view feature is used to process automatic driving perception tasks, how to improve the accuracy of automatic driving perception becomes an important technical problem. SUMMARY

[0005] The present disclosure provides an automatic driving perception method, device, electronic device, and readable storage medium to solve at least one of the above-mentioned defects.

[0006] According to a first aspect of the present disclosure, an automatic driving perception method is provided, which comprises:

[0007] Obtaining a bird's eye view feature, the bird's eye view feature being extracted based on environmental perception data of at least one modality, the environmental perception data being collected for a target space in which a target object is located;

[0008] Dividing the target space into a plurality of voxels, and constructing an initial voxel feature corresponding to each voxel based on the position of each voxel;

[0009] Extracting a target voxel feature based on the initial voxel feature and the bird's eye view feature;

[0010] Performing perception processing based on the target voxel feature to obtain a perception result of the target space, the perception processing including occupancy prediction and at least one related perception processing related to occupancy prediction.

[0011] According to a second aspect of the present disclosure, an automatic driving perception device is provided, which comprises:

[0012] A bird's eye view feature acquisition module is configured to obtain a bird's eye view feature, the bird's eye view feature being extracted based on environmental perception data of at least one modality, the environmental perception data being collected for a target space in which a target object is located;

[0013] an initial voxel feature construction module, configured to divide the target space into a plurality of voxels, and construct initial voxel features corresponding to the voxels respectively based on positions of the voxels;

[0014] a target voxel feature extraction module, configured to extract target voxel features based on the initial voxel features and the aerial view features;

[0015] a perception processing module, configured to perform perception processing based on the target voxel features to obtain a perception result of the target space, the perception processing including occupancy prediction and at least one related perception processing related to the occupancy prediction.

[0016] According to a third aspect of the present disclosure, an electronic device is provided, which comprises:

[0017] at least one processor; and

[0018] a memory in communication with the at least one processor; wherein

[0019] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the automatic driving perception method.

[0020] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the automatic driving perception method.

[0021] According to a fifth aspect of the present disclosure, a computer program product is provided, which comprises a computer program, and the computer program, when executed by a processor, implements the automatic driving perception method.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0024] Figure 1 is a flowchart of an automatic driving perception method provided by an embodiment of the present disclosure;

[0025] Figure 2 is a structural schematic diagram of an automatic driving perception network provided by an embodiment of the present disclosure.

[0026] Figure 3is a structural schematic diagram of an automatic driving perception device provided by an embodiment of the present disclosure.

[0027] Figure 4 is a block diagram of an electronic device for implementing an automatic driving perception method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the present disclosure to assist in understanding them. These should be considered as merely exemplary. Thus, those of ordinary skill in the art will recognize various changes and modifications of the embodiments described herein, which do not depart from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0029] Bird's Eye View (BEV) is a perspective from which an object or scene is viewed from above, as if a bird were looking down at the ground. A bird's eye view feature is a feature of a bird's eye view.

[0030] Occupancy Prediction is an automatic driving perception task, which refers to predicting the state of each voxel in a scene. A voxel can generally be identified as one of three states: free, occupied, or unknown.

[0031] Bird's eye view features can be used to process some automatic driving perception tasks, such as occupancy prediction, target detection, etc. When bird's eye view features are used to process automatic driving perception tasks, how to improve the accuracy of automatic driving perception becomes an important technical problem.

[0032] In related technologies, when bird's eye view features are used to process occupancy prediction tasks and other automatic driving perception tasks, the processing of these automatic driving perception tasks is generally independent, that is, a separate backbone network is used to extract bird's eye view features from environmental perception data, and then separate detection head networks are used to process perception tasks and output perception results. In this way, the network models of different automatic driving perception tasks do not share parameters at all.

[0033] In addition, multiple automatic driving perception tasks can also share the same backbone network, which collects bird's eye view features from environmental perception data, and then separate detection head networks are used to process perception tasks and output perception results. In this way, the network models of different automatic driving perception tasks only share a small number of parameters.

[0034] The different automatic driving perception task network models do not share parameters at all or share only a small number of parameters, which causes the different automatic driving perception task network models to independently process a large number of parameters, resulting in large data calculation and high network model latency.

[0035] In addition, the automatic driving perception tasks are all used for perception of environmental information, and there is a certain correlation between different automatic driving perception tasks. If the correlation between different automatic driving perception tasks can be effectively utilized, the accuracy of automatic driving perception can be effectively improved. However, the above method does not effectively utilize the correlation between different automatic driving perception tasks.

[0036] The automatic driving perception method, device, electronic device, and readable storage medium provided by the embodiments of the present disclosure aim to solve at least one of the above technical problems of the prior art.

[0037] Figure 1 A flowchart of an automatic driving perception method provided by an embodiment of the present disclosure is shown, as shown in Figure 1 The method can mainly include the following steps.

[0038] Step S110: Obtain an aerial view feature, the aerial view feature being extracted based on at least one modal of environmental perception data, the environmental perception data being collected for a target space where a target object is located;

[0039] Step S120: Divide the target space into a plurality of voxels, and respectively construct initial voxel features corresponding to each voxel based on the position of each voxel;

[0040] Step S130: Extract a target voxel feature based on the initial voxel features and the aerial view feature;

[0041] Step S140: Perform perception processing based on the target voxel feature to obtain a perception result of the target space, the perception processing including occupancy prediction and at least one related perception processing related to the occupancy prediction.

[0042] The target object can include, but is not limited to, an autonomous vehicle, an autonomous robot, etc. The target space is a space where the target object is located, and as an example, the target space can have a center point of the target object as a coordinate origin.

[0043] The target object can collect environmental perception data of the target space through an environmental perception device.

[0044] As an example, the environmental perception device can be a sensor, and the environmental perception data can be data collected by the sensor.

[0045] The environmental perception data can reflect environmental information in the target space. The bird's eye view feature can be extracted based on the environmental perception data.

[0046] In the embodiments of the present disclosure, the target space can be divided into a plurality of voxels. The number and size of the voxels can be determined according to actual requirements. According to the positions of the voxels, initial voxel features corresponding to the voxels can be constructed. The initial voxel features can reflect the mutual relationship between the voxels in the target space.

[0047] In the embodiments of the present disclosure, the target voxel feature can be extracted based on the initial voxel feature and the bird's eye view feature. The target voxel feature is a feature expression of the bird's eye view feature in the target space.

[0048] After obtaining the target voxel feature, the target voxel feature can be used for occupancy prediction and related perception processing.

[0049] The related perception processing is a perception processing task related to the occupancy prediction. The related perception processing can include one or more. It can be understood that the related perception processing involved in the present case is 3D perception based on the target voxel feature.

[0050] In the embodiments of the present disclosure, the occupancy prediction and the related perception processing can share the target voxel feature, and the target voxel feature is used for perception processing respectively. Therefore, more parameters in the network model corresponding to the occupancy prediction task and the related perception processing task are shared, and the data calculation amount when performing multiple autonomous driving perception tasks is reduced, and the time delay of the network model is effectively reduced.

[0051] The occupancy prediction task is an accurate description of the environmental information in the target space. The related perception processing task is also used for perception of the environmental information in the target space. The two tasks have certain correlation. By jointly processing the occupancy prediction task and the related perception processing task, the correlation between the occupancy prediction and the related perception processing can be effectively utilized, and the accuracy of the occupancy prediction result and the related perception processing result can be effectively improved, that is, the accuracy of the autonomous driving perception is improved.

[0052] The method provided by the embodiments of the present disclosure can obtain a bird's eye view feature, the bird's eye view feature is extracted based on environment perception data of at least one mode, and the environment perception data is collected for a target space in which a target object is located; the target space is divided into a plurality of voxels, and initial voxel features corresponding to the voxels are respectively constructed based on positions of the voxels; target voxel features are extracted based on the initial voxel features and the bird's eye view feature; and perception processing is performed based on the target voxel features to obtain a perception result of the target space, the perception processing including occupancy prediction and at least one related perception processing. Based on the scheme, occupancy prediction tasks and related perception processing tasks can be jointly processed, the data calculation amount of autonomous driving perception processing is effectively reduced, and the accuracy of autonomous driving perception is improved.

[0053] In an optional embodiment of the present disclosure, the target voxel features are extracted based on the initial voxel features and the bird's eye view feature, including:

[0054] The initial voxel features are subjected to global self-attention processing to obtain self-attention voxel features;

[0055] The target voxel features are obtained by performing deformable cross-attention processing based on the self-attention voxel features and the bird's eye view feature.

[0056] In the embodiments of the present disclosure, the initial voxel features can reflect the mutual relationship between the voxels in the target space. By performing global self-attention processing on the initial voxel features, the information of all the voxels in the target space can be effectively learned to generate global voxel feature representation.

[0057] As an example, the initial voxel features can be subjected to linear transformation to obtain key vectors, value vectors and query vectors of the initial voxel features, and the key vectors, the value vectors and the query vectors are subjected to feature fusion based on a self-attention mechanism to obtain the self-attention voxel features.

[0058] The deformable cross-attention processing based on the self-attention voxel features and the bird's eye view feature is cross-attention processing of the bird's eye view feature and positions adjacent to the voxels, so as to improve the processing efficiency of the target voxel features and reduce the calculation amount.

[0059] In an optional embodiment of the present disclosure, the target voxel features are obtained by performing deformable cross-attention processing based on the self-attention voxel features and the bird's eye view feature, including:

[0060] The bird's eye view feature is subjected to local feature extraction based on position information of the voxel corresponding to the self-attention voxel feature to obtain local bird's eye view features corresponding to the self-attention voxel features;

[0061] cross-attention processing of each attention voxel feature and the corresponding local aerial view feature to obtain cross-attention voxel features;

[0062] The target voxel feature is determined based on the cross-attention voxel feature.

[0063] In the embodiments of the present disclosure, the position information of the voxel can be represented by the coordinates of the center point of the voxel. By taking the center point of the voxel as the reference point, the features near the position of each voxel can be extracted from the aerial view features, i.e., local aerial view features. Therefore, cross-attention processing can be performed based on the self-attention voxel feature and the corresponding local aerial view feature to obtain cross-attention voxel features.

[0064] As an example, a linear transformation can be performed on the local aerial view feature to obtain a key vector and a value vector of the local aerial view feature, the self-attention voxel feature corresponding to the position of the local aerial view feature is taken as a query vector, and feature fusion is performed on the key vector, the value vector and the query vector based on the cross-attention mechanism to obtain the cross-attention voxel feature.

[0065] In the embodiments of the present disclosure, the cross-attention voxel feature can be taken as the target voxel feature, or the cross-attention voxel feature can be further transformed to obtain the target voxel feature.

[0066] In an optional embodiment of the present disclosure, the target voxel feature is determined based on the cross-attention voxel feature, comprising:

[0067] The cross-attention voxel feature is taken as the initial voxel feature, and global self-attention processing is performed on the initial voxel feature to obtain self-attention voxel features;

[0068] Based on the position information of the voxel corresponding to the self-attention voxel feature, local feature extraction is performed on the aerial view features to obtain local aerial view features corresponding to each attention voxel feature;

[0069] Cross-attention processing is performed on each attention voxel feature and the corresponding local aerial view feature to obtain cross-attention voxel features;

[0070] The cross-attention voxel feature satisfying the preset condition is determined as the target voxel feature.

[0071] In the embodiments of the present disclosure, the cross-attention voxel feature can be taken as the initial voxel feature, and the global self-attention processing is performed again on the initial voxel feature to obtain self-attention voxel features, and deformable cross-attention processing is performed on the self-attention voxel features and the aerial view features to obtain cross-attention voxel features. If the cross-attention voxel features obtained at this time satisfy the preset condition, the cross-attention voxel features can be determined as the target voxel features.

[0072] The preset condition can be set as that the execution number of the following specified steps reaches a preset number. The specified steps are: taking the cross attention voxel feature as the initial voxel feature, performing global self-attention processing on the initial voxel feature to obtain a self-attention voxel feature, performing local feature extraction on the bird's eye view feature based on the position information of the voxel corresponding to the self-attention voxel feature to obtain a local bird's eye view feature corresponding to each self-attention voxel feature; and performing cross attention processing on each self-attention voxel feature and the corresponding local bird's eye view feature to obtain the cross attention voxel feature.

[0073] The preset number in the preset condition can be set according to actual needs, so as to ensure that the obtained target voxel feature has a good learning effect on the bird's eye view feature, for example, the preset number can be 2 times. The preset condition can also be set in other forms, which is not limited in the embodiments of the present disclosure.

[0074] In an optional embodiment of the present disclosure, the initial voxel feature corresponding to each voxel is constructed based on the position of each voxel, including:

[0075] Position encoding is performed based on the position of each voxel to obtain a position feature;

[0076] The position feature is subjected to multilayer perception processing to obtain a multilayer perception result;

[0077] The multilayer perception result is subjected to at least one linear transformation to obtain the initial voxel feature corresponding to each voxel.

[0078] In the embodiments of the present disclosure, the position of each voxel can be encoded to obtain a position feature, for example, the center point coordinates of each voxel can be subjected to sinusoidal position encoding (Sinusoidal Position Encoding).

[0079] The position feature can be subjected to multilayer perception processing (Multilayer Perceptron, MLP) to mine the correlation between the position features corresponding to each voxel.

[0080] The multilayer perception result can be subjected to at least one linear transformation to obtain the initial voxel feature corresponding to each voxel.

[0081] In an optional embodiment of the present disclosure, the target voxel feature is subjected to perception processing to obtain a perception result of the target space, including:

[0082] The target voxel feature is subjected to deconvolution processing to obtain a deconvoluted voxel feature;

[0083] Occupancy prediction is performed based on the deconvoluted voxel feature to obtain an occupancy prediction result.

[0084] In the embodiments of the present disclosure, the target voxel feature contains relatively dense features, and the target voxel feature can be deconvoluted to obtain deconvoluted voxel features containing relatively sparse features. Then, the deconvoluted voxel features are used for occupancy prediction to obtain an occupancy prediction result.

[0085] In an optional embodiment of the present disclosure, the related perception processing includes at least one of the following:

[0086] Target detection;

[0087] Lane line segmentation;

[0088] Driveable region segmentation.

[0089] The target detection can be understood as a rough representation of the target space, the lane line segmentation can be understood as a perception of the lane line in the target space, and the driveable region segmentation can be understood as a perception of the driveable region in the target space. The target detection, the lane line segmentation, and the driveable region segmentation are all related to the occupancy prediction.

[0090] In an optional embodiment of the present disclosure, in response to the related perception processing including target detection, the perception processing is performed based on the target voxel feature to obtain a perception result of the target space, including:

[0091] The height information of the target voxel feature is compressed to obtain compressed bird's eye view features;

[0092] The target detection is performed based on the compressed bird's eye view features to obtain a target detection result.

[0093] In the embodiments of the present disclosure, when the target voxel feature is used for target detection, the height information of the target voxel feature can be compressed, that is, the Z-axis of the target voxel feature is flattened to obtain compressed bird's eye view features in a bird's eye view space, so that the target detection is performed based on the compressed bird's eye view features.

[0094] In an optional embodiment of the present disclosure, the environmental perception data includes at least one of the following:

[0095] Image data;

[0096] Point cloud data.

[0097] In the embodiments of the present disclosure, the environmental data can include at least one of image data collected by a surround-view camera and point cloud data collected by a laser radar.

[0098] In the embodiments of the present disclosure, the bird's eye view features can be extracted based on the image data collected by the surround-view camera, or the bird's eye view features can be extracted based on the point cloud data collected by the laser radar, or the two kinds of environmental perception data can be fused to extract the bird's eye view features.

[0099] It can be understood that the environment perception data can also include other data that can be used to extract the bird's eye view features, which are not limited in the present disclosure.

[0100] In an optional embodiment of the present disclosure, in response to the environment perception data being image data, the bird's eye view features are acquired, including:

[0101] The bird's eye view features are extracted from the image data based on a preset bird's eye view feature extraction model.

[0102] In the embodiments of the present disclosure, the bird's eye view feature extraction model can be used to extract the bird's eye view features from the image data. The bird's eye view feature extraction model can be obtained by pre-training, or a known model can be used.

[0103] As an example, the bird's eye view feature extraction model can select the encoder of the known bird's eye view Former. The image data is input into the bird's eye view Former (i.e. BEVFormer), and the bird's eye view features processed by the encoder are acquired.

[0104] As an example, the embodiments of the present disclosure can also provide an autonomous driving perception network, which includes a backbone network, a voxel feature extraction network, and a detection head network corresponding to different autonomous driving perception tasks.

[0105] The backbone network is used to extract the bird's eye view features from the environment perception data. For example, when the environment perception data is image data, the backbone network can be the encoder of the BEVFormer.

[0106] The bird's-eye view features output by the backbone network can be input into the voxel feature extraction network. The voxel feature extraction network includes a position encoding network, a global self-attention sub-network, and a deformable cross-attention sub-network. The position encoding network can include a position encoding layer, a multilayer perceptron layer, and at least one linear transformation layer. The center point coordinates of the voxels are input into the position encoding layer to obtain position features. These position features are then input into the multilayer perceptron layer to obtain multilayer perceptron results. The multilayer perceptron results are then input into the linear transformation layer for at least one linear transformation to obtain initial voxel features. The initial voxel features are input into the global self-attention sub-network to obtain self-attention voxel features. The self-attention voxel features, bird's-eye view features, and position information of each voxel are input into the deformable cross-attention sub-network to obtain cross-attention voxel features. The global self-attention sub-network and the deformable cross-attention sub-network can be configured with two layers. The second-layer global self-attention sub-network can be referred to as the second global self-attention network, and the second-layer deformable cross-attention sub-network can be referred to as the second deformable cross-attention sub-network. The cross-attention voxel features are input into the second global self-attention network to obtain the second self-attention voxel features. The second self-attention voxel features, the bird's-eye view features, and the position information of each voxel are input into the second deformable cross-attention sub-network to obtain the target voxel features output by the second deformable cross-attention sub-network.

[0107] In this example, the relevant perception processing task is target detection, that is, the detection head network in the autonomous driving perception network in this example includes the target detection head network and the occupancy detection head network.

[0108] In this example, the loss function of the autonomous driving perception network can be constructed based on the loss function of the target detection head network and the loss function of the occupancy detection head network, as shown in Equation (I):

[0109] L all =L occ +L det (Formula 1)

[0110] Among them, L all Let L represent the loss function of the autonomous driving perception network. occ L represents the loss function of the occupancy detection head network. det Let represent the loss function of the object detection head network. The loss function of the object detection head network can be the standard binary cross-entropy loss, while the loss function of the object detection head network is the sum of the classification loss and the regression loss.

[0111] During the prediction phase, inputting image data from six perspectives collected by surround-view cameras into the autonomous driving perception network yields occupancy perception output and target detection bounding boxes.

[0112] As an example, the training method of the autonomous driving perception network provided by the embodiments of the present disclosure can include:

[0113] obtaining sample bird's eye view features, the sample bird's eye view features being extracted based on sample environment perception data of at least one modality, the sample environment perception data being collected for a sample target space in which a sample target object is located;

[0114] dividing the sample target space into a plurality of voxels, and constructing sample initial voxel features corresponding to each voxel based on the position of each voxel;

[0115] extracting sample target voxel features based on the sample initial voxel features and the sample bird's eye view features;

[0116] performing perception processing based on the sample target voxel features to obtain a sample perception result of the target space, the perception processing including occupancy prediction and at least one related perception processing related to the occupancy prediction;

[0117] adjusting parameters of the autonomous driving perception network according to the difference between the sample perception result and the real environment information in the sample target space, to obtain a trained autonomous driving perception network.

[0118] As an example, when using the autonomous driving perception network trained based on the above training method to perform autonomous driving perception processing, the environment perception data can be input into the trained autonomous driving perception network to obtain a perception processing result.

[0119] As an example, Figure 2 is a structural schematic diagram of an autonomous driving perception network provided by the embodiments of the present disclosure.

[0120] The autonomous driving perception network provided in this example is a joint learning framework for target detection and occupancy prediction, the input of the autonomous driving perception network is image data of 6 perspectives collected by a surround-view camera, and the output is a 3D detection frame and an occupancy state of each voxel in a 3D space. The autonomous driving perception network in this example can provide target voxel features for both target detection and occupancy prediction, so as to promote each other and reduce the time delay of the network.

[0121] As shown in Figure 2 , the processing flow based on the autonomous driving perception network is as follows:

[0122] First, the image data of 6 perspectives collected by the surround-view camera is processed, the image features are extracted by using an image backbone network, and the image features are processed by using an encoder in the bird's eye view Former to obtain bird's eye view features. The bird's eye view features 210 (Fused BEV features) are fused, i.e., the bird's eye view features extracted based on the image data.

[0123] The entire 3D space is divided into 2*25*25 voxels with the host vehicle as the origin, the center point of each voxel is taken as a fixed reference point 220, the center point coordinates of each voxel are subjected to sinusoidal position coding 231 to obtain position features, and the position features are subjected to linear transformation 232 to obtain initial voxel features, i.e. Figure 2 Q in the formula. In this example, two linear transformations can be performed.

[0124] After obtaining the initial voxel features Q, the initial voxel features Q are sent into the deformable attention module 240 together with the fused bird's eye view features 210 for decoding to obtain updated voxel features 250, i.e. the aforementioned target voxel features. The initial voxel features Q can be subjected to global self-attention 242 processing and deformable cross-attention 241 processing in the deformable attention module.

[0125] The detection head network in this example has two, one is the occupancy detection head network 270, which is responsible for predicting whether each voxel in the space is occupied, and outputs 1 if occupied and 0 if not occupied; the other is the target detection head network 270, which is responsible for outputting the center point, size and category information of the detection box. The target voxel features are shared by the two prediction heads.

[0126] In the occupancy detection head network 270, the target voxel features 250 are subjected to 3D deconvolution upsampling 271 with a stride of 2 to obtain an occupancy feature 272 of 16*200*200, and the occupancy feature 272 is input into an occupancy prediction multi-layer perceptron 273 to obtain an occupancy prediction result.

[0127] In the target detection head network 260, the target voxel features 250 are subjected to Z-axis pooling 261 processing, i.e. flattening in the Z-axis direction into the BEV space to obtain target query features 262, taking the x and y coordinates of the voxel in the BEV space as the reference point, and inputting the target query features into a target detection multi-layer perceptron 263 to predict the classification and regression information of the detection box to obtain the final target detection result.

[0128] Based on the same principle as shown in Figure 1 Figure 3 shows a structure schematic diagram of an automatic driving perception device provided by an embodiment of the present disclosure, as shown in Figure 3 The automatic driving perception device 30 can include:

[0129] The bird's eye view feature acquisition module 310 is configured to acquire bird's eye view features, the bird's eye view features being extracted based on at least one kind of modal environment perception data, the environment perception data being collected for a target space in which a target object is located.​

[0130] The initial voxel feature construction module 320 is configured to divide the target space into a plurality of voxels, and construct initial voxel features corresponding to the voxels respectively based on positions of the voxels.

[0131] The target voxel feature extraction module 330 is configured to extract target voxel features based on the initial voxel features and the aerial view features.

[0132] The perception processing module 340 is configured to perform perception processing based on the target voxel features to obtain a perception result of the target space, the perception processing including occupancy prediction and at least one related perception processing related to the occupancy prediction.

[0133] The device provided by the embodiments of the present disclosure is configured to obtain aerial view features, the aerial view features being extracted based on environment perception data of at least one modality, the environment perception data being collected for a target space in which a target object is located; divide the target space into a plurality of voxels, and construct initial voxel features corresponding to the voxels respectively based on positions of the voxels; extract target voxel features based on the initial voxel features and the aerial view features; and perform perception processing based on the target voxel features to obtain a perception result of the target space, the perception processing including occupancy prediction and at least one related perception processing related to the occupancy prediction. Based on the present solution, occupancy prediction tasks and related perception processing tasks can be jointly processed, thereby effectively reducing the data calculation amount of autonomous driving perception processing and improving the accuracy of autonomous driving perception.

[0134] Optionally, the target voxel feature extraction module is specifically configured to:

[0135] perform global self-attention processing on the initial voxel features to obtain self-attention voxel features;

[0136] perform deformable cross-attention processing on the self-attention voxel features and the aerial view features to obtain the target voxel features.

[0137] Optionally, when the target voxel feature extraction module performs deformable cross-attention processing on the self-attention voxel features and the aerial view features to obtain the target voxel features, the target voxel feature extraction module is specifically configured to:

[0138] perform local feature extraction on the aerial view features based on position information of voxels corresponding to the self-attention voxel features to obtain local aerial view features corresponding to the self-attention voxel features;

[0139] perform cross-attention processing on the self-attention voxel features and the corresponding local aerial view features respectively to obtain cross-attention voxel features;

[0140] determine the target voxel features based on the cross-attention voxel features.

[0141] Optionally, in the step of determining the target voxel feature based on the cross-attention voxel feature, the target voxel feature is determined by:

[0142] performing global self-attention processing on the initial voxel feature based on the cross-attention voxel feature to obtain a self-attention voxel feature;

[0143] performing local feature extraction on the bird's eye view feature based on position information of a voxel corresponding to the self-attention voxel feature to obtain a local bird's eye view feature corresponding to the self-attention voxel feature;

[0144] performing cross-attention processing on each self-attention voxel feature and a corresponding local bird's eye view feature to obtain a cross-attention voxel feature;

[0145] determining the cross-attention voxel feature that meets a preset condition as the target voxel feature.

[0146] Optionally, in the step of constructing the initial voxel feature corresponding to each voxel based on a position of each voxel, the initial voxel feature is constructed by:

[0147] performing position encoding based on the position of each voxel to obtain a position feature;

[0148] performing multi-layer perception processing on the position feature to obtain a multi-layer perception result;

[0149] performing at least one linear transformation on the multi-layer perception result to obtain the initial voxel feature corresponding to each voxel.

[0150] Optionally, the perception processing module is specifically configured to:

[0151] performing deconvolution processing on the target voxel feature to obtain a deconvoluted voxel feature;

[0152] performing occupancy prediction based on the deconvoluted voxel feature to obtain an occupancy prediction result.

[0153] Optionally, the related perception processing includes at least one of:

[0154] target detection;

[0155] lane line segmentation;

[0156] drivable area segmentation.

[0157] Optionally, in response to the related perception processing including target detection, the perception processing module is specifically configured to:

[0158] compressing height information of the target voxel feature to obtain a compressed bird's eye view feature;

[0159] Target detection is performed based on the features of the compressed bird's-eye view, and the target detection results are obtained.

[0160] Optionally, the environmental perception data includes at least one of the following:

[0161] Image data;

[0162] Point cloud data.

[0163] Optionally, in response to the environmental perception data being image data, the bird's-eye view feature acquisition module is specifically used for:

[0164] Based on a pre-defined bird's-eye view feature extraction model, bird's-eye view features are extracted from image data.

[0165] It is understood that the above-described modules of the autonomous driving perception device in the embodiments of this disclosure have the ability to implement... Figure 1 The embodiments shown illustrate the functions of corresponding steps in the autonomous driving perception method. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. These modules can be software and / or hardware, and each module can be implemented individually or integrated from multiple modules. For a detailed description of the functions of each module in the aforementioned autonomous driving perception device, please refer to [link to relevant documentation]. Figure 1 The corresponding description of the autonomous driving perception method in the illustrated embodiments will not be repeated here.

[0166] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0167] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0168] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the autonomous driving perception method provided in the embodiments of this disclosure.

[0169] Compared with the prior art, the electronic device extracts the bird's eye view feature based on the environment perception data of at least one modality, the environment perception data is collected for a target space where a target object is located, divides the target space into a plurality of voxels, and constructs initial voxel features corresponding to the voxels based on the positions of the voxels. The target voxel feature is extracted based on the initial voxel feature and the bird's eye view feature. The perception processing is performed based on the target voxel feature to obtain the perception result of the target space, and the perception processing includes occupancy prediction and at least one related perception processing. Based on the scheme, the occupancy prediction task and the related perception processing task can be jointly processed, the data calculation amount of automatic driving perception processing is effectively reduced, and the accuracy of automatic driving perception is improved.

[0170] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to make a computer execute the automatic driving perception method provided by the embodiments of the present disclosure.

[0171] Compared with the prior art, the electronic device extracts the bird's eye view feature based on the environment perception data of at least one modality, the environment perception data is collected for a target space where a target object is located, divides the target space into a plurality of voxels, and constructs initial voxel features corresponding to the voxels based on the positions of the voxels. The target voxel feature is extracted based on the initial voxel feature and the bird's eye view feature. The perception processing is performed based on the target voxel feature to obtain the perception result of the target space, and the perception processing includes occupancy prediction and at least one related perception processing. Based on the scheme, the occupancy prediction task and the related perception processing task can be jointly processed, the data calculation amount of automatic driving perception processing is effectively reduced, and the accuracy of automatic driving perception is improved.

[0172] The computer program product includes a computer program, and the computer program implements the automatic driving perception method provided by the embodiments of the present disclosure when executed by a processor.

[0173] Compared with the prior art, the electronic device extracts the bird's eye view feature based on the environment perception data of at least one modality, the environment perception data is collected for a target space where a target object is located, divides the target space into a plurality of voxels, and constructs initial voxel features corresponding to the voxels based on the positions of the voxels. The target voxel feature is extracted based on the initial voxel feature and the bird's eye view feature. The perception processing is performed based on the target voxel feature to obtain the perception result of the target space, and the perception processing includes occupancy prediction and at least one related perception processing. Based on the scheme, the occupancy prediction task and the related perception processing task can be jointly processed, the data calculation amount of automatic driving perception processing is effectively reduced, and the accuracy of automatic driving perception is improved.

[0174] Figure 4 A schematic block diagram of an example electronic device 40 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0175] As shown in Figure 4 The electronic device 40 includes a computing unit 410 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 420 or a computer program loaded into a random access memory (RAM) 430 from a storage unit 480. Various programs and data required for the operation of the device 40 can also be stored in the RAM 430. The computing unit 410, the ROM 420, and the RAM 430 are connected to each other through a bus 440. An input / output (I / O) interface 450 is also connected to the bus 440.

[0176] Various components in the device 40 are connected to the I / O interface 450, including an input unit 460, such as a keyboard, a mouse, etc.; an output unit 470, such as various types of displays, speakers, etc.; the storage unit 480, such as a magnetic disk, an optical disk, etc.; and a communication unit 490, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 490 allows the device 40 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0177] The computing unit 410 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 410 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 410 performs the autonomous driving perception method provided in the embodiments of the present disclosure. For example, in some embodiments, the autonomous driving perception method provided in the embodiments of the present disclosure can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 480. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 40 via the ROM 420 and / or the communication unit 490. When the computer program is loaded into the RAM 430 and executed by the computing unit 410, one or more steps of the autonomous driving perception method provided in the embodiments of the present disclosure can be performed. Alternatively, in other embodiments, the computing unit 410 can be configured to perform the autonomous driving perception method provided in the embodiments of the present disclosure by any other appropriate means, such as by means of firmware.

[0178] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0179] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0180] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0181] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0182] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0183] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0184] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0185] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An automatic driving perception method, comprising: obtaining an aerial view feature, the aerial view feature being extracted based on at least one modality of environment perception data, the environment perception data being collected for a target space in which a target object is located; dividing the target space into a plurality of voxels, and constructing an initial voxel feature corresponding to each voxel based on a position of the voxel; extracting a target voxel feature based on the initial voxel feature and the aerial view feature; performing perception processing based on the target voxel feature to obtain a perception result of the target space, the perception processing including occupancy prediction and at least one related perception processing associated with occupancy prediction.

2. The method of claim 1, wherein, The extracting of the target voxel feature based on the initial voxel feature and the aerial view feature comprises: performing global self-attention processing on the initial voxel feature to obtain a self-attention voxel feature; performing deformable cross-attention processing on the self-attention voxel feature and the aerial view feature to obtain the target voxel feature.

3. The method of claim 2, wherein, The performing of the deformable cross-attention processing on the self-attention voxel feature and the aerial view feature to obtain the target voxel feature comprises: performing local feature extraction on the aerial view feature based on position information of the voxel corresponding to the self-attention voxel feature to obtain a local aerial view feature corresponding to each self-attention voxel feature; performing cross-attention processing on each self-attention voxel feature and the corresponding local aerial view feature to obtain a cross-attention voxel feature; determining the target voxel feature based on the cross-attention voxel feature.

4. The method of claim 3, wherein, The determining of the target voxel feature based on the cross-attention voxel feature comprises: performing global self-attention processing on the cross-attention voxel feature as the initial voxel feature to obtain a self-attention voxel feature; performing local feature extraction on the aerial view feature based on position information of the voxel corresponding to the self-attention voxel feature to obtain a local aerial view feature corresponding to each self-attention voxel feature; performing cross-attention processing on each self-attention voxel feature and the corresponding local aerial view feature to obtain a cross-attention voxel feature; determining the cross-attention voxel feature satisfying a preset condition as the target voxel feature.

5. The method of any one of claims 1-4, wherein, The constructing of the initial voxel feature corresponding to each voxel based on a position of the voxel comprises: performing position encoding on the position of the voxel to obtain a position feature; performing multi-layer perception processing on the position feature to obtain a multi-layer perception result; performing at least one linear transformation on the multi-layer perception result to obtain the initial voxel feature corresponding to each voxel.

6. The method of any one of claims 1-5, wherein, The performing of the perception processing based on the target voxel feature to obtain the perception result of the target space comprises: performing deconvolution processing on the target voxel feature to obtain a deconvoluted voxel feature; performing occupancy prediction based on the deconvoluted voxel feature to obtain an occupancy prediction result.

7. The method of any one of claims 1-6, wherein, The related perception processing includes at least one of the following: target detection; lane line segmentation; drivable area segmentation.

8. The method of claim 7, wherein, In response to the related perception processing including target detection, the perception processing based on the target voxel feature is performed to obtain a perception result of the target space, including: compressing height information of the target voxel feature to obtain a compressed bird's eye view feature; performing target detection based on the compressed bird's eye view feature to obtain a target detection result.

9. The method of any one of claims 1-8, wherein, The environment perception data includes at least one of: image data; point cloud data.

10. The method of claim 9, wherein, In response to the environment perception data being image data, the bird's eye view feature includes: extracting a bird's eye view feature from the image data based on a preset bird's eye view feature extraction model.

11. An automatic driving perception device, comprising: a bird's eye view feature acquisition module configured to acquire a bird's eye view feature, the bird's eye view feature being extracted based on at least one modal environment perception data, the environment perception data being collected for a target space in which a target object is located; an initial voxel feature construction module configured to divide the target space into a plurality of voxels and construct an initial voxel feature corresponding to each voxel based on a position of each voxel; a target voxel feature extraction module configured to extract a target voxel feature based on the initial voxel feature and the bird's eye view feature; a perception processing module configured to perform perception processing based on the target voxel feature to obtain a perception result of the target space, the perception processing including occupancy prediction and at least one related perception processing related to occupancy prediction.

12. The apparatus of claim 11, wherein, The target voxel feature extraction module is specifically configured to: perform global self-attention processing on the initial voxel feature to obtain a self-attention voxel feature; perform deformable cross-attention processing on the self-attention voxel feature and the bird's eye view feature to obtain a target voxel feature.

13. The apparatus of claim 12, wherein, When the target voxel feature extraction module performs deformable cross-attention processing on the self-attention voxel feature and the bird's eye view feature to obtain a target voxel feature, it is specifically configured to: perform local feature extraction on the bird's eye view feature based on position information of the voxel corresponding to the self-attention voxel feature to obtain a local bird's eye view feature corresponding to each self-attention voxel feature; perform cross-attention processing on each self-attention voxel feature and the corresponding local bird's eye view feature to obtain a cross-attention voxel feature; determine the target voxel feature based on the cross-attention voxel feature.

14. The apparatus of claim 13, wherein, When the target voxel feature extraction module determines the target voxel feature based on the cross-attention voxel feature, it is specifically configured to: perform global self-attention processing on the initial voxel feature to obtain a self-attention voxel feature, the initial voxel feature being the cross-attention voxel feature; perform local feature extraction on the bird's eye view feature based on position information of the voxel corresponding to the self-attention voxel feature to obtain a local bird's eye view feature corresponding to each self-attention voxel feature; perform cross-attention processing on each self-attention voxel feature and the corresponding local bird's eye view feature to obtain a cross-attention voxel feature; determine the cross-attention voxel feature satisfying a preset condition as the target voxel feature.

15. The apparatus of any of claims 11-14, wherein, The initial voxel feature construction module is specifically configured to: perform position coding based on the position of each voxel to obtain a position feature; perform multi-layer perception processing on the position feature to obtain a multi-layer perception result; perform at least one linear transformation on the multi-layer perception result to obtain the initial voxel feature corresponding to each voxel.

16. The apparatus of any one of claims 11-15, wherein, The perception processing module is specifically configured to: perform deconvolution processing on the target voxel feature to obtain a deconvolution processed voxel feature; perform occupancy prediction based on the deconvolution processed voxel feature to obtain an occupancy prediction result.

17. The apparatus of any of claims 11-16, wherein, The related perception processing includes at least one of: target detection; lane line segmentation; drivable area segmentation. 18.An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

19. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-10. 20.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-10.

Images