Voxel semantic category determination method, device, computer equipment and storage medium

Abstract

The present disclosure relates to the field of computer technology and discloses a method, apparatus, computer equipment, and storage medium for determining voxel semantic categories. The method comprises: obtaining image information, image pixel depth distribution of the image information, and point cloud data of the image information; obtaining voxel features based on the image information and the image pixel depth distribution, and obtaining grid features of a three-dimensional grid based on the point cloud features, wherein the voxel features are pixel features with height information; obtaining image features at multiple preset perspectives based on the voxel features, and obtaining point cloud features at multiple preset perspectives based on the grid features; obtaining the target voxel semantic category to which the image information belongs based on the image features at the preset perspectives, the point cloud features at the preset perspectives, and a target model, wherein the target model is used to determine the type of obstacle to which the image information belongs. The method solves the problem in related technologies that the lack of representational capabilities of two-dimensional grid maps cannot meet the complex and diverse autonomous driving scenarios in reality.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, apparatus, computer device, and storage medium for determining a voxel semantic category. Background Art

[0002] Environmental perception is a prerequisite for autonomous driving systems and provides an effective basis for path planning. Stable and reliable obstacle perception technology plays a key role in enabling autonomous vehicles to execute actions such as stopping and avoiding obstacles.

[0003] Currently, the industry generally uses a three-dimensional bounding box approach to represent the location, size, and orientation of obstacles in the environment. This representation method is suitable for most standard obstacle types. However, real-world environments are extremely complex and diverse. Autonomous vehicles may encounter unusual obstacles such as forklifts, cranes with extended legs or arms, and operating steel grabbers. The three-dimensional bounding box representation method is not very effective for these unusual obstacles and may be larger or smaller than the actual obstacle size, affecting the safety and efficiency of the autonomous driving system. To represent the autonomous driving system's operational area, some industries have adopted a grid map approach. These grids are marked as occupied where obstacles are present, allowing the planning module to perform path planning based on the grid map. However, the two-dimensional planar grid map currently used by the industry generally lacks height information, resulting in weak overall representation capabilities and unsuitable for three-dimensional space. Therefore, two-dimensional grid maps are clearly unable to meet the complex and diverse needs of real-world autonomous driving scenarios.

[0004] Therefore, the current related technologies cannot meet the complex and diverse autonomous driving scenarios in reality due to the lack of representation capabilities of two-dimensional grid maps. Summary of the Invention

[0005] In view of this, the present disclosure provides a voxel semantic category determination method, apparatus, computer device and storage medium to solve the problem in related technologies that the lack of representation capabilities of two-dimensional grid maps cannot meet the complex and diverse autonomous driving scenarios in reality.

[0006] In a first aspect, the present disclosure provides a method for determining a voxel semantic category, the method comprising:

[0007] Acquire image information, image pixel depth distribution of the image information, and point cloud data of the image information;

[0008] The voxel features are obtained based on the image information and the depth distribution of the image pixels, and the grid features of the three-dimensional grid are obtained based on the point cloud features, wherein the voxel features are pixel features with height information;

[0009] Acquire image features at multiple preset viewing angles based on voxel features, and acquire point cloud features at multiple preset viewing angles based on grid features;

[0010] The target voxel semantic category to which the image information belongs is obtained according to the image features at a preset viewing angle, the point cloud features at a preset viewing angle, and the target model, wherein the target model is used to determine the type of obstacle to which the image information belongs.

[0011] In the embodiment of the present disclosure, image features at a preset perspective and point cloud features at a preset perspective are determined through image information, image pixel depth distribution and point cloud data. The target model is input based on the image features at the preset perspective and the point cloud features at the preset perspective to obtain the target voxel semantic category, and the obstacle type to which the image information belongs is determined. Since the embodiment of the present disclosure obtains image features and point cloud features at multiple preset perspectives, combined with the image features and point cloud features from multiple perspectives, the represented information is richer, the output target voxel semantic category is more accurate, and it can effectively handle special-shaped obstacles, solving the problem in related technologies that the lack of representation capabilities of two-dimensional grid maps cannot meet the complex and diverse autonomous driving scenarios in reality.

[0012] In an optional embodiment, obtaining voxel features based on image information and image pixel depth distribution includes:

[0013] Input the image information into the image feature extraction network to obtain image features;

[0014] Take the outer product of the image features and the image pixel depth distribution to obtain the cone features;

[0015] The space where the frustum feature is located is converted into a three-dimensional grid to obtain each sampling point on the three-dimensional grid;

[0016] The sampling points are projected into the camera coordinate system through the camera calibration parameters, and the frustum features of the corresponding positions of each sampling point in the camera coordinate system are extracted to obtain the voxel features of each sampling point.

[0017] In the embodiment of the present disclosure, image features are extracted from the image information, and the outer product of the image features and the image pixel depth distribution is performed to obtain the cone features, and then the cone features of each sampling point on the three-dimensional grid projected to the camera coordinate system are obtained, and then the voxel features with height information are determined to meet the detailed characterization of the three-dimensional information.

[0018] In an optional embodiment, extracting the view cone features corresponding to the positions of each sampling point in the camera coordinate system to obtain the voxel features of each sampling point includes:

[0019] Get the frustum features of the same sampling point in multiple camera coordinate systems;

[0020] Accumulate multiple cone features to obtain the voxel features of the same sampling point;

[0021] Traverse all sampling points and obtain the voxel features of each sampling point.

[0022] In the embodiment of the present disclosure, since the same sampling point appears in the fields of view of multiple cameras, the cone features of the same sampling point in multiple camera coordinate systems are obtained, and then all sampling points are traversed to generate all voxel features, which is conducive to the subsequent acquisition of accurate image features.

[0023] In an optional embodiment, obtaining the mesh features of the three-dimensional mesh according to the point cloud features includes:

[0024] Inputting the point cloud data into a point cloud feature extraction network to obtain point cloud features;

[0025] The space where the point cloud features are located is converted into a three-dimensional grid, and the global features in each grid cell on the three-dimensional grid are obtained;

[0026] The grid features of the three-dimensional grid are obtained according to the global features within the grid cells.

[0027] In the disclosed embodiment, point cloud features are obtained based on point cloud data, and then point cloud features within each grid unit on the three-dimensional grid are determined, thereby determining the grid features of the three-dimensional grid, which facilitates feature fusion with image features.

[0028] In an optional embodiment, obtaining image features at multiple preset viewing angles based on voxel features, and obtaining point cloud features at multiple preset viewing angles based on grid features, includes:

[0029] Performing a collapse operation on the voxel features along a first direction to obtain image features at a first viewing angle, and performing a collapse operation on the voxel features along a second direction to obtain image features at a second viewing angle;

[0030] The mesh features are collapsed in a first direction to obtain point cloud features at a first perspective, and the mesh features are collapsed in a second direction to obtain point cloud features at a third perspective.

[0031] In the embodiment of the present disclosure, a collapse operation is performed on voxel features in a preset direction to obtain image features at a preset perspective, and a collapse operation is performed on grid features in a preset direction to obtain point cloud features at a preset perspective, so that the representation information of the image information is richer and the representation capability is stronger.

[0032] In an optional embodiment, obtaining the target voxel semantic category to which the image information belongs based on the image features at a preset viewing angle, the point cloud features at a preset viewing angle, and the target model includes:

[0033] Copying and extending the image features and point cloud features at the first viewing angle along the first direction to obtain corresponding first three-dimensional spatial features, copying and extending the image features at the second viewing angle along the second direction to obtain corresponding second three-dimensional spatial features, and copying and extending the point cloud features at the third viewing angle along the second direction to obtain corresponding third three-dimensional spatial features;

[0034] fusing the first three-dimensional spatial feature, the second three-dimensional spatial feature, and the third three-dimensional spatial feature to obtain a fused target feature;

[0035] The target features are input into the target model to obtain multiple voxel semantic categories and the probability value of each voxel semantic category, and the target voxel semantic category is determined.

[0036] In the disclosed embodiment, the image features and point cloud features from the first perspective are copied and expanded along the first direction, and the image features from the second perspective and the point cloud features from the third perspective are copied and expanded along the second direction. Then, the three-dimensional spatial features are fused, and the fused target features are input into the target model. The accurate obstacle category is output, laying the foundation for the subsequent effective processing of various special-shaped obstacles.

[0037] In an optional embodiment, before obtaining the target voxel semantic category to which the image information belongs based on the image features at the preset perspective, the point cloud features at the preset perspective, and the target model, the method further includes:

[0038] Acquire training sample data, wherein the training sample data includes image training sample data and point cloud training sample data, and the training sample data carries point cloud semantic labels corresponding to the training sample data;

[0039] Input the training sample data into the initial model to obtain the initial voxel semantic category;

[0040] Compare the initial voxel semantic category with the point cloud semantic label, adjust the initial weight parameters in the initial model until the initial voxel semantic category is consistent with the point cloud semantic label, obtain the intermediate weight parameters after adjusting the initial weight parameters, and determine the intermediate model;

[0041] Obtain image fault data and point cloud fault data and add them to training sample data to obtain new training sample data;

[0042] Input the new training sample data into the intermediate model to obtain the intermediate voxel semantic category;

[0043] The intermediate voxel semantic category is compared with the point cloud semantic label, and the intermediate weight parameters in the intermediate model are adjusted until the intermediate voxel semantic category is consistent with the point cloud semantic label, and the target weight parameters after adjusting the intermediate weight parameters are obtained to determine the target model.

[0044] In the disclosed embodiment, the training model of normal training sample data and the training model of new training sample data with added image fault data are combined, so that the obtained target model can be applicable to the working condition of partial sensor failure, and the solution has high robustness.

[0045] In a second aspect, the present disclosure provides a device for determining a voxel semantic category, the device comprising:

[0046] A first acquisition module is used to acquire image information, image pixel depth distribution of the image information, and point cloud data of the image information;

[0047] The first obtaining module is used to obtain voxel features based on image information and image pixel depth distribution, and obtain grid features of the three-dimensional grid based on point cloud features, wherein the voxel feature is a pixel feature with height information;

[0048] A second acquisition module is used to acquire image features under multiple preset viewing angles based on voxel features, and to acquire point cloud features under multiple preset viewing angles based on grid features;

[0049] The second obtaining module is used to obtain the target voxel semantic category to which the image information belongs based on the image features at a preset perspective, the point cloud features at a preset perspective, and the target model, wherein the target model is used to determine the type of obstacle to which the image information belongs.

[0050] In a third aspect, the present disclosure provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the voxel semantic category determination method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0051] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the voxel semantic category determination method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 is a flowchart of a method for determining voxel semantic categories according to an embodiment of the present disclosure;

[0054] Figure 2 is a schematic diagram of a process for extracting multi-view features of an image according to an embodiment of the present disclosure;

[0055] Figure 3 Schematic diagram of a model training strategy combining two modes: complete input data and partially missing input data according to an embodiment of the present disclosure;

[0056] Figure 4 is a schematic diagram of the overall process of the method for determining voxel semantic categories according to an embodiment of the present disclosure;

[0057] Figure 5 is a structural block diagram of a device for determining voxel semantic categories according to an embodiment of the present disclosure;

[0058] Figure 6 Schematic diagram of the hardware structure of the computer device according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0059] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present disclosure.

[0060] When planning a path for an autonomous vehicle, it's necessary to understand the location, size, and direction of obstacles. Currently, some approaches use grid maps, where grids with obstacles are set to occupied. The planning module can then perform path planning based on the grid map. However, the two-dimensional planar grid map, which is currently widely used in the industry, lacks sufficient representational capabilities. For example, when the boom of a crane or steel grabber moves in three-dimensional space, if it's raised in the air and above the autonomous vehicle, it doesn't affect the autonomous driving route. However, because the two-dimensional grid map lacks height information, the space occupied by the boom will be projected onto the grid map, potentially causing the autonomous driving system to make unnecessary avoidance stops or detours, thus affecting its safety and efficiency. Therefore, the two-dimensional grid maps currently in common use clearly cannot meet the complex and diverse autonomous driving scenarios in reality.

[0061] In order to solve the problem in related technologies that the two-dimensional grid map is insufficient in representation capability and cannot meet the complex and diverse autonomous driving scenarios in reality, the present disclosure proposes a voxel semantic category determination method, such as Figure 1 As shown, the method can be applied to the server side, and the method may include the following steps:

[0062] Step S101 : acquiring image information, image pixel depth distribution of the image information, and point cloud data of the image information.

[0063] Optionally, in the embodiment of the present disclosure, image information of the vehicle's surrounding area on the road it is traveling on can be obtained using image acquisition devices such as cameras and webcams, and then the image information can be preprocessed, mainly including image cropping, scaling, pixel normalization and other processing.

[0064] Based on the image information, the depth distribution of image pixels can be estimated, and then the depth information can be extracted from the image information, and then the depth information can be converted into point cloud data.

[0065] Step S102 , obtaining voxel features based on the image information and the image pixel depth distribution, and obtaining mesh features of the three-dimensional mesh based on the point cloud features, wherein the voxel features are pixel features with height information.

[0066] Optionally, based on the image information and the image pixel depth distribution, these two information are processed to obtain voxel features with height information. At the same time, based on the point cloud features, the space is first divided into a three-dimensional grid, and then a pooling operation is used on the point cloud features within each grid unit to generate the features of the grid unit, and then the grid features of the entire three-dimensional grid are obtained.

[0067] Step S103 : acquiring image features at multiple preset viewing angles based on the voxel features, and acquiring point cloud features at multiple preset viewing angles based on the grid features.

[0068] Optionally, in the embodiment of the present disclosure, the preset perspectives include a BEV perspective (i.e., a bird's-eye view), a Front perspective (i.e., a front perspective), and a Range perspective (i.e., an interval perspective), and then image features under multiple preset perspectives are obtained based on voxel features, which may be image features under the BEV perspective and the Front perspective; point cloud features under multiple preset perspectives are obtained based on grid features, which may be point cloud features under the BEV perspective and the Range perspective.

[0069] Step S104 , obtaining the target voxel semantic category to which the image information belongs based on the image features at the preset viewing angle, the point cloud features at the preset viewing angle, and the target model, wherein the target model is used to determine the obstacle type to which the image information belongs.

[0070] Optionally, the image features and point cloud features obtained at the preset viewing angle are input into a target model. After data processing and analysis by the target model, the target voxel semantic category to which the image information belongs is output. It should be noted that the voxel semantic category can be obstacle types in autonomous driving scenarios, such as vehicles, pedestrians, motorcycles, construction signs, and cones.

[0071] In the embodiment of the present disclosure, image features at a preset perspective and point cloud features at a preset perspective are determined through image information, image pixel depth distribution and point cloud data. The target model is input based on the image features at the preset perspective and the point cloud features at the preset perspective to obtain the target voxel semantic category, and the obstacle type to which the image information belongs is determined. Since the embodiment of the present disclosure obtains image features and point cloud features at multiple preset perspectives, combined with the image features and point cloud features from multiple perspectives, the represented information is richer, the output target voxel semantic category is more accurate, and it can effectively handle special-shaped obstacles, solving the problem in related technologies that the lack of representation capabilities of two-dimensional grid maps cannot meet the complex and diverse autonomous driving scenarios in reality.

[0072] In some optional implementations, obtaining voxel features based on image information and image pixel depth distribution includes:

[0073] Input the image information into the image feature extraction network to obtain image features;

[0074] Take the outer product of the image features and the image pixel depth distribution to obtain the cone features;

[0075] The space where the frustum feature is located is converted into a three-dimensional grid to obtain each sampling point on the three-dimensional grid;

[0076] The sampling points are projected into the camera coordinate system through the camera calibration parameters, and the frustum features of the corresponding positions of each sampling point in the camera coordinate system are extracted to obtain the voxel features of each sampling point.

[0077] Optionally, in the embodiment of the present disclosure, the preprocessed image information can be input into the image feature extraction network, and common image backbone networks (such as ResNet (Residual Network), Vgg (Visual Geometry Group Network), DLA (Deep Layer Aggregation), etc.) can be used to extract image features.

[0078] By taking the outer product of the image features (dimensional format such as C*H*W, where C represents the number of image channels, H represents the height of the image, and W represents the width of the image) and the pixel depth distribution (dimensional format such as D*H*W, where D represents the depth of the image), we can obtain the cone features (dimensional format such as C*D*H*W). The space where the cone features are located is converted into a three-dimensional grid to obtain the sampling points on the three-dimensional grid. The sampling points on the three-dimensional grid are then projected to the camera coordinate system through the camera calibration parameters. The cone features of each sampling point at the corresponding position in the camera coordinate system are directly extracted to obtain the voxel features corresponding to each sampling point.

[0079] In the embodiment of the present disclosure, image features are extracted from the image information, and the outer product of the image features and the image pixel depth distribution is performed to obtain the cone features, and then the cone features of each sampling point on the three-dimensional grid projected to the camera coordinate system are obtained, and then the voxel features with height information are determined to meet the detailed characterization of the three-dimensional information.

[0080] In some optional implementations, extracting the view cone features corresponding to the positions of each sampling point in the camera coordinate system to obtain the voxel features of each sampling point includes:

[0081] Get the frustum features of the same sampling point in multiple camera coordinate systems;

[0082] Accumulate multiple cone features to obtain the voxel features of the same sampling point;

[0083] Traverse all sampling points and obtain the voxel features of each sampling point.

[0084] Optionally, considering that the autonomous driving system can be configured with multiple cameras, the same three-dimensional space sampling point may appear in the field of view of multiple cameras. In this case, the cone features of the same sampling point in multiple camera coordinate systems are obtained, and these cone features are accumulated to obtain the voxel features corresponding to the sampling point. After traversing all sampling points on the three-dimensional grid, all voxel features can be generated (dimensional format such as C*Z*X*Y).

[0085] In the embodiment of the present disclosure, since the same sampling point appears in the fields of view of multiple cameras, the cone features of the same sampling point in multiple camera coordinate systems are obtained, and then all sampling points are traversed to generate all voxel features, which is conducive to the subsequent acquisition of accurate image features.

[0086] In some optional implementations, obtaining mesh features of a three-dimensional mesh based on point cloud features includes:

[0087] Input the point cloud data into the point cloud feature extraction network to obtain point cloud features;

[0088] The space where the point cloud features are located is converted into a three-dimensional grid, and the global features in each grid cell on the three-dimensional grid are obtained;

[0089] The grid features of the three-dimensional grid are obtained based on the global features within the grid cells.

[0090] Optionally, the point cloud data is preprocessed, mainly including operations such as point cloud downsampling and ground point cloud filtering. Next, the preprocessed point cloud data is input into the point cloud feature extraction network, and a PointNet-like network structure or a fully connected layer can be used to extract point cloud features. Then, the space where the point cloud features are located is first divided into a three-dimensional grid, and then the point cloud features within each grid cell are pooled to generate the global features of the grid cell.

[0091] The grid characteristics of the entire three-dimensional grid can be obtained from the global characteristics of each grid cell.

[0092] In the disclosed embodiment, point cloud features are obtained based on point cloud data, and then point cloud features within each grid unit on the three-dimensional grid are determined, thereby determining the grid features of the three-dimensional grid, which facilitates feature fusion with image features.

[0093] In some optional implementations, obtaining image features at multiple preset viewing angles based on voxel features, and obtaining point cloud features at multiple preset viewing angles based on grid features, includes:

[0094] Performing a collapse operation on the voxel features along a first direction to obtain image features at a first viewing angle, and performing a collapse operation on the voxel features along a second direction to obtain image features at a second viewing angle;

[0095] The mesh features are collapsed in a first direction to obtain point cloud features at a first perspective, and the mesh features are collapsed in a second direction to obtain point cloud features at a third perspective.

[0096] Optionally, by collapsing the voxel features along a preset direction, such as the first direction Z, the first perspective, i.e., the image features under the BEV perspective (the dimensional format is such as C*X*Y respectively) can be obtained. By collapsing the voxel features along a preset direction, such as the second direction X, the second perspective, i.e., the image features under the Front perspective (the dimensional format is such as C*Z*Y) can be obtained. The above-mentioned collapse operation can be achieved through the pooling operation or aggregation operation commonly used in neural networks.

[0097] By collapsing the entire three-dimensional grid feature along a preset direction, such as the first direction Z direction, the first perspective, that is, the point cloud feature under the BEV perspective, can be obtained. By collapsing the entire three-dimensional grid feature along a preset direction, such as the second direction, such as the X direction, the third perspective, that is, the point cloud feature under the Range perspective, can be obtained. The above-mentioned collapse operation can still be achieved through the pooling operation or aggregation operation commonly used in neural networks.

[0098] In the embodiment of the present disclosure, a collapse operation is performed on voxel features in a preset direction to obtain image features at a preset perspective, and a collapse operation is performed on grid features in a preset direction to obtain point cloud features at a preset perspective, so that the representation information of the image information is richer and the representation capability is stronger.

[0099] Based on the above embodiments, in some optional implementations, such as Figure 2 As shown, Figure 2 FIG. 1 is a flow chart of multi-view feature extraction of an image according to an embodiment of the present disclosure, wherein the specific steps include the following:

[0100] The depth distribution is estimated from the image features, and the pixel depth is obtained from the depth distribution estimate. The pixel depth and image features are subjected to outer product operation to obtain the frustum features.

[0101] Divide the image into 3D grids to obtain 3D spatial sampling points, combine them with the camera's internal and external parameters, project the sampling points into the camera coordinate system, extract the corresponding cone features, or when the same 3D spatial sampling point appears in multiple camera fields of view, accumulate the cone features, and then obtain voxel features based on the cone features;

[0102] The voxel features are collapsed along the Z direction to obtain the BEV perspective image features, and the voxel features are collapsed along the X direction to obtain the Front perspective image features.

[0103] In some optional implementations, obtaining the target voxel semantic category to which the image information belongs based on the image features at a preset viewing angle, the point cloud features at a preset viewing angle, and the target model includes:

[0104] The image features and point cloud features at the first viewing angle are copied and expanded along the first direction to obtain corresponding first three-dimensional spatial features, the image features at the second viewing angle are copied and expanded along the second direction to obtain corresponding second three-dimensional spatial features, and the point cloud features at the third viewing angle are copied and expanded along the second direction to obtain corresponding third three-dimensional spatial features;

[0105] Fusing the first three-dimensional spatial feature, the second three-dimensional spatial feature, and the third three-dimensional spatial feature to obtain a fused target feature;

[0106] The target features are input into the target model to obtain multiple voxel semantic categories and the probability value of each voxel semantic category, and the target voxel semantic category is determined.

[0107] Optionally, a feature encoder is first used, and stacked convolutional layers are used to enhance image features and point cloud features at different perspectives. After the image features and point cloud features are processed by the feature encoder, the image features and point cloud features at the first perspective, i.e., the BEV perspective, are copied and expanded along the Z direction to obtain corresponding first three-dimensional spatial features, the image features at the second perspective, i.e., the Front perspective, are copied and expanded along the X direction to obtain corresponding second three-dimensional spatial features, and the point cloud features at the third perspective, i.e., the Range perspective, are copied and expanded along the X direction to obtain corresponding third three-dimensional spatial features.

[0108] The first three-dimensional spatial feature, the second three-dimensional spatial feature and the third three-dimensional spatial feature are fused through aggregation, pooling or serial operation to obtain a fused target feature.

[0109] The target features are then input into the target model, which outputs multiple voxel semantic categories and the probability values ​​of each voxel semantic category, such as obstacle types and probabilities in autonomous driving scenarios, such as vehicles, pedestrians, motorcycles, construction signs, cones, etc. Finally, the target voxel semantic category corresponding to the maximum probability is used as the obstacle category identified from the image information.

[0110] In the disclosed embodiment, the image features and point cloud features from the first perspective are copied and expanded along the first direction, and the image features from the second perspective and the point cloud features from the third perspective are copied and expanded along the second direction. Then, the three-dimensional spatial features are fused, and the fused target features are input into the target model. The accurate obstacle category is output, laying the foundation for the subsequent effective processing of various special-shaped obstacles.

[0111] In some optional embodiments, before obtaining the target voxel semantic category to which the image information belongs based on the image features at the preset viewing angle, the point cloud features at the preset viewing angle, and the target model, the method further includes:

[0112] Acquire training sample data, wherein the training sample data includes image training sample data and point cloud training sample data, and the training sample data carries point cloud semantic labels corresponding to the training sample data;

[0113] Input the training sample data into the initial model to obtain the initial voxel semantic category;

[0114] Compare the initial voxel semantic category with the point cloud semantic label, adjust the initial weight parameters in the initial model until the initial voxel semantic category is consistent with the point cloud semantic label, obtain the intermediate weight parameters after adjusting the initial weight parameters, and determine the intermediate model;

[0115] Obtain image fault data and point cloud fault data and add them to training sample data to obtain new training sample data;

[0116] Input the new training sample data into the intermediate model to obtain the intermediate voxel semantic category;

[0117] The intermediate voxel semantic category is compared with the point cloud semantic label, and the intermediate weight parameters in the intermediate model are adjusted until the intermediate voxel semantic category is consistent with the point cloud semantic label, and the target weight parameters after adjusting the intermediate weight parameters are obtained to determine the target model.

[0118] Optionally, considering that in real scenarios, the sensors of the autonomous driving system may malfunction or fail, in order to ensure the safety of the autonomous driving system, the perception module needs to work under such harsh working conditions. In the embodiment of the present disclosure, in the first stage, training sample data is first obtained. The training sample data is some complete and correct sample data. At the same time, the training sample data includes image training sample data and point cloud training sample data. The training sample data carries the marked point cloud semantic labels corresponding to the training sample data as true values. Then, the training sample data is input into the initial model to obtain the initial voxel semantic category, and the initial voxel semantic category is compared with the point cloud semantic label. The initial weight parameters in the initial model are adjusted until the initial voxel semantic category is consistent with the point cloud semantic label, and the intermediate weight parameters after the initial weight parameters are adjusted are obtained, and the intermediate model is determined.

[0119] In the second phase, a condition in which some lidar or camera sensors malfunction or fail is simulated. Some image and point cloud malfunction data (i.e., some data input is missing) is added as network model input and added to the training sample data to generate new training sample data. The previously labeled point cloud semantic labels are still used as the ground truth. This new training sample data is input into the intermediate model to obtain intermediate voxel semantic categories. These intermediate voxel semantic categories are compared with the point cloud semantic labels, and the intermediate weight parameters in the intermediate model are adjusted until they align with the point cloud semantic labels. This adjustment results in the target weight parameters, which are then used to determine the target model.

[0120] At the same time, if Figure 3 As shown, Figure 3 It is a schematic diagram of a model training strategy combining two modes, namely, complete input data and partially missing input data, according to an embodiment of the present disclosure. As can be seen from the figure, the model training includes two stages: first-stage training and second-stage training.

[0121] The first stage of training: input the image data and point cloud data into the neural network model training, compare them with the annotated point cloud semantic label data, adjust the model weight, and obtain the model weight WO;

[0122] The second stage of training: Part of the image data or point cloud data is input into the neural network model training, compared with the annotated point cloud semantic label data, and the model weight WO is adjusted to obtain the model weight W1, which is used as the weight parameter of the neural network model used in the final deployment.

[0123] In the disclosed embodiment, the training model of normal training sample data and the training model of new training sample data with added image fault data are combined, so that the obtained target model can be applicable to the working condition of partial sensor failure, and the solution has high robustness.

[0124] Based on the contents of the above embodiments, Figure 4 As shown, Figure 4 FIG. 5 is a schematic diagram of the overall process of the method for determining voxel semantic categories according to an embodiment of the present disclosure. The specific process is as follows:

[0125] S1 image multi-view feature extraction module: acquires image data, performs image preprocessing, uses the image feature extraction network to extract image features, inputs the extracted image features into the view conversion module to obtain BEV view image features and Front view image features, and then performs feature encoder processing on the BEV view image features and Front view image features.

[0126] S2 point cloud multi-view feature extraction module: acquire point cloud data, perform point cloud preprocessing, use the point cloud feature extraction network to extract point cloud features, input the extracted point cloud features into the dimension reduction module to obtain BEV view point cloud features and Range view point cloud features, and then perform feature encoder processing on the BEV view point cloud features and Range view point cloud features.

[0127] S3 multi-modal and multi-view feature fusion module: After being processed by the feature encoder, the input is input into the dimension expansion module for dimension expansion, and then input into the spatial feature aggregation module to fuse the image features and point cloud features.

[0128] S4 voxel semantic occupancy discrimination module: The fused features are input into the semantic occupancy discrimination module and the semantic occupancy category is output.

[0129] In this embodiment, a voxel semantic category determination device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments. The details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0130] This embodiment provides a voxel semantic category determination device, such as Figure 5 Shown, including:

[0131] A first acquisition module 501 is used to acquire image information, image pixel depth distribution of the image information, and point cloud data of the image information;

[0132] A first obtaining module 502 is configured to obtain voxel features based on the image information and the depth distribution of the image pixels, and obtain mesh features of the three-dimensional mesh based on the point cloud features, wherein the voxel features are pixel features with height information;

[0133] A second acquisition module 503 is configured to acquire image features at multiple preset viewing angles based on voxel features, and acquire point cloud features at multiple preset viewing angles based on grid features;

[0134] The second obtaining module 504 is used to obtain the target voxel semantic category to which the image information belongs based on the image features at a preset perspective, the point cloud features at a preset perspective, and the target model, wherein the target model is used to determine the obstacle type to which the image information belongs.

[0135] In some optional implementations, the first obtaining module 502 includes:

[0136] A first obtaining unit is used to input image information into an image feature extraction network to obtain image features;

[0137] The second obtaining unit is used to perform outer product between the image feature and the image pixel depth distribution to obtain the cone feature;

[0138] The third unit is used to convert the space where the cone feature is located into a three-dimensional grid, and obtain each sampling point on the three-dimensional grid;

[0139] The fourth obtaining unit is used to project the sampling points to the camera coordinate system through the camera calibration parameters, extract the cone features of the corresponding positions of each sampling point in the camera coordinate system, and obtain the voxel features of each sampling point.

[0140] In some optional embodiments, the fourth obtaining unit includes:

[0141] The first acquisition submodule is used to obtain the cone features of the same sampling point in multiple camera coordinate systems;

[0142] The accumulation submodule is used to accumulate multiple cone features to obtain the voxel features of the same sampling point;

[0143] A submodule is obtained, which is used to traverse all sampling points and obtain the voxel features of each sampling point.

[0144] In some optional implementations, the first obtaining module 502 includes:

[0145] The fifth obtaining unit is used to input the point cloud data into the point cloud feature extraction network to obtain point cloud features;

[0146] Sixth, a unit is obtained, which is used to convert the space where the point cloud features are located into a three-dimensional grid, and obtain the global features in each grid unit on the three-dimensional grid;

[0147] The seventh step is to obtain a unit, which is used to obtain the grid features of the three-dimensional grid according to the global features in the grid unit.

[0148] In some optional implementations, the second acquisition module 503 includes:

[0149] A first collapse operation unit is configured to perform a collapse operation on the voxel features along a first direction to obtain image features at a first viewing angle, and perform a collapse operation along a second direction to obtain image features at a second viewing angle;

[0150] The second collapse operation unit is used to perform a collapse operation on the mesh features in a first direction to obtain point cloud features at a first perspective, and to perform a collapse operation on the mesh features in a second direction to obtain point cloud features at a third perspective.

[0151] In some optional implementations, the second obtaining module 504 includes:

[0152] an expansion unit, configured to copy and expand the image features and point cloud features at the first viewing angle along a first direction to obtain corresponding first three-dimensional spatial features, copy and expand the image features at the second viewing angle along a second direction to obtain corresponding second three-dimensional spatial features, and copy and expand the point cloud features at the third viewing angle along the second direction to obtain corresponding third three-dimensional spatial features;

[0153] a fusion unit, configured to fuse the first three-dimensional spatial feature, the second three-dimensional spatial feature, and the third three-dimensional spatial feature to obtain a fused target feature;

[0154] The determination unit is used to input the target feature into the target model, obtain multiple voxel semantic categories and the probability value of each voxel semantic category, and determine the target voxel semantic category.

[0155] In some optional embodiments, the device further comprises:

[0156] A third acquisition module is configured to acquire training sample data before obtaining the target voxel semantic category to which the image information belongs based on the image features at the preset perspective, the point cloud features at the preset perspective, and the target model, wherein the training sample data includes image training sample data and point cloud training sample data, and the training sample data carries a point cloud semantic label corresponding to the training sample data;

[0157] The third module is used to input the training sample data into the initial model to obtain the initial voxel semantic category;

[0158] A first determination module is configured to compare the initial voxel semantic category with the point cloud semantic label, adjust the initial weight parameters in the initial model until the initial voxel semantic category is consistent with the point cloud semantic label, obtain intermediate weight parameters after adjusting the initial weight parameters, and determine the intermediate model;

[0159] A fourth acquisition module is used to acquire image fault data and point cloud fault data and add them to the training sample data to obtain new training sample data;

[0160] The fourth obtaining module is used to input the new training sample data into the intermediate model to obtain the intermediate voxel semantic category;

[0161] The second determination module is used to compare the intermediate voxel semantic category with the point cloud semantic label, adjust the intermediate weight parameters in the intermediate model until the intermediate voxel semantic category is consistent with the point cloud semantic label, obtain the target weight parameters after adjusting the intermediate weight parameters, and determine the target model.

[0162] The voxel semantic category determination device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0163] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0164] The present disclosure also provides a computer device having the above Figure 5 The voxel semantic category determination device shown.

[0165] See also Figure 6 , Figure 6 is a structural diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 6 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0166] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0167] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0168] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created based on the use of a computer device for displaying a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0169] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0170] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0171] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0172] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A voxel semantic category determination method, characterized in that: The method comprises: Acquiring image information, image pixel depth distribution of the image information, and point cloud data of the image information; Obtaining voxel features based on the image information and the depth distribution of the image pixels, and obtaining grid features of a three-dimensional grid based on the point cloud data, wherein the voxel features are pixel features with height information; Acquire image features at multiple preset viewing angles based on the voxel features, and acquire point cloud features at multiple preset viewing angles based on the grid features; Obtaining a target voxel semantic category to which the image information belongs based on the image features at the preset viewing angle, the point cloud features at the preset viewing angle, and a target model, wherein the target model is used to determine the type of obstacle to which the image information belongs; The acquiring of image features at multiple preset viewing angles according to the voxel features, and acquiring of point cloud features at multiple preset viewing angles according to the grid features, includes: performing a collapse operation on the voxel features along a first direction to obtain image features at a first viewing angle, and performing a collapse operation along a second direction to obtain image features at a second viewing angle; The grid features are collapsed along the first direction to obtain point cloud features at the first viewing angle, and the grid features are collapsed along the second direction to obtain point cloud features at a third viewing angle.

2. The method according to claim 1, characterized in that The obtaining of voxel features according to the image information and the image pixel depth distribution includes: Inputting the image information into an image feature extraction network to obtain image features; Performing an outer product of the image feature and the image pixel depth distribution to obtain a viewing cone feature; The space where the cone feature is located is converted into a three-dimensional grid to obtain sampling points on the three-dimensional grid; The sampling points are projected onto a camera coordinate system using camera calibration parameters, and the view cone features at positions corresponding to the respective sampling points in the camera coordinate system are extracted to obtain the voxel features of the respective sampling points.

3. The method according to claim 2, characterized in that The extracting the view cone features at positions corresponding to the respective sampling points in the camera coordinate system to obtain the voxel features of the respective sampling points includes: Obtaining the view cone features of the same sampling point in multiple camera coordinate systems; Accumulating the multiple visual cone features to obtain the voxel feature of the same sampling point; All sampling points are traversed to obtain the voxel features of each sampling point.

4. The method according to claim 1, wherein The step of obtaining a grid feature of a three-dimensional grid according to the point cloud data includes: Inputting the point cloud data into a point cloud feature extraction network to obtain point cloud features; The space where the point cloud features are located is converted into a three-dimensional grid, and the global features in each grid cell on the three-dimensional grid are obtained; The grid features of the three-dimensional grid are obtained according to the global features within the grid cells.

5. The method according to claim 1, wherein The obtaining, based on the image features at the preset viewing angle, the point cloud features at the preset viewing angle, and the target model, a target voxel semantic category to which the image information belongs, includes: Copying and extending the image features and point cloud features at the first viewing angle along the first direction to obtain corresponding first three-dimensional spatial features, copying and extending the image features at the second viewing angle along the second direction to obtain corresponding second three-dimensional spatial features, and copying and extending the point cloud features at the third viewing angle along the second direction to obtain corresponding third three-dimensional spatial features; fusing the first three-dimensional spatial feature, the second three-dimensional spatial feature, and the third three-dimensional spatial feature to obtain a fused target feature; The target feature is input into the target model to obtain multiple voxel semantic categories and a probability value of each voxel semantic category, and the target voxel semantic category is determined.

6. The method according to claim 1, characterized in that Before obtaining the target voxel semantic category to which the image information belongs based on the image features at the preset viewing angle, the point cloud features at the preset viewing angle, and the target model, the method further includes: Acquire training sample data, wherein the training sample data includes image training sample data and point cloud training sample data, and the training sample data carries point cloud semantic labels corresponding to the training sample data; Inputting the training sample data into an initial model to obtain initial voxel semantic categories; Comparing the initial voxel semantic category with the point cloud semantic label, adjusting initial weight parameters in the initial model until the initial voxel semantic category is consistent with the point cloud semantic label, obtaining intermediate weight parameters after adjusting the initial weight parameters, and determining an intermediate model; Acquire image fault data and point cloud fault data and add them to the training sample data to obtain new training sample data; Inputting the new training sample data into the intermediate model to obtain intermediate voxel semantic categories; The intermediate voxel semantic category is compared with the point cloud semantic label, and the intermediate weight parameter in the intermediate model is adjusted until the intermediate voxel semantic category is consistent with the point cloud semantic label, so as to obtain the target weight parameter after adjusting the intermediate weight parameter and determine the target model.

7. A voxel semantic category determination device, characterized in that: The device comprises: A first acquisition module is used to acquire image information, image pixel depth distribution of the image information, and point cloud data of the image information; a first obtaining module, configured to obtain voxel features based on the image information and the depth distribution of the image pixels, and obtain grid features of a three-dimensional grid based on the point cloud data, wherein the voxel features are pixel features having height information; A second acquisition module is configured to acquire image features at multiple preset viewing angles based on the voxel features, and acquire point cloud features at multiple preset viewing angles based on the grid features; acquiring image features at multiple preset viewing angles based on the voxel features, and acquiring point cloud features at multiple preset viewing angles based on the grid features, comprising: collapsing the voxel features in a first direction to acquire image features at the first viewing angle, and collapsing the voxel features in a second direction to acquire image features at the second viewing angle; collapsing the grid features in the first direction to acquire point cloud features at the first viewing angle, and collapsing the grid features in the second direction to acquire point cloud features at a third viewing angle; The second obtaining module is used to obtain the target voxel semantic category to which the image information belongs based on the image features at the preset perspective, the point cloud features at the preset perspective, and the target model, wherein the target model is used to determine the obstacle type to which the image information belongs.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the voxel semantic category determination method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the voxel semantic category determination method according to any one of claims 1 to 6.

Images