Spatial semantic occupancy label generation method and apparatus, electronic device, and storage medium

Abstract

The application discloses a kind of space semantic occupancy label generation method, device, electronic equipment and storage medium, by obtaining the environment point cloud data and environment image corresponding to vehicle operating environment;The object in the vehicle operating environment is predicted based on the environment point cloud data and the environment image category, obtains the pseudo label corresponding to the environment point cloud data, by the pseudo label, the initial sparse label corresponding to environment point cloud data is carried out dense processing, obtains the space semantic occupancy label corresponding to the vehicle operating environment.Through pseudo label, the initial sparse label corresponding to environment point cloud data is carried out dense processing, to reduce the sparsity of initial sparse label, obtain more dense more accurate space semantic occupancy label.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a spatial semantic occupancy label generation method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Generally, the method of autonomous driving perception can describe the three-dimensional spatial structure of the surrounding environment using voxel representation to achieve semantic scene understanding, and assist the autonomous driving system to have a fine-grained understanding of the surfaces and objects in its vicinity.

[0003] In the related art, when describing a three-dimensional scene using voxel representation, an annotator needs to perform manual annotation of semantic information based on point cloud data of the surrounding environment. However, the density of the label obtained by the point cloud data annotation method in the related art needs to be improved. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, the present application provides a spatial semantic occupancy label generation method, device, electronic equipment and storage medium.

[0005] The present application provides a spatial semantic occupancy label generation method, characterized in that the method comprises:

[0006] Obtaining environment point cloud data and environment images corresponding to a vehicle operating environment; wherein the environment point cloud data corresponds to an initial sparse label; the initial sparse label is used to describe the category information of an object in the vehicle operating environment;

[0007] Performing category prediction on the object in the vehicle operating environment based on the environment point cloud data and the environment images, to obtain prediction category data as a pseudo label corresponding to the environment point cloud data;

[0008] Performing dense processing on the initial sparse label through the pseudo label to obtain a spatial semantic occupancy label corresponding to the vehicle operating environment.

[0009] In one embodiment, the performing category prediction on the object in the vehicle operating environment based on the environment point cloud data and the environment images to obtain prediction category data comprises:

[0010] Performing convolution processing on the environment point cloud data to obtain three-dimensional point cloud features corresponding to the environment point cloud data;

[0011] Performing feature extraction and projection on the environment images to obtain three-dimensional image features corresponding to the environment images;

[0012] perform category prediction on the object in the vehicle operating environment based on the three-dimensional point cloud feature and the three-dimensional image feature, to obtain the prediction category data.

[0013] In one of the embodiments, the convolution processing based on the environment point cloud data to obtain the three-dimensional point cloud feature corresponding to the environment point cloud data comprises:

[0014] voxelizing the environment point cloud data to obtain environment voxel data;

[0015] extracting features from the environment voxel data through three-dimensional sparse convolution to obtain the three-dimensional point cloud feature.

[0016] In one of the embodiments, the feature extraction and projection based on the environment image to obtain the three-dimensional image feature corresponding to the environment image comprises:

[0017] extracting features from the environment image to obtain a two-dimensional image feature of the environment image;

[0018] projecting the two-dimensional image feature to a three-dimensional space based on a two-dimensional depth map corresponding to the two-dimensional image feature and internal and external parameters of a shooting device shooting the environment image, to obtain the three-dimensional image feature.

[0019] In one of the embodiments, the category prediction on the object in the vehicle operating environment based on the three-dimensional point cloud feature and the three-dimensional image feature to obtain the prediction category data comprises:

[0020] splicing the three-dimensional point cloud feature and the three-dimensional image feature to obtain a three-dimensional spliced feature;

[0021] performing convolution processing on the three-dimensional spliced feature to obtain a corresponding three-dimensional fusion feature;

[0022] performing category prediction on the object in the vehicle operating environment based on the three-dimensional fusion feature to obtain the prediction category data.

[0023] In one of the embodiments, the dense processing of the initial sparse label based on the pseudo label to obtain the spatial semantic occupancy label corresponding to the vehicle operating environment comprises:

[0024] complementarily processing the initial sparse label based on the pseudo label to obtain a dense processed to-be-screened label corresponding to the environment point cloud data;

[0025] determining a to-be-screened label to be reserved based on the to-be-screened label projected into the environment image;

[0026] Determine the reserved to-be-screened label based on the to-be-screened label.

[0027] In one of the embodiments, the determining the reserved to-be-screened label based on the to-be-screened label projected into the environment image comprises:

[0028] In response to the selection operation of any category of to-be-screened label, project the selected any category of to-be-screened label into the environment image.

[0029] In the case of receiving the reservation operation for the any category of to-be-screened label, determine the any category of to-be-screened label as the reserved to-be-screened label.

[0030] In one of the embodiments, the environment point cloud data corresponds to a three-dimensional bounding box and object semantic information of an object in the vehicle operating environment; and the generation manner of the initial sparse label comprises:

[0031] Determine dynamic point cloud data and static point cloud data in the environment point cloud data according to the three-dimensional bounding box and the object semantic information;

[0032] Perform a projection operation based on the static point cloud data to obtain fused static point cloud data;

[0033] Perform fusion processing on the dynamic point cloud data according to the position information of the three-dimensional bounding box to obtain fused dynamic point cloud data;

[0034] Superimpose the fused static point cloud data and the fused dynamic point cloud data to obtain superimposed point cloud data;

[0035] Perform voxelization operation on the superimposed point cloud data to obtain the initial sparse label corresponding to the environment point cloud data.

[0036] In one of the embodiments, the prediction category data is obtained by a target object classification model; and the training process of the object classification model comprises:

[0037] Obtain an environment point cloud data sample set; wherein the environment point cloud data sample set comprises a plurality of environment point cloud data samples, and each environment point cloud data sample corresponds to an initial sparse label and an environment image sample;

[0038] Input the environment point cloud data sample and the environment image sample into an object classification model to perform category prediction, and obtain the category data corresponding to the environment point cloud data sample;

[0039] Update the object classification model according to the category data corresponding to the environment point cloud data sample and the initial sparse label to obtain the target object classification model.

[0040] The application provides a spatial semantic occupancy label generation device, the device comprising:

[0041] A data acquisition module is configured to acquire environment point cloud data and an environment image corresponding to a vehicle operating environment, wherein the environment point cloud data corresponds to an initial sparse label, and the initial sparse label is used to describe category information of an object in the vehicle operating environment.

[0042] A category prediction module is configured to perform category prediction on the object in the vehicle operating environment based on the environment point cloud data and the environment image, to obtain prediction category data as a pseudo label corresponding to the environment point cloud data.

[0043] A dense processing module is configured to perform dense processing on the initial sparse label based on the pseudo label, to obtain a spatial semantic occupancy label corresponding to the vehicle operating environment.

[0044] The application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in any of the embodiments when executing the computer program.

[0045] The application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the method described in any of the embodiments when executed by a processor.

[0046] The spatial semantic occupancy label generation method comprises the following steps: acquiring environment point cloud data and an environment image corresponding to a vehicle operating environment; performing category prediction on an object in the vehicle operating environment based on the environment point cloud data and the environment image, to obtain a pseudo label corresponding to the environment point cloud data; and performing dense processing on an initial sparse label corresponding to the environment point cloud data based on the pseudo label, to obtain a spatial semantic occupancy label corresponding to the vehicle operating environment. The initial sparse label corresponding to the environment point cloud data is processed based on the pseudo label, so as to reduce the sparsity of the initial sparse label, and a more dense and accurate spatial semantic occupancy label is obtained.

[0047] Additional aspects and advantages of the application will be described in the following description and will become apparent from the following detailed description or will be learned from the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1a A plan view of a scene is provided by the bird's eye view representation method for the embodiments of the present description.

[0049] Figure 1b A three-dimensional spatial structure of a scene is provided by the voxel representation method for the embodiments of the present description.

[0050] Figure 1c Flowchart of the initial sparse label annotation process provided for the embodiments of the present description.

[0051] Figure 2a Flowchart of the spatial semantic occupancy label generation method provided for the embodiments of the present description.

[0052] Figure 2b Visualized processed initial sparse label provided for the embodiments of the present description.

[0053] Figure 2c Visualized processed spatial semantic occupancy label provided for the embodiments of the present description.

[0054] Figure 3a Flowchart of the spatial semantic occupancy label generation method provided for the embodiments of the present description;

[0055] Figure 3b Structure diagram of the object classification model provided for the embodiments of the present description.

[0056] Figure 4 Flowchart of the spatial semantic occupancy label generation method provided for the embodiments of the present description.

[0057] Figure 5 Flowchart of the spatial semantic occupancy label generation method provided for the embodiments of the present description.

[0058] Figure 6 Flowchart of the spatial semantic occupancy label generation method provided for the embodiments of the present description.

[0059] Figure 7a Flowchart of the spatial semantic occupancy label generation method provided for the embodiments of the present description.

[0060] Figure 7b Schematic diagram of the to-be-screened label projected to the environment image provided for the embodiments of the present description.

[0061] Figure 7c Schematic diagram of the to-be-screened label projected to the environment image provided for the embodiments of the present description.

[0062] Figure 7d Schematic diagram of the to-be-screened label projected to the environment image provided for the embodiments of the present description.

[0063] Figure 7e Schematic diagram of the spatial semantic occupancy label generation method provided for the embodiments of the present description.

[0064] Figure 7fObject classes and color information that need to be calibrated in an environmental image provided for an embodiment of the present specification.

[0065] Figure 7g A schematic diagram of a pseudo label and an initial label that reserve a sidewalk class provided for an embodiment of the present specification.

[0066] Figure 8 A flowchart of a generation manner of an initial sparse label provided for an embodiment of the present specification.

[0067] Figure 9 A flowchart of a training manner of an object classification model provided for an embodiment of the present specification.

[0068] Figure 10 A structural schematic diagram of a spatial semantic occupancy label generation apparatus provided for an embodiment of the present specification.

[0069] Figure 11 A structural schematic diagram of an electronic device provided for an embodiment of the present specification. DETAILED DESCRIPTION

[0070] Embodiments of the present application are described in detail below with reference to the attached drawings, which show by way of example, embodiments in which the same or similar elements have the same or similar reference numbers and in which:

[0071] In the related art, an autonomous driving system perceives the surrounding environment through a perception module, and a method of autonomous driving perception can adopt a bird's eye view representation method. Please refer to Figure 1a , which describes the planar information of a scene through the bird's eye view representation method. It is obvious that although the bird's eye view representation method is relatively efficient, the bird's eye view presents a single plane and is difficult to describe the three-dimensional structure in the surrounding environment.

[0072] In addition, the method of autonomous driving perception can also describe the surrounding environment in a voxel representation method. Please refer to Figure 1b , which describes the three-dimensional spatial structure of a scene through the voxel representation method. The voxel representation method can provide more fine and accurate spatial information of the road surface structure, pedestrians, vehicles, etc. for the autonomous driving perception algorithm.

[0073] Please refer to Figure 1cFor the autonomous driving scene, the SemanticKITTI dataset (ADataset for Semantic Scene Understanding of LiDAR Sequences) can be used to describe the evaluation benchmark of the three-dimensional scene in the voxel representation method. The specific process includes the following steps:

[0074] S102, the input multi-frame point cloud is determined as static point cloud and dynamic point cloud.

[0075] Specifically, the input multi-frame point cloud is determined as static point cloud (road surface, trees, etc.) and dynamic point cloud (car, pedestrian, etc.) by a human annotator.

[0076] S104, the static point cloud is fused in multiple frames.

[0077] Specifically, assuming that there are N frames of continuous point clouds, labeled as 1-N, the point clouds of frames 2-N are projected into the world coordinate system of the first frame by using the extrinsic matrix.

[0078] S106, the dynamic point cloud is fused in multiple frames.

[0079] Specifically, for the dynamic point cloud, the point cloud of each object in frames 1-N is determined by a human calibrator, and then the point cloud of each object is gradually fused.

[0080] S108, the semantic information of the point cloud is labeled.

[0081] Specifically, in the multi-frame fused point cloud, the human annotator labels the semantic information (i.e. the category of the object) of the point cloud.

[0082] S110, voxel processing the labeled point cloud.

[0083] Specifically, the initial structure of the point cloud is (M, 4), where M is the number of point clouds, and 4 is equal to the 3D coordinates and 1D semantic information of the point cloud. After voxel processing, the structure of the small block obtained is (H, W, D, 1), that is, (H, W, D) is the length, width and height of the cubic structure, and each small block in the cubic structure retains 1D semantic information. The semantic information in the small block may be the object category, or it may be "empty", that is, there is no object occupying the space where the small block is located. The cubic voxel structure and the semantic information corresponding to the cubic voxel structure can generate a corresponding initial label. It should be noted that the initial label has a certain sparseness, which can also be called an initial sparse label.

[0084] By analyzing the labeling process of the initial sparse label described above, it can be found that, on the one hand, the SemanticKITTI dataset labels 22 segments, about 9000 frames, which requires a larger amount of labeling for model training in large-scale neural network training. On the other hand, the labeling strategy described above relies on the fusion of point clouds, such as extrinsic parameter projection or manual alignment, but since the point cloud itself is sparse, this will cause a large number of empty areas, such as a space area that is originally occupied by an object, but since the point cloud does not shoot into the space area, the space area is misjudged as "empty", that is, the initial sparse label needs to be densified to improve the label accuracy.

[0085] Based on this, the embodiment of the present specification provides a way of densifying the initial sparse label corresponding to the environment point cloud data by the pseudo label, to label the environment point cloud data. Specifically, the environment point cloud data and the environment image corresponding to the vehicle operating environment are obtained; the object in the vehicle operating environment is classified and predicted based on the environment point cloud data and the environment image, to obtain the pseudo label corresponding to the environment point cloud data, the initial sparse label corresponding to the environment point cloud data is densified by the pseudo label, to obtain the space semantic occupancy label corresponding to the vehicle operating environment. Not only can the labeling work of the labeler be reduced, the labeling efficiency can be improved, but also the sparsity of the initial sparse label can be reduced, to obtain a more dense and accurate space semantic occupancy label.

[0086] It should be noted that the vehicle operating environment mentioned in the embodiment of the present specification can be a 360 ° surrounding environment of the vehicle, which can be the front field of view environment of the vehicle, the environment behind the vehicle, or the environment of a specified area (such as a blind area) around the vehicle, and the embodiment of the present specification does not limit it. The vehicle or other electronic devices can implement the space semantic occupancy label generation method through the components (including hardware and software) contained therein.

[0087] Please refer to Figure 2a The embodiment of the present specification provides a space semantic occupancy label generation method, which can include the following steps:

[0088] S210, obtaining the environment point cloud data and the environment image corresponding to the vehicle operating environment.

[0089] The vehicle operating environment can be the surrounding environment of the vehicle when driving, and the object in the vehicle operating environment can include at least one of the road surface, pedestrians, green plants, other vehicles, etc. The environment point cloud data corresponds to an initial sparse label. The initial sparse label is used to describe the class information of the object in the vehicle operating environment.

[0090] Specifically, in some embodiments, the vehicle can be configured with a perception module, and sensors of the perception module can include at least one of a camera, a Lidar, and a millimeter-wave radar. An environment image corresponding to the vehicle operating environment can be obtained through the camera. Environment point cloud data corresponding to the vehicle operating environment can be obtained through the Lidar. It is easy to understand that the environment image corresponds to a two-dimensional space, and the environment point cloud data corresponds to a three-dimensional space. In some embodiments, the environment point cloud data and the environment image corresponding to the vehicle operating environment can be obtained from some public data sets, such as the Panoptic nuScenes data set (A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking, a large-scale benchmark for LiDAR panoptic segmentation and tracking). The training set and the test set provided by the Panoptic nuScenes data set have 850 scene segments, a total of 35,000 frames, each frame is annotated with semantic information, and each object has a 3D annotation box.

[0091] In S220, the objects in the vehicle operating environment are classified and predicted based on the environment point cloud data and the environment image, and prediction class data is obtained as pseudo-labels corresponding to the environment point cloud data.

[0092] In some cases, due to the sparsity of the environment point cloud data, the initial sparse label generated by annotating the environment point cloud data is also sparse to a certain extent. Therefore, in order to improve the accuracy of the label, it is necessary to densify the initial sparse label corresponding to the environment point cloud data. Before the densification processing, the pseudo-labels used for densification processing of the initial sparse label need to be determined.

[0093] Specifically, in some embodiments, the features of the objects in the vehicle operating environment are contained in the environment point cloud data and the environment image corresponding to the vehicle operating environment. Therefore, on the one hand, the environment point cloud data corresponding to the vehicle operating environment can be feature-extracted, and the first prediction data can be obtained based on the extracted features. On the other hand, the second prediction data can also be obtained based on the environment image corresponding to the vehicle operating environment. Finally, by comprehensively considering the first prediction data and the second prediction data, the prediction class data that can be used as pseudo-labels can be obtained according to the corresponding confidence or a preset processing rule.

[0094] In yet some embodiments, since the environment point cloud data corresponding to the vehicle running environment is three-dimensional data, the feature extraction on the environment point cloud data corresponding to the vehicle running environment can obtain three-dimensional data features. The environment image corresponding to the vehicle running environment is two-dimensional data, and thus the two-dimensional data can be converted into corresponding three-dimensional space data. The feature extraction on the converted three-dimensional space data can also obtain three-dimensional data features. The three-dimensional data features corresponding to the environment point cloud data and the three-dimensional data features corresponding to the environment image are fused, and the fused three-dimensional data features are used for prediction to obtain the predicted category data as the pseudo label.

[0095] In S230, the initial sparse label is densely processed by the pseudo label to obtain the spatial semantic occupancy label corresponding to the vehicle running environment.

[0096] In some cases, please refer to Figure 2b , Figure 2b The initial sparse label after the visualization processing is shown. The visualized initial sparse label has some holes 202, and it can be seen that the initial sparse label has a certain sparsity, and thus the initial sparse label needs to be densely processed by the pseudo label. Therefore, the pseudo label supplements the initial sparse label, and the processed initial sparse label is the spatial semantic occupancy label, which realizes the densification of the initial sparse label. Please refer to Figure 2c , Figure 2c The spatial semantic occupancy label after the visualization processing is shown, and it can be seen that the pseudo label has filled the holes in the initial sparse label.

[0097] Specifically, the pseudo label corresponding to the environment point cloud data is obtained by performing category prediction on the objects in the vehicle running environment. For any category of object in the vehicle running environment, the any category of object corresponds to a pseudo label and an initial sparse label. The holes in the initial sparse label can be filled by the pseudo label corresponding to the any category of object to densely process the initial sparse label, and thus the spatial semantic occupancy label corresponding to the vehicle running environment is obtained.

[0098] Exemplarily, the square in the cuboid structure obtained after the voxelization processing corresponds to one-dimensional semantic information, and the semantic information can be an object category or empty. For any square in the cuboid structure, the corresponding initialized label can be an object category or empty, and the corresponding pseudo label can also be an object category or empty. The pseudo label corresponds to a confidence, and the pseudo label with a confidence higher than a preset confidence threshold is used to perform dense processing on the initial sparse label. The preset confidence threshold can be set according to actual conditions, for example, the preset confidence threshold can be set to 0.8. Exemplarily, a method of performing dense processing on the initial sparse label by the pseudo label is described. For any square, if the corresponding initialized label of the square is empty and the corresponding pseudo label of the square is an object category A, it is determined that the object category of the square is A; if the corresponding initialized label of the square is an object category A and the corresponding pseudo label of the square is an object category B, it is determined that the object category of the square is A; if the corresponding initialized label of the square is an object category A and the corresponding pseudo label of the square is empty, it is determined that the object category of the square is A.

[0099] It should be noted that in some cases, since the pseudo label is obtained by machine learning algorithm, the pseudo label will have certain noise, and therefore, when the object category of the initialized label of any square is different from the object category of the pseudo label of the square, and the object category of the initialized label of any square is not empty, the object category of the initialized label of the square is used as the criterion, and the object category of the initialized label of the square is not changed. When the object category of the initialized label of any square is empty, in order to fill the hole corresponding to the initialized label, the object category of the pseudo label of the square is used to fill the hole corresponding to the square.

[0100] In the above method for generating spatial semantic occupancy label, the environment point cloud data and the environment image corresponding to the vehicle operating environment are obtained; the object in the vehicle operating environment is classified and predicted based on the environment point cloud data and the environment image, to obtain a pseudo label corresponding to the environment point cloud data, and the initial sparse label corresponding to the environment point cloud data is processed densely by the pseudo label, to obtain a spatial semantic occupancy label corresponding to the vehicle operating environment. Not only can the labeling work of the labelers be reduced and the labeling efficiency be improved, but also the sparsity of the initial sparse label can be reduced, to obtain a more dense and more accurate spatial semantic occupancy label.

[0101] In some embodiments, referring to Figure 3a , the object in the vehicle operating environment is classified and predicted based on the environment point cloud data and the environment image, to obtain prediction category data, which can include the following steps:

[0102] S310, performing three-dimensional convolution processing according to the environment point cloud data, to obtain three-dimensional point cloud features corresponding to the environment point cloud data.

[0103] The convolution processing is used to extract useful features corresponding to the vehicle operating environment from the environment point cloud data. Specifically, sparse convolution processing can be performed on the environment point cloud data using a three-dimensional convolution kernel to extract three-dimensional point cloud features corresponding to the environment point cloud data. Dense convolution processing can also be performed on the environment point cloud data using a three-dimensional convolution kernel to extract three-dimensional point cloud features corresponding to the environment point cloud data. Voxelization processing can also be performed on the environment point cloud data, and convolution processing can be performed on the voxelized environment point cloud data to extract three-dimensional point cloud features corresponding to the environment point cloud data.

[0104] S320, feature extraction and projection are performed according to the environment image to obtain three-dimensional image features corresponding to the environment image.

[0105] Specifically, since the environment image is a two-dimensional image and lacks depth information, in order to extract corresponding three-dimensional image features from the environment image, first, feature extraction is performed from the environment image, and the extracted image features are projected into a three-dimensional space to obtain three-dimensional image features corresponding to the environment image.

[0106] S330, class prediction is performed on the object in the vehicle operating environment based on the three-dimensional point cloud features and the three-dimensional image features to obtain prediction class data.

[0107] Specifically, the three-dimensional point cloud features and the three-dimensional image features are fused to obtain fused three-dimensional features, and the fused three-dimensional features are used to perform class prediction on the object in the vehicle operating environment to obtain prediction class data that can be used as pseudo labels.

[0108] Exemplarily, the environment point cloud data and the vehicle surround view image are classified by an object classification model. The vehicle surround view image can be an environment image obtained by image acquisition of 360 degrees of the vehicle. Please refer to Figure 3b , Figure 3b The model structure of the object classification model is schematically shown. The object classification model includes a first branch 302, a second branch 304, a feature fusion module 306 connected with the first branch 302 and the second branch 304 respectively, a three-dimensional convolution module 308 connected with the feature fusion module 306, and a classifier 310 connected with the three-dimensional convolution module 308. The first branch 302 includes a voxelization module and a sparse convolution module, and the second branch 304 includes a feature extraction module and a feature projection module. The convolution module of the first branch 302 and the feature projection module of the second branch 304 are connected with the feature fusion module 306 respectively.

[0109] Please continue to refer to Figure 3b, input the environment point cloud data (N, 3) into the object classification model, voxelize the environment point cloud data (N, 3) through a voxelization module to obtain environment voxel data (512, 512, 40, C1), and perform convolution processing on the environment voxel data (512, 512, 40, C1) through a sparse convolution module to obtain corresponding three-dimensional voxel features (128, 128, 10, C2). Please continue to refer to Figure 3b , input the vehicle surround view image (6, H, W, 3) into the object classification model, perform two-dimensional feature processing on the vehicle surround view image (6, H, W, 3) through a feature extraction module to obtain surround view image features (6, H / 8, W / 8, 3), and perform projection processing on the surround view image features through a feature projection module to obtain corresponding three-dimensional image features (128, 128, 10, C2).

[0110] Please continue to refer to Figure 3b , the three-dimensional voxel features (128, 128, 10, C2) and the three-dimensional image features (128, 128, 10, C2) are transmitted to the feature fusion module 306 for feature fusion to obtain fused voxel features (128, 128, 10, 2*C2). The fused voxel features (128, 128, 10, 2*C2) are input into the feature extraction module 308 for feature extraction to obtain voxel features (128, 128, 10, C4). The voxel features (128, 128, 10, C4) are input into the classifier to obtain predicted class data (128, 128, 10, 17). It can be understood that the classifier is essentially a 3D convolution, which compresses the feature dimension to 17 through the classifier, and 17 can be understood as the number of categories to be classified.

[0111] In the above implementation, the three-dimensional point cloud features corresponding to the environment point cloud data and the three-dimensional image features corresponding to the environment image are fused, and the objects in the vehicle operating environment are classified based on the fused three-dimensional features to obtain predicted class data, which prepares data basis for filling the initialized label.

[0112] In some embodiments, please refer to Figure 4 , the three-dimensional point cloud features corresponding to the environment point cloud data can include the following steps:

[0113] S410, voxelizing the environment point cloud data to obtain environment voxel data.

[0114] S420, performing feature extraction on the environment voxel data through three-dimensional sparse convolution to obtain three-dimensional point cloud features.

[0115] Specifically, the environmental point cloud data can be divided into a plurality of voxel blocks in the form of three-dimensional matrices in combination with the three-dimensional coordinates corresponding to the environmental point cloud data, so as to filter the point cloud data, realize preprocessing of the environmental point cloud data, and maintain the integrity of the environmental point cloud data. Through voxelization processing of the environmental point cloud data, environmental voxel data is obtained. Three-dimensional voxel features are obtained by performing convolution operation on the environmental voxel data by using three-dimensional sparse convolution, as the three-dimensional point cloud features corresponding to the environmental point cloud data.

[0116] In the above embodiments, the environmental point cloud data is voxelized to obtain environmental voxel data, so as to realize preprocessing of the environmental point cloud data, and further to perform convolution processing on the environmental voxel data to obtain three-dimensional point cloud features that can accurately describe the environmental point cloud data.

[0117] In some embodiments, referring to Figure 5 According to the feature extraction and projection of the environmental image, the three-dimensional image features corresponding to the environmental image can include the following steps:

[0118] S510, feature extraction is performed on the environmental image to obtain two-dimensional image features of the environmental image.

[0119] S520, according to the two-dimensional depth map corresponding to the two-dimensional image features and the internal and external parameters of the shooting device for shooting the environmental image, the two-dimensional image features are projected into a three-dimensional space to obtain three-dimensional image features.

[0120] In the above embodiments, the environmental point cloud data is voxelized to obtain environmental voxel data, so as to realize preprocessing of the environmental point cloud data, and further to perform convolution processing on the environmental voxel data to obtain three-dimensional point cloud features that can accurately describe the environmental point cloud data.

[0121] Further, the shooting device can be calibrated in advance to obtain the internal and external parameters of the shooting device. According to the two-dimensional depth map and the internal and external parameters of the shooting device, the two-dimensional image features are projected into a three-dimensional space (3D space) to obtain three-dimensional image features.

[0122] In the above implementation, by obtaining the three-dimensional image features corresponding to the environment image based on the environment image, the feature data dimensions of the environment point cloud data and the environment image are consistent, the feature fusion of the environment point cloud data and the environment image can be realized, the accuracy of the pseudo label is improved, and the densification result of the pseudo label on the initial sparse label is improved.

[0123] In some embodiments, referring to Figure 6 , the class prediction on the object in the vehicle operating environment based on the three-dimensional point cloud features and the three-dimensional image features to obtain the predicted class data can include the following steps:

[0124] S610, splicing the three-dimensional point cloud features and the three-dimensional image features to obtain three-dimensional splicing features.

[0125] S620, performing convolution processing on the three-dimensional splicing features to obtain corresponding three-dimensional fusion features.

[0126] S630, performing class prediction on the object in the vehicle operating environment based on the three-dimensional fusion features to obtain predicted class data.

[0127] As described above, by performing convolution processing based on the environment point cloud data, the three-dimensional point cloud features (H, W, D, C1) corresponding to the environment point cloud data have been obtained, and by performing feature extraction and projection based on the environment image, the three-dimensional image features (H, W, D, C2) corresponding to the environment image have been obtained. In order to improve the accuracy of the pseudo label, the three-dimensional point cloud features (H, W, D, C1) corresponding to the environment point cloud data are fused with the three-dimensional image features (H, W, D, C2) corresponding to the environment image. Specifically, the three-dimensional point cloud features (H, W, D, C1) and the three-dimensional image features (H, W, D, C2) are spliced at the feature level to obtain three-dimensional fusion features (H, W, D, C1+C2). The three-dimensional convolution kernel is used to extract the fusion features of the three-dimensional fusion features (H, W, D, C1+C2), and the extracted fusion features are input into the classifier for class prediction to obtain the predicted class data (H, W, D, N+1). Wherein, N is the number of classes of objects in the vehicle surrounding environment, and 1 corresponds to the label of "empty".

[0128] In the above implementation, the accuracy of the pseudo label is improved by feature fusion of the environment point cloud data and the environment image, and the densification result of the pseudo label on the initial sparse label is further improved.

[0129] In some embodiments, referring to Figure 7a , the initial sparse label is densified by the pseudo label to obtain the spatial semantic occupancy label corresponding to the vehicle operating environment, which can include the following steps:

[0130] S710, supplementing the initial sparse label by the pseudo label to obtain a dense to-be-screened label corresponding to the environment point cloud data.

[0131] S720, determining a retained to-be-screened label according to the to-be-screened label projected into the environment image.

[0132] S730, determining a spatial semantic occupancy label based on the retained to-be-screened label.

[0133] In some cases, since the pseudo label is obtained by prediction based on the environment point cloud data and the environment image, there may be some noise, and therefore the spatial semantic occupancy label obtained by using the pseudo label to perform dense processing on the initial sparse label needs to be further optimized.

[0134] Specifically, by fusing the pseudo label with the initial sparse label, a more dense to-be-screened label can be obtained. In order to improve the accuracy of the spatial semantic occupancy label, the to-be-screened label needs to be further screened. Therefore, please refer to Figure 7b , Figure 7c , Figure 7d projecting the to-be-screened label into the environment image, and determining an accurate label in the to-be-screened label based on the projection, that is, the label that can be retained, to obtain a retained to-be-screened label. The final retained to-be-screened label is taken as the spatial semantic occupancy label. Exemplarily, please refer to the spatial semantic occupancy label result diagram shown in Figure 7e , Figure 7e

[0135] It should be noted that different colors can be used to represent different object categories in the environment image. Exemplarily, different object categories are represented by different RGB values. The object categories existing in the environment image include at least one of noise, obstacle, bicycle, bus, car, building, motorcycle, pedestrian, traffic barrier, trailer, truck, drivable area, other plane, sidewalk, terrain, man-made object, and green plant. Exemplarily, please refer to the object categories and color information to be labeled in the environment image shown in Figure 7f , Figure 7f

[0136] In the above embodiments, by screening the dense to-be-screened label corresponding to the environment point cloud data, accurate labels are retained to determine the spatial semantic occupancy label. On the basis of densifying the initial sparse label, the accuracy of the label is further improved, and more accurate spatial information such as road surface structure, pedestrian, and vehicle is provided for the automatic driving perception algorithm.

[0137] ​​In some embodiments, determining the reserved screening label according to the screening label projected into the environment image comprises: in response to a selection operation on any category of screening label, projecting the selected any category of screening label into the environment image; in the case that a reservation operation is received for any category of screening label, determining the any category of screening label as the reserved screening label.

[0138] In order to reduce the workload of manual labeling, in the present embodiment, it can be determined whether the screening label needs to be reserved or deleted category by category. Specifically, for any category of screening label, different object categories can be displayed on the labeling operation interface, and whether to display control is provided at the corresponding position of the different object categories. For example, if the whether to display control of any object category is in an un-displayable state, in response to a click operation on the whether to display control, the selected screening label of the any object category is selected as a screening label that needs to be projected into the environment image, the selected screening label of the any category is projected into the environment image, and if the quality of the screening label of the any category meets the preset quality requirement, the screening label of the any category is reserved. For example, please refer to FIGS. 8A and 8B, which show the reserved pseudo-label and initial label of the sidewalk category. Figure 7g , Figure 7g

[0139] In the above embodiment, by projecting the screening label of a single category onto the surround view image one by one, the visual auxiliary calibration of the label is realized, the screening label is intuitively displayed, and the labeling personnel can easily audit the displayed screening label to determine whether to reserve the displayed screening label, thereby reducing the difficulty of labeling work and improving the labeling efficiency.

[0140] In some embodiments, the environment point cloud data corresponds to a three-dimensional labeling box and object semantic information of an object in a vehicle operating environment; please refer to FIG. 8A, which shows the generation of an initial sparse label. Figure 8 The generation method of the initial sparse label can include the following steps:

[0141] S810, determining dynamic point cloud data and static point cloud data in the environment point cloud data according to the three-dimensional labeling box and the object semantic information.

[0142] S820, performing a projection operation based on the static point cloud data to obtain fused static point cloud data.

[0143] S830, performing fusion processing on the dynamic point cloud data according to the position information of the three-dimensional labeling box to obtain fused dynamic point cloud data.

[0144] S840, superimposing the fused static point cloud data and the fused dynamic point cloud data to obtain superimposed point cloud data.

[0145] ​S850, voxelizing the superimposed point cloud data to obtain initial sparse labels corresponding to the environment point cloud data.

[0146] Specifically, the Panoptic nuScenes dataset includes multi-frame environment point cloud data. Each frame of environment point cloud data is annotated with object semantic information, and each object corresponds to a 3D annotation box. The object semantic information can describe the category of the object in the vehicle operating environment. Since objects of different categories have different motion states, the object semantic information can be used to determine whether the point cloud data corresponding to the object in the vehicle operating environment is static point cloud data or dynamic point cloud data. Further, the position information of the three-dimensional annotation box can also be used to determine whether the object in the vehicle operating environment is in a stationary state or a motion state. For example, the point cloud data of any object in the categories of bicycle, bus, car, motorcycle, pedestrian, trailer, and truck is dynamic point cloud data. The point cloud data of any object in the categories of building, traffic barrier, drivable area, other plane, sidewalk, artificial object, and greenery is static point cloud data. For multi-frame static point cloud data, the other frames of static point cloud data except the current frame can be projected into the current frame according to the extrinsic parameters of the point cloud, and the multi-frame static point cloud data can be fused to obtain fused static point cloud data. For multi-frame dynamic point cloud data, the multi-frame dynamic point cloud data can be fused according to the position information of the three-dimensional annotation box to obtain fused dynamic point cloud data. Or, the dynamic point cloud data can be projected into the corresponding 3D annotation box and fused. Further, the fused static point cloud data and the fused dynamic point cloud data are superimposed to obtain superimposed point cloud data. Finally, the superimposed point cloud data is voxelized to obtain initial sparse labels corresponding to the environment point cloud data.

[0147] In the above embodiments, the initial sparse labels corresponding to the environment point cloud data are generated to provide data preparation for finally generating spatial semantic occupancy labels.

[0148] In some embodiments, the predicted category data is obtained by predicting through a target classification model. Please refer to Figure 9 The training process of the object classification model can include the following steps:

[0149] S910, obtaining an environment point cloud data sample set.

[0150] The environment point cloud data sample set includes a plurality of environment point cloud data samples, and each environment point cloud data sample corresponds to an initial sparse label and an environment image sample. Specifically, the environment point cloud data and the environment image corresponding to the vehicle operating environment can be obtained from some public datasets to construct the environment point cloud data sample set. The environment point cloud data and the environment image corresponding to the vehicle operating environment can also be obtained by data acquisition through a perception device to construct the environment point cloud data sample set.

[0151] S920, input the environment point cloud data sample and the environment image sample to the object classification model for class prediction to obtain class data corresponding to the environment point cloud data sample.

[0152] The object classification model comprises a first branch, a second branch, a feature fusion module connected with the first branch and the second branch respectively, a three-dimensional convolution module connected with the feature fusion module, and a classifier connected with the three-dimensional convolution module.

[0153] Specifically, the first branch is used for feature extraction and projection of the environment image sample to obtain three-dimensional image features corresponding to the environment image sample. The second branch is used for three-dimensional convolution processing of the environment point cloud data sample to obtain three-dimensional point cloud features corresponding to the environment point cloud data sample. The feature fusion module is used for feature fusion of the three-dimensional image features corresponding to the environment image sample and the three-dimensional point cloud features corresponding to the environment point cloud data sample to obtain fused voxel features. The fused voxel features are input into the classifier to obtain class data corresponding to the environment point cloud data sample.

[0154] S930, updating the object classification model according to the class data corresponding to the environment point cloud data sample and the initial sparse label to obtain a target object classification model.

[0155] Specifically, loss data is determined according to the class data corresponding to the environment point cloud data sample and the initial sparse label, and the object classification model is updated in parameters according to the loss data to obtain an updated object classification model. The model training process is repeatedly performed until a model stopping condition is met to obtain the target object classification model.

[0156] In some embodiments, the present specification provides a spatial semantic occupancy label generation method, which can comprise the following steps:

[0157] S1002, obtaining environment point cloud data and environment images corresponding to a vehicle operating environment.

[0158] The environment point cloud data correspond to three-dimensional annotation boxes and object semantic information of objects in the vehicle operating environment.

[0159] S1004, determining dynamic point cloud data and static point cloud data in the environment point cloud data according to the three-dimensional annotation boxes and the object semantic information.

[0160] S1006, performing a projection operation based on the static point cloud data to obtain fused static point cloud data.

[0161] S1008, fuse the dynamic point cloud data according to the position information of the three-dimensional marking frame to obtain fused dynamic point cloud data.

[0162] S1010, superimpose the fused static point cloud data and the fused dynamic point cloud data to obtain superimposed point cloud data.

[0163] S1012, voxelize the superimposed point cloud data to obtain an initial sparse label corresponding to the environmental point cloud data.

[0164] The initial sparse label is used to describe the category information of the object in the vehicle operating environment.

[0165] S1014, perform convolution processing on the environmental point cloud data to obtain a three-dimensional point cloud feature corresponding to the environmental point cloud data.

[0166] Specifically, voxelize the environmental point cloud data to obtain environmental voxel data; and perform feature extraction on the environmental voxel data by three-dimensional sparse convolution to obtain the three-dimensional point cloud feature.

[0167] S1016, perform feature extraction and projection on the environmental image to obtain a three-dimensional image feature corresponding to the environmental image.

[0168] Specifically, perform feature extraction on the environmental image to obtain a two-dimensional image feature of the environmental image; project the two-dimensional image feature to a three-dimensional space according to a two-dimensional depth map corresponding to the two-dimensional image feature and internal and external parameters of a shooting device shooting the environmental image to obtain the three-dimensional image feature.

[0169] S1018, perform category prediction on the object in the vehicle operating environment based on the three-dimensional point cloud feature and the three-dimensional image feature to obtain prediction category data as a pseudo label corresponding to the environmental point cloud data.

[0170] Specifically, splice the three-dimensional point cloud feature and the three-dimensional image feature to obtain a three-dimensional spliced feature; perform convolution processing on the three-dimensional spliced feature to obtain a corresponding three-dimensional fused feature; and perform category prediction on the object in the vehicle operating environment based on the three-dimensional fused feature to obtain the prediction category data.

[0171] S1020, supplement the initial sparse label by the pseudo label to obtain a dense to-be-screened label corresponding to the environmental point cloud data.

[0172] S1022, in response to a selection operation on a to-be-screened label of any category, project the selected to-be-screened label of any category to the environmental image.

[0173] S1024, in a case where a reservation operation on a to-be-screened label of any category is received, determine that the to-be-screened label of any category is a reserved to-be-screened label.

[0174] In S1026, a spatial semantic occupancy label is determined based on the reserved label to be screened.

[0175] Referring to Figure 10 The embodiments of the present specification provide a spatial semantic occupancy label generation apparatus 1000, which comprises a data acquisition module 1010, a category prediction module 1020, and a dense processing module 1030.

[0176] The data acquisition module 1010 is configured to acquire environment point cloud data and environment images corresponding to a vehicle operating environment; wherein the environment point cloud data corresponds to an initial sparse label; the initial sparse label is used to describe the category information of an object in the vehicle operating environment;

[0177] The category prediction module 1020 is configured to perform category prediction on the object in the vehicle operating environment based on the environment point cloud data and the environment images, to obtain prediction category data as pseudo labels corresponding to the environment point cloud data.

[0178] The dense processing module 1030 is configured to perform dense processing on the initial sparse label through the pseudo labels, to obtain spatial semantic occupancy labels corresponding to the vehicle operating environment.

[0179] For specific limitations of the spatial semantic occupancy label generation apparatus, refer to the limitations of the spatial semantic occupancy label generation method in the foregoing, which will not be repeated here. Each module in the above spatial semantic occupancy label generation apparatus can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the electronic device in hardware form, or can be stored in the memory in the electronic device in software form, so as to be called and executed by the processor to perform the operations corresponding to each of the above modules.

[0180] In some embodiments, an electronic device, which can be a terminal, is provided, and an internal structure diagram thereof can be as shown in Figure 11As shown in the figure. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the electronic device is used to communicate with external terminals in a wired or wireless manner. Wireless mode can be achieved through WIFI, operator network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement a spatial semantic occupancy label generation method. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad provided on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0181] Those skilled in the art can understand that, Figure 11 The structure shown in the figure is only a block diagram of part of the structure related to the scheme disclosed in the present specification, and does not constitute a limitation on the electronic device to which the scheme disclosed in the present specification is applied. Specifically, the electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0182] In some embodiments, an electronic device is provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the method steps in the above embodiments.

[0183] In some embodiments, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the method steps in the above embodiments.

[0184] In some embodiments, a computer program product is also provided, which includes instructions that can be executed by the processor of the electronic device to implement the method steps in the above embodiments.

[0185] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in each embodiment provided in the description can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0186] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0187] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified and limited.

[0188] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection", "fixing" and the like should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0189] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that variations, modifications, substitutions and changes can be made by those skilled in the art without departing from the scope of the present application.

Claims

1. A method for spatial semantic occupancy label generation, the method comprising: The method comprises: acquiring environment point cloud data and environment images corresponding to a vehicle operating environment; wherein the environment point cloud data corresponds to an initial sparse label; the initial sparse label is used to describe category information of an object in the vehicle operating environment; performing category prediction on the object in the vehicle operating environment based on the environment point cloud data and the environment images, to obtain prediction category data as pseudo-labels corresponding to the environment point cloud data; performing dense processing on the initial sparse label through the pseudo-labels, to obtain spatial semantic occupancy labels corresponding to the vehicle operating environment.

2. The method of claim 1, wherein, The performing category prediction on the object in the vehicle operating environment based on the environment point cloud data and the environment images, to obtain prediction category data, comprises: performing convolution processing according to the environment point cloud data, to obtain three-dimensional point cloud features corresponding to the environment point cloud data; performing feature extraction and projection according to the environment images, to obtain three-dimensional image features corresponding to the environment images; performing category prediction on the object in the vehicle operating environment based on the three-dimensional point cloud features and the three-dimensional image features, to obtain the prediction category data.

3. The method of claim 2, wherein, The performing convolution processing according to the environment point cloud data, to obtain three-dimensional point cloud features corresponding to the environment point cloud data, comprises: performing voxelization processing on the environment point cloud data, to obtain environment voxel data; performing feature extraction on the environment voxel data through three-dimensional sparse convolution, to obtain the three-dimensional point cloud features.

4. The method of claim 2, wherein, The performing feature extraction and projection according to the environment images, to obtain three-dimensional image features corresponding to the environment images, comprises: performing feature extraction on the environment images, to obtain two-dimensional image features of the environment images; projecting the two-dimensional image features to a three-dimensional space according to a two-dimensional depth map corresponding to the two-dimensional image features and internal and external parameters of a shooting device shooting the environment images, to obtain the three-dimensional image features.

5. The method of claim 2, wherein, The performing category prediction on the object in the vehicle operating environment based on the three-dimensional point cloud features and the three-dimensional image features, to obtain the prediction category data, comprises: performing splicing on the three-dimensional point cloud features and the three-dimensional image features, to obtain three-dimensional spliced features; performing convolution processing on the three-dimensional spliced features, to obtain corresponding three-dimensional fusion features; performing category prediction on the object in the vehicle operating environment based on the three-dimensional fusion features, to obtain the prediction category data.

6. The method according to any one of claims 1 to 5, characterized in that, The performing dense processing on the initial sparse label through the pseudo-labels, to obtain spatial semantic occupancy labels corresponding to the vehicle operating environment, comprises: performing supplementary processing on the initial sparse label through the pseudo-labels, to obtain dense processed to-be-screened labels corresponding to the environment point cloud data; determining retained to-be-screened labels according to to-be-screened labels projected into the environment images; determining the spatial semantic occupancy labels based on the retained to-be-screened labels.

7. The method of claim 6, wherein, The determining retained to-be-screened labels according to to-be-screened labels projected into the environment images, comprises: in response to a selection operation on any category of to-be-screened labels, projecting the selected any category of to-be-screened labels to the environment images; In a case where a reservation operation for the to-be-screened label of any category is received, it is determined that the to-be-screened label of any category is a reserved to-be-screened label.

8. The method of claim 1, wherein, The environment point cloud data corresponds to a three-dimensional bounding box and object semantic information of an object in the vehicle operating environment; and the initial sparse label is generated in the following manner: determining dynamic point cloud data and static point cloud data in the environment point cloud data according to the three-dimensional bounding box and the object semantic information; performing a projection operation based on the static point cloud data to obtain fused static point cloud data; performing fusion processing on the dynamic point cloud data according to position information of the three-dimensional bounding box to obtain fused dynamic point cloud data; superimposing the fused static point cloud data and the fused dynamic point cloud data to obtain superimposed point cloud data; performing voxelization on the superimposed point cloud data to obtain the initial sparse label corresponding to the environment point cloud data.

9. The method of claim 1, wherein, The prediction category data is obtained by a target object classification model; and a training process of the object classification model includes: obtaining an environment point cloud data sample set; wherein the environment point cloud data sample set includes a plurality of environment point cloud data samples, and each environment point cloud data sample corresponds to an initial sparse label and an environment image sample; inputting the environment point cloud data sample and the environment image sample into an object classification model to perform category prediction, to obtain category data corresponding to the environment point cloud data sample; updating the object classification model according to the category data corresponding to the environment point cloud data sample and the initial sparse label, to obtain the target object classification model.

10. A spatial semantic occupancy label generation apparatus, characterized by, The apparatus includes: a data acquisition module configured to acquire environment point cloud data and an environment image corresponding to a vehicle operating environment; wherein the environment point cloud data corresponds to an initial sparse label; and the initial sparse label is used to describe category information of an object in the vehicle operating environment; a category prediction module configured to perform category prediction on the object in the vehicle operating environment based on the environment point cloud data and the environment image, to obtain prediction category data as a pseudo label corresponding to the environment point cloud data; a dense processing module configured to perform dense processing on the initial sparse label based on the pseudo label, to obtain a spatial semantic occupancy label corresponding to the vehicle operating environment. 11.An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program comprises the following steps of: The processor executes the computer program to implement the steps of the method of any one of claims 1 to 9.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 9.

Images