Scene semantic occupancy prediction method based on semantic-distance adaptive gaussian

Abstract

The application provides a scene semantic occupancy prediction method based on semantic-distance adaptive Gauss, and relates to the technical field of machine learning. The method comprises the following steps: extracting features and collecting point clouds from a surround view image sequence around a vehicle, inputting a feature map into a double-branch prediction network to obtain semantic information and position information of original sampling points, screening the original sampling point cloud set to obtain initial Gauss anchor points, and distributing the initial Gauss anchor points to different spatial resolution levels; according to the levels and the semantic information, adjusting a preset basic semantic scale to obtain a modified scale of a Gauss ellipsoid, randomly adding a rotation vector to each Gauss ellipsoid to obtain initial four-dimensional attributes of the Gauss ellipsoid; using a Gauss attribute iterative model to dynamically adjust the initial four-dimensional attributes for multiple rounds, and finally projecting to a target voxel grid to generate a dense 3D semantic occupancy prediction, so that the prediction result is more accurate. The method realizes intelligent dynamic allocation of computing resources and efficient real-time three-dimensional scene understanding capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian. BACKGROUND

[0002] The 3D semantic occupancy perception task of the automatic driving scene aims to reconstruct the three-dimensional geometric structure and semantic information of the dynamic environment in which the vehicle travels through various sensor data, obtain the occupancy state and semantic class label of each voxel unit in the scene, and provide dense spatial understanding for subsequent vehicle path planning and decision planning.

[0003] The existing neural rendering method based on 3D Gaussian sputtering represents the scene as a set of three-dimensional Gaussian ellipsoids in space by using a learnable three-dimensional Gaussian primitive, projects the ellipsoid to a three-dimensional voxel space through Gaussian sputtering technology, calculates the Gaussian semantic weight of each voxel center point, and generates a dense voxel-level occupancy prediction result.

[0004] The existing method adopts a homogenization representation strategy, which uses a unified sampling strategy, scale constraint and semantic label optimization for all objects in the scene. This equalization processing leads to an imbalance in the allocation of computing resources between key dynamic targets and secondary backgrounds, so that the boundaries of safety-critical objects are blurred due to insufficient resolution, and the secondary background area is over-detailed, causing computational redundancy, ultimately affecting the perception accuracy of key targets in the automatic driving scene and the overall resource utilization efficiency of the system. SUMMARY

[0005] The problem to be solved by the present application is that the existing method equally allocates resources to all Gaussian ellipsoids in the scene, resulting in low resolution of key targets due to insufficient computing resources.

[0006] To solve the above problems, in a first aspect, the present application provides a scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian, comprising:

[0007] Extracting features and collecting point clouds from the surround view image sequence around the vehicle to obtain a feature map and an original sample point cloud set;

[0008] Inputting the feature map into a double-branch prediction network to obtain semantic information and position information of the original sample points;

[0009] According to the semantic information and position information of the original sample point cloud set, the farthest point sampling method and the dynamic semantic perception sampling method are used in cooperation to filter the original sample point cloud set as the initialization Gaussian anchor point;

[0010] According to the position information of the initialized Gaussian anchor points, the Euclidean distances between all the initialized Gaussian anchor points and the vehicle are determined, wherein the maximum and minimum values of the Euclidean distances are taken as the maximum Euclidean distance and the minimum Euclidean distance respectively;

[0011] According to the Euclidean distance of each initialized Gaussian anchor point from the vehicle, the maximum Euclidean distance and the minimum Euclidean distance, the level to which each initialized Gaussian anchor point is assigned is determined;

[0012] According to the level and semantic information of the initialized Gaussian anchor points, the preset basic scale is adaptively adjusted to obtain a corrected scale, and after a rotation vector is randomly added, initial four-dimensional attributes of a Gaussian ellipsoid based on the initialized Gaussian anchor points are obtained;

[0013] The initial four-dimensional attributes of the Gaussian ellipsoid are dynamically adjusted through multiple rounds using a Gaussian attribute iterative model, projected to a target voxel grid, and dense 3D semantic occupancy prediction is generated.

[0014] Optionally, the original sampling point cloud set is screened as the initialized Gaussian anchor points according to the semantic information and position information of the original sampling point cloud set, in cooperation with the farthest point sampling method and the dynamic semantic perception sampling method, comprising:

[0015] According to the semantic information of the original sampling points, the types of objects contained in the scene are identified, the points in the original sampling point cloud set are divided into foreground sampling points and background sampling points, and the points with empty semantic categories are removed to obtain an original sampling non-empty point set, wherein the foreground sampling points correspond to dynamic object categories, and the background sampling points correspond to static environmental element categories;

[0016] The proportion of the number of foreground sampling points to the total number of original sampling non-empty points is taken as a foreground object proportion, the proportion weights of the farthest point sampling and the semantic perception sampling in the total sampling budget are determined according to the foreground object proportion, and the sampling quota of the farthest point sampling and the sampling quota of the semantic perception sampling are determined;

[0017] In the farthest point sampling stage, the farthest point sampling algorithm is performed on the original sampling non-empty point set according to the sampling quota of the farthest point sampling to determine a first Gaussian anchor point set;

[0018] In the semantic perception sampling stage, the supplemental sampling quota allocated to each foreground object is determined according to the preset priority weight corresponding to the type of each foreground object, the number of points of the foreground object and the sampling quota of the semantic perception sampling; for the original sampling non-empty points not selected in the farthest point sampling stage, the sampling weight of the original sampling non-empty points not selected is determined according to the occupancy probability prediction value and the local density of the original sampling non-empty points, and the Gaussian anchor point set supplemented by each foreground object is determined according to the size of the sampling weight and the sampling quota, and the Gaussian anchor point sets supplemented by all the foreground objects constitute a second Gaussian anchor point set;

[0019] The first Gaussian anchor point set determined in the farthest point sampling stage and the second Gaussian anchor point set determined in the semantic-aware sampling stage are fused to form a final initialization Gaussian anchor point set.

[0020] Optionally, after determining the level to which each initialization Gaussian anchor point is assigned according to the Euclidean distance of each initialization Gaussian anchor point from the vehicle, the maximum Euclidean distance and the minimum Euclidean distance, the method further comprises:

[0021] According to the semantic information of the initialization Gaussian anchor point and the preset semantic importance division table, the semantic importance factor of the initialization Gaussian anchor point is determined.

[0022] According to the semantic importance factor, the level offset and the level correction model, the level of each initialization Gaussian anchor point is adjusted to determine the corrected level of the initialization Gaussian anchor point.

[0023] Optionally, the preset basic scale is adaptively adjusted according to the level and the semantic information of the initialization Gaussian anchor point to obtain a corrected scale, and an initial four-dimensional attribute of a Gaussian ellipsoid based on the initialization Gaussian anchor point is obtained after a rotation vector is randomly added, comprising:

[0024] According to the semantic information of the initialization Gaussian anchor point, a corresponding preset basic semantic scale is called.

[0025] The total number of levels, the corrected level of the initialization Gaussian anchor point and the preset basic semantic scale are input into a scale correction model to obtain a corrected scale of the Gaussian ellipsoid based on the initialization Gaussian anchor point.

[0026] The rotation vector attribute of each Gaussian ellipsoid is randomly initialized, a rotation vector is added to each initialization Gaussian anchor point to obtain an initial four-dimensional attribute of an optimized Gaussian ellipsoid.

[0027] Optionally, the initial four-dimensional attribute of the Gaussian ellipsoid is dynamically adjusted using a Gaussian attribute iterative model, and the initial four-dimensional attribute of the Gaussian ellipsoid is projected onto a target voxel grid to generate a dense 3D semantic occupancy prediction, comprising:

[0028] The initial four-dimensional attribute of the Gaussian ellipsoid is dynamically adjusted using a Gaussian attribute iterative model to obtain a four-dimensional iterative attribute of the Gaussian ellipsoid.

[0029] According to the position, the corrected scale and the rotation vector attribute in the four-dimensional iterative attribute, a joint gradient norm is determined.

[0030] According to the semantic importance factor and the joint gradient norm, a level migration value of each Gaussian ellipsoid is determined.

[0031] According to the Euclidean distance, the maximum Euclidean distance and the corrected level corresponding to the initialization Gaussian anchor point, a migration threshold value of each Gaussian ellipsoid is determined.

[0032] If the level migration value is greater than the migration threshold value, then according to the migration threshold value, the corrected level and the total number of levels, the post-migration level is determined;

[0033] The total number of levels, the post-migration level and the preset basic semantic scale are input into a scale correction model to obtain an optimized scale of the Gaussian ellipsoid and update the four-dimensional iterative attribute of the Gaussian ellipsoid;

[0034] The four-dimensional iterative attribute of the Gaussian ellipsoid is cyclically and dynamically adjusted using a Gaussian attribute iterative model until an iterative termination condition is met, and the four-dimensional iterative attribute of the iteratively optimized Gaussian ellipsoid is obtained.

[0035] Optionally, the signature extraction and point cloud collection on the surround view image sequence around the vehicle are performed to obtain a feature map and an original sampling point cloud set, including:

[0036] The feature extraction network is used to perform feature extraction on the surround view image sequence around the vehicle to obtain the feature map;

[0037] The rays are emitted from the camera center on the feature map pixel by pixel, and the rays are interval sampled at equal sampling intervals in a preset sampling depth range to obtain the original sampling point cloud set.

[0038] Optionally, before the feature map is input into the double-branch prediction network to obtain the semantic information and the position information of the original sampling point, the method further includes:

[0039] The feature map is input into the double-branch prediction network to obtain the predicted occupancy distribution and the predicted semantic distribution of the sampling point;

[0040] The real occupancy label and the real semantic label of each sampling point are queried from the 3D voxel occupancy ground truth field, and the loss value is determined using a joint loss function in combination with the predicted occupancy distribution and the predicted semantic distribution;

[0041] The double-branch prediction network is adjusted according to the loss value to determine the trained double-branch prediction network.

[0042] In a second aspect, the present application further provides a scene semantic occupancy prediction system based on semantic-distance adaptive Gaussian, including:

[0043] The point cloud collection module is configured to perform signature extraction and point cloud collection on the surround view image sequence around the vehicle to obtain a feature map and an original sampling point cloud set;

[0044] The sampling point analysis module is configured to input the feature map into the double-branch prediction network to obtain the semantic information and the position information of the original sampling point;

[0045] The sampling point screening module is configured to screen the original sampling point cloud set according to semantic information and position information of the original sampling point cloud set, in cooperation with a farthest point sampling method and a dynamic semantic perception sampling method, to serve as an initial Gaussian anchor point.

[0046] The anchor point level division module is configured to determine the Euclidean distance between all initial Gaussian anchor points and the vehicle according to position information of the initial Gaussian anchor points, wherein a maximum value and a minimum value of the Euclidean distance are taken as a maximum Euclidean distance and a minimum Euclidean distance, respectively; and to determine a level to which each initial Gaussian anchor point is assigned according to the Euclidean distance of each initial Gaussian anchor point from the vehicle, the maximum Euclidean distance and the minimum Euclidean distance.

[0047] The Gaussian attribute initialization module is configured to adaptively adjust a preset basic scale according to the level and semantic information of the initial Gaussian anchor point to obtain a corrected scale, and to obtain initial four-dimensional attributes of a Gaussian ellipsoid based on the initial Gaussian anchor point after adding a rotation vector randomly.

[0048] The optimization prediction module is configured to perform multi-round dynamic adjustment on the initial four-dimensional attributes of the Gaussian ellipsoid using a Gaussian attribute iterative model, project the initial four-dimensional attributes to a target voxel grid, and generate a dense 3D semantic occupancy prediction.

[0049] In a third aspect, the present application provides an electronic device, comprising a memory and a processor.

[0050] The memory is configured to store a computer program.

[0051] The processor is configured to implement the scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian when executing the computer program.

[0052] In a fourth aspect, the present application provides a computer readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian is implemented.

[0053] The present application provides a scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian.

[0054] The signature extraction and point cloud collection are performed on the surround view image sequence of the vehicle, to obtain a feature map and an original sampling point cloud set, then the feature map is input into a double-branch prediction network to obtain semantic information and position information of the original sampling points, and according to the semantic information and position information of the original sampling point cloud set, the farthest point sampling and dynamic semantic perception sampling methods are used in cooperation to perform optimization screening on the original sampling point cloud set, so that the limited Gaussian ellipsoid resources are preferentially concentrated in the safety-critical dynamic objects and complex structure regions, and finally a better initialization Gaussian anchor point is obtained. Further, according to the position information of the initialization Gaussian anchor point, the Euclidean distance between all initialization Gaussian anchor points and the vehicle is determined, and then the Euclidean distance, the maximum Euclidean distance and the minimum Euclidean distance of each initialization Gaussian anchor point from the vehicle are used to determine the level to which each initialization Gaussian anchor point is assigned, and subsequent different constraint Gaussian ellipsoid scales can be allocated according to the level, so that the key objects can obtain sufficient resolution; according to the level and semantic information of the initialization Gaussian anchor point, the preset basic semantic scale is adaptively adjusted to obtain a modified scale of the Gaussian ellipsoid based on the initialization Gaussian anchor point, which is negatively correlated with the semantic importance of the object, and a rotation vector is randomly added to each Gaussian ellipsoid to obtain the initial four-dimensional attributes of the Gaussian ellipsoid; the initial four-dimensional attributes of the Gaussian ellipsoid are dynamically adjusted for multiple rounds by using a Gaussian attribute iteration model to optimize the initial four-dimensional attributes of the Gaussian ellipsoid, the expression of the four-dimensional attributes is more accurate, and the four-dimensional attributes are projected to a target voxel grid to generate a dense 3D semantic occupancy prediction, and the prediction result is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0056] Figure 1 A flowchart of a scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian provided by an embodiment of the present application;

[0057] Figure 2 A data processing process diagram of a scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian provided by an embodiment of the present application;

[0058] Figure 3 A structure diagram of a scene semantic occupancy prediction system based on semantic-distance adaptive Gaussian provided by an embodiment of the present application. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application are described clearly and completely. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0061] like Figure 1 As shown in the embodiment of this application, a scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian is provided, including:

[0062] S1: Extract features and collect point clouds from the surround view image sequence around the vehicle to obtain feature maps and original sampling point sets.

[0063] S2: Input the feature map into the dual-branch prediction network to obtain the semantic and location information of the original sampling points.

[0064] S3: Based on the semantic and location information of the original sampling point cloud, the farthest point sampling method and the dynamic semantic perception sampling method are used in combination to filter the original sampling point cloud and use it as the initial Gaussian anchor point.

[0065] S4: Based on the position information of the initialized Gaussian anchor points, determine the Euclidean distance between all initialized Gaussian anchor points and the vehicle, where the maximum and minimum Euclidean distances are used as the maximum and minimum Euclidean distances, respectively.

[0066] S5: Determine the level to which each initial Gaussian anchor point is assigned based on the Euclidean distance, maximum Euclidean distance, and minimum Euclidean distance from the vehicle to each initial Gaussian anchor point.

[0067] S6: Adaptively adjust the preset base scale according to the hierarchy and semantic information of the initialized Gaussian anchor points to obtain the corrected scale. After randomly adding rotation vectors, obtain the initial four-dimensional attributes of the Gaussian ellipsoid based on the initialized Gaussian anchor points. The four-dimensional attributes include position, semantics, corrected scale, and rotation vector.

[0068] S7: Use a Gaussian property iteration model to dynamically adjust the initial four-dimensional properties of the Gaussian ellipsoid in multiple rounds, project it onto the target voxel grid, and generate dense 3D semantic occupancy prediction.

[0069] In the optional embodiment, the feature extraction and point cloud collection are performed on the surround view image sequence around the vehicle to obtain a feature map and an original sampling point cloud set, then the feature map is input into a double-branch prediction network to obtain semantic information and position information of the original sampling points, and according to the semantic information and position information of the original sampling point cloud set, the farthest point sampling and dynamic semantic perception sampling methods are used in cooperation to perform optimization screening on the original sampling point cloud set, so that the limited Gaussian ellipsoid resources are preferentially concentrated in the safety-critical dynamic objects and complex structure regions, and finally a better initialization Gaussian anchor point is obtained. Further, according to the position information of the initialization Gaussian anchor point, the Euclidean distance between all initialization Gaussian anchor points and the vehicle is determined, and then the Euclidean distance, the maximum Euclidean distance and the minimum Euclidean distance of each initialization Gaussian anchor point from the vehicle are used to determine the level to which each initialization Gaussian anchor point is assigned, and subsequent different constraint Gaussian ellipsoid scales can be allocated according to the level, so that the key objects can obtain sufficient resolution; according to the level and semantic information of the initialization Gaussian anchor point, the preset basic semantic scale is adaptively adjusted to obtain a modified scale of the Gaussian ellipsoid based on the initialization Gaussian anchor point, which is negatively correlated with the semantic importance of the object, and a rotation vector is randomly added to each Gaussian ellipsoid to obtain the initial four-dimensional attributes of the Gaussian ellipsoid; the initial four-dimensional attributes of the Gaussian ellipsoid are dynamically adjusted for multiple rounds using a Gaussian attribute iteration model to optimize the initial four-dimensional attributes of the Gaussian ellipsoid, the expression of the four-dimensional attributes is more accurate, and the four-dimensional attributes are projected to a target voxel grid to generate a dense 3D semantic occupancy prediction, and the prediction result is more accurate. Through the semantic-distance adaptive dynamic 3D Gaussian scene construction method, a three-dimensional environment model with geometric details and semantic information can be constructed in real time, so that the automatic driving system can more accurately identify key targets in complex scenes such as intersection blind areas and dense traffic, and the perception misjudgment risk can be significantly reduced. The above semantic-distance coupled Gaussian resource allocation strategy realizes intelligent allocation of computing resources, provides a vehicle-mounted platform with efficient and real-time three-dimensional scene understanding capability, and can improve the calculation efficiency of three-dimensional scene representation.

[0070] As shown in the following, each step is described in detail. Figure 2

[0071] S1: feature extraction and point cloud collection are performed on the surround view image sequence around the vehicle to obtain a feature map and an original sampling point cloud set, and the step includes the following detailed steps.

[0072] Specifically, a multi-view camera sensor mounted on the vehicle is used to synchronously collect a multi-view surround color image sequence of the driving scene, and an occupancy prediction model such as SurroundOcc, Occ3D, etc. is used to generate a dense 3D voxel occupancy ground truth label set corresponding to the time stamp.

[0073] S11: feature extraction is performed on the surround view image sequence around the vehicle using a feature extraction network to obtain a feature map. ​

[0074] Specifically, the multi-view camera images are fed into a feature extraction network, such as Figure 2 As shown, the ResNet101 network is selected as the backbone and is combined with the FPN neck network, which is responsible for extracting multi-scale visual feature maps from the images of each view. Given the look-around RGB image sequence at each time step and the corresponding camera internal and external parameters, the ResNet101 network is responsible for extracting and abstracting features from the input images layer by layer, and outputs multi-level feature maps with decreasing resolution. The FPN network fuses the features of different levels output by the backbone network, and outputs a set of feature maps with decreasing spatial resolution but each level fused with strong semantics and high resolution information, for example, the first level output first level scale feature map with higher size and resolution.

[0075] In addition, in order to accurately depict the complex road environment, objects and structures frequently appearing in the autonomous driving scene are summarized into 17 representative semantic categories, including bicycle, bus, car, engineering vehicle, motorcycle, pedestrian, trailer, truck, guardrail, traffic cone, sidewalk, unknown object, drivable pavement, artificial structure, other flat ground, natural terrain and vegetation. The designed scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian realizes the unified modeling of environment geometry and semantics through a set of sparse learnable semantic Gaussian primitives. Specifically, this method uses a 3D representation method centered on the vehicle to represent an autonomous driving scene using a set of 3D semantic Gaussian distributions Each 3D Gaussian is represented as an ellipsoid in space, and each Gaussian ellipsoid is represented by a learnable vector in the form of , respectively representing the position, semantic logic value, scale and rotation vector in the Gaussian attribute, where c increases one dimension to represent the occupancy probability.

[0076] S12: From the camera center, a ray is emitted on the feature map, and in the preset sampling depth range, interval sampling is performed on the ray at equal sampling intervals to obtain an original sampling point cloud set.

[0077] Specifically, the first level feature map in the multi-scale feature map extracted by the image feature extraction network is selected, and the origin a and direction e of the ray emitted from the camera center corresponding to each pixel unit of the first level feature map are calculated. The ray is parameterized as where t is a depth scalar, and the sampling depth range is as shown in Figure 2 In the fixed sampling depth range, b reference points are sampled at equal intervals to form an original three-dimensional point sampling set .

[0078] S2: input the feature map into the double-branch prediction network to obtain semantic information and position information of the original sampling points.

[0079] Specifically, the first-level feature map is input into the double-branch prediction network for processing, as shown in the following formula (1). Figure 2 As shown, both branches are composed of a relu activation layer, a fully connected layer and a Softmax classifier, and the image features are directly decoded into the predicted occupancy distribution of each sampling point along the corresponding ray pixel by pixel and the predicted semantic distribution From the predicted occupancy distribution and the predicted semantic distribution of each sampling point, the occupancy probability and the semantic category of each sampling point can be obtained. Before using the double-branch prediction network, the double-branch prediction network can be pre-trained, and the training process specifically includes:

[0080] S21: input the feature map into the double-branch prediction network to obtain the predicted occupancy distribution and the predicted semantic distribution of the sampling points.

[0081] S22: query the real occupancy label and the real semantic label of each sampling point from the 3D voxel occupancy ground truth field, combine the predicted occupancy distribution and the predicted semantic distribution, and use a joint loss function to determine the loss value.

[0082] Specifically, according to the voxel grid where the b sampling points fall, the real occupancy label and the real semantic label of each sampling point are queried from the 3D voxel occupancy ground truth field, and the training process of the double-branch prediction network is supervised by two types of one-hot vector labels of occupancy and semantics, wherein the joint loss function is as follows.

[0083]

[0084] wherein, represents the weight coefficient of the occupancy loss, represents the binary cross-entropy loss function of the occupancy, represents the weight coefficient of the semantic loss, represents the total number of semantic categories, represents the real semantic label value of the kth sampling point in the category c, represents the semantic prediction value of the kth sampling point in the category c, and b represents the total number of original sampling points.

[0085] S23: adjust the double-branch prediction network according to the loss value to determine the trained double-branch prediction network.

[0086] Specifically, the loss value after each training is calculated in real time according to the loss function, and if the loss value is within the acceptable range or reaches the specified number of training times, the training can be stopped, and the parameters of the double-branch prediction network obtained after training are fixed. The training of this part belongs to local training, which can fix the parameters of the double-branch prediction network locally, and then use global training to train the entire network architecture of the method. The training strategy of local first and global second can ensure the accuracy of the local network and improve the training speed. Multiple local training can be carried out at the same time, and then when global training is carried out, the training speed of the global network is further accelerated due to the optimized local network.

[0087] S3: According to the semantic information and position information of the original sampling point cloud set, the farthest point sampling method and the dynamic semantic perception sampling method are used in cooperation to screen the original sampling point cloud set as the initialization of the Gaussian anchor point.

[0088] Good initialization will eventually significantly affect the accuracy of model prediction, so it is necessary to further screen high-quality points on the basis of the original sampling set as the initialization of the Gaussian ellipsoid. Most existing methods use the farthest distance sampling method to ensure that a small part of points with high occupancy probability can support the entire scene. However, this method simply relies on spatial distance information to select the point with the highest occupancy probability, which can ensure the spatial dispersion of the sampling points in the scene, but completely ignores the important semantic distribution characteristics in the autonomous driving scene. The farthest point sampling pursues spatial uniformity, but the core requirement of 3D occupancy prediction is to reconstruct the objects that have a substantial impact on driving decisions. Therefore, a new initial point sampling method is designed, which establishes a double-layer sampling architecture of semantic perception layer and geometric distribution layer. The semantic perception layer is responsible for capturing key dynamic objects and important structure information in the scene, and the geometric distribution layer ensures the integrity and uniformity of spatial coverage through farthest point sampling, effectively solving the problem of different classes but equal resources in complex traffic environments.

[0089] Specifically, in the sampling execution phase, the method optimizes scene representation through foreground and background classification statistics and a dynamic proportion allocation mechanism: first, the dynamic foreground and static background point clouds are divided according to semantic labels, and the proportion of foreground objects is used as an indicator to dynamically allocate the budget quota of the farthest point sampling and semantic perception sampling, so as to focus on foreground accuracy in traffic-intensive scenarios and strengthen spatial coverage in open scenarios. Then, a two-stage sampling strategy is adopted: first, a spatially uniform distribution of geometric basis is constructed through farthest point sampling; then, according to the priority weight allocation algorithm, the number of supplementary samples is allocated to each foreground class through semantic perception sampling, and for the foreground points that have not been selected, a weighted sampling mechanism is constructed by combining the occupancy probability prediction value and three-dimensional local distribution density information, and the representative points in the high-occupancy-probability and low-density areas are preferentially selected. Finally, the two types of sampling results are fused, and under strict memory constraints, the limited Gaussian resources are focused on the safety-critical dynamic objects and complex structure areas, significantly improving the perception integrity of sparse targets such as pedestrians and bicycles, while avoiding redundant calculations in uniform background areas, achieving balanced optimization of scene geometric structure and semantic accuracy.

[0090] S31: According to the semantic information of the original sampling points, the object types contained in the scene are identified, the points in the original sampling point cloud set are divided into foreground sampling points and background sampling points, and the points with empty semantic categories are removed, to obtain a set of original sampling non-empty points, wherein the foreground sampling points correspond to dynamic object categories, and the background sampling points correspond to static environmental element categories.

[0091] Specifically, the original sampling set is subjected to semantic distribution statistical analysis, and is divided into foreground sampling points and background sampling points according to the characteristics of the semantic categories, wherein the foreground sampling points correspond to dynamic element categories, including bicycles, buses, cars, engineering vehicles, motorcycles, pedestrians, trailers and trucks, and the background sampling points correspond to static environmental element categories, including guardrails, traffic cones, sidewalks, unknown objects, drivable road surfaces, artificial structures, other flat ground, natural terrain and vegetation.

[0092] S32: The proportion of the number of foreground sampling points to the total number of original sampling non-empty points is taken as the proportion of foreground objects, and the proportion weights of the farthest point sampling and the semantic perception sampling in the total sampling budget are determined according to the proportion of foreground objects, and the sampling quota of the farthest point sampling and the sampling quota of the semantic perception sampling are determined.

[0093] Specifically, the proportion of foreground objects is directly taken as the proportion weight corresponding to the semantic perception sampling, and the proportion weight corresponding to the farthest point sampling is determined by using 100% minus the proportion weight corresponding to the semantic perception sampling. The total sampling budget (such as the preset total number of sampling points) is multiplied by the proportion weight corresponding to the farthest point sampling and the proportion weight corresponding to the semantic perception sampling, respectively, to correspondingly obtain the sampling quota (number of sampling points) of the farthest point sampling and the sampling quota of the semantic perception sampling.

[0094] S33: In the farthest point sampling stage, a farthest point sampling algorithm is performed on the original sampling non-empty point set according to the determined farthest point sampling quota to determine a first Gaussian anchor point set; the farthest point sampling algorithm selects a candidate point with the maximum distance from all points in the current selected point set by iteration. The farthest point sampling algorithm is used to generate a point set uniformly distributed in a three-dimensional space.

[0095] S34: In the semantic-aware sampling stage, a supplementary sampling quota allocated to each foreground object is determined according to a preset priority weight corresponding to a type of each foreground object, a point number of the foreground object, and a sampling quota of the semantic-aware sampling; for an original sampling non-empty point (i.e., a to-be-sampled point) that is not selected in the farthest point sampling stage, a sampling weight of the to-be-sampled point is determined according to an occupancy probability prediction value and a local density of the original sampling non-empty point, and a Gaussian anchor point set supplemented by each foreground object is determined according to the size of the sampling weight and the sampling quota, and the Gaussian anchor point sets supplemented by all foreground objects constitute a second Gaussian anchor point set. The formula is as follows:

[0096]

[0097] wherein, represents the semantic-aware sampling quota, represents the supplementary sampling quota of each foreground object, represents the point number of the foreground object, represents the preset priority weight of the foreground object, and k represents the number of object categories included in the scene.

[0098] The sampling weight is calculated in combination with the occupancy probability of a point in the original sampling non-empty point set that has not been selected and the local distribution density information calculated based on the spatial proximity relationship.

[0099] Specifically, I same-category neighboring points closest to the to-be-sampled point are found, and the local density of the to-be-sampled point is calculated as follows:

[0100]

[0101] wherein, I is the number of same-category neighboring points of the to-be-sampled point, is the constant pi, is the square of the Euclidean distance of the to-be-sampled point to the i-th neighboring point, and the density values of all to-be-sampled points in the foreground object are normalized to obtain a relative density value

[0102]

[0103] wherein, represents the i-th neighboring point, ​​The sampling weights of each sampling point This represents the predicted occupancy probability of the sampling point. This indicates the sparsity of the region where the point is located. These are the weights for occupancy information and density information, respectively. Weighted sampling is performed, and points that meet the supplementary sampling quota for this category are selected.

[0104] S35: The first Gaussian anchor point set determined in the farthest point sampling stage and the second Gaussian anchor point set determined in the semantic perception sampling stage will be fused to form the final initialized Gaussian anchor point set.

[0105] Existing technologies suffer from problems such as insufficient semantic modeling and homogenized resource allocation, for example Figure 2 As shown, this application proposes a semantic-distance dual adaptive Gaussian scaling mechanism, which uses both semantic category information and distance information for Gaussian scale allocation. It proposes a dynamic hierarchical approach based on viewpoint distance and incorporates the semantic and dynamic characteristics of autonomous driving scenarios to design a more practical hierarchical control strategy. This enables Gaussian anchor points to form a 3D semantic-distance adaptive Gaussian scene with hierarchical structure and perception priority. The steps include the following detailed steps.

[0106] S4: Based on the position information of the initialized Gaussian anchor points, determine the Euclidean distance between the initialized Gaussian anchor points and the origin of the vehicle coordinate system. The maximum and minimum Euclidean distances of all initialized Gaussian anchor points are taken as the maximum Euclidean distance. and minimum Euclidean distance .

[0107] Specifically, the Euclidean distance between the initialized Gaussian anchor point and the vehicle coordinate system is calculated. ,in, Indicates the first The coordinate information of each Gaussian anchor point The coordinates of the vehicle's center are used to obtain the minimum and maximum distances of all anchor points from the vehicle's center, and a bounding box is then constructed accordingly.

[0108] S5: Determine the level to which each initial Gaussian anchor point is assigned based on its Euclidean distance, maximum Euclidean distance, and minimum Euclidean distance.

[0109] Specifically, the formula for calculating the total number of levels in the spatial resolution layer is as follows:

[0110]

[0111] in, This represents the floor function.

[0112] The initial Gaussian anchor points are then mapped to the corresponding resolution level according to the distance from the vehicle center by a non-linear level mapping function, and the level of the initial Gaussian anchor point is determined using the non-linear level mapping function as follows:

[0113]

[0114] wherein d represents the distance of the i-th initial Gaussian anchor point from the vehicle center. The mapping function dynamically determines the anchor point level division, and each level corresponds to a different Gaussian scale. This level division criterion ensures that the corresponding granularity of the Gaussian representation can be obtained through the appropriate level regardless of the distance of the target in the scene. The level of each initial Gaussian anchor point is adjusted according to the semantic importance factor, the level offset, and the level correction model.

[0115] S51: Determine the semantic importance factor of the initial Gaussian anchor point according to the semantic information of the initial Gaussian anchor point and the preset semantic importance division table.

[0116] Specifically, the semantic information is further incorporated into the level decision. An independent level bias or threshold is set for each semantic category, so that anchor points of different categories can fall into different levels under the same distance condition. Specifically, we combine the semantic priority information to grade the elements of the 17 semantic categories in the autonomous driving scene and introduce a semantic importance factor , as shown in Table 1. Then, according to the autonomous driving decision requirements, the foreground dynamic objects such as vehicles and pedestrians are placed in high priority, and the static backgrounds such as vegetation and buildings are placed in low priority.

[0117] Table 1 Semantic category importance division table

[0118]

[0119] S52: Adjust the level of each initial Gaussian anchor point according to the semantic importance factor, the level offset, and the level correction model, and determine the corrected level of the initial Gaussian anchor point.

[0120] Specifically, to solve the problem of over-degrading small objects at a long distance (such as traffic cone cylinders 50m away) due to distance penalty, we set up a semantic priority mechanism to protect the level of key objects. By introducing a semantic importance factor and a level offset , the original level is semantically corrected: the distance constraint is completely removed for key objects, and the highest fine level is directly assigned, and the level of medium and low level objects is adaptively adjusted. The level correction model is as follows:

[0121]

[0122] wherein d represents the distance of the i-th initial Gaussian anchor point from the vehicle center. The mapping function dynamically determines the anchor point level division, and each level corresponds to a different Gaussian scale. This level division criterion ensures that the corresponding granularity of the Gaussian representation can be obtained through the appropriate level regardless of the distance of the target in the scene. ​​a semantic importance factor of the initialized Gaussian anchor point, a level offset of the initialized Gaussian anchor point, a level of the initialized Gaussian anchor point, a level of the initialized Gaussian anchor point, a level of the initialized Gaussian anchor point, D represents a total number of levels, a level of the initialized Gaussian anchor point, a modified level of the initialized Gaussian anchor point.

[0123] S6: adaptively adjusting a preset basic scale according to the level and semantic information of the initialized Gaussian anchor point to obtain a modified scale, and obtaining an initial four-dimensional attribute of the Gaussian ellipsoid based on the initialized Gaussian anchor point after adding a random rotation vector.

[0124] Specifically, an empirical mapping rule of Gaussian scale is established: according to semantic importance and distance from the vehicle, a Gaussian kernel with larger scale and sparse distribution is used to represent elements with low spatial level, so as to cover a wide area and save computing resources; a Gaussian kernel with smaller scale and higher accuracy is used to represent elements with high spatial level, so as to ensure geometric details and model prediction accuracy. This step includes the following detailed steps.

[0125] S61: according to the semantic information of the initialized Gaussian anchor point, a corresponding preset basic semantic scale is called.

[0126] Specifically, after obtaining the modified level of the anchor point according to the anchor point distance and semantic information, a scale regulation mechanism is designed according to the assumed Gaussian scale mapping rule, and a final scale is allocated to each Gaussian anchor point to balance the semantic importance and spatial resolution requirements. The basic semantic scale corresponding to different semantic categories is:

[0127]

[0128] For example,

[0129] S62: inputting the total number of levels, the modified level of the initialized Gaussian anchor point and the preset basic semantic scale into a scale correction model to obtain the modified scale of the Gaussian ellipsoid based on the initialized Gaussian anchor point.

[0130] Specifically, a level modulation parameter and a power-law coefficient are introduced, and the modified scale is determined by the basic semantic scale , the level modulation parameter and , and the specific formula is:

[0131]

[0132] wherein, is an element-wise multiplication, and the final modified scale is physically constrained, and the constraint formula is: , is a clamping function. Through the above spatial level and semantic coupling Gaussian scale assignment mechanism, each Gaussian element automatically obtains the appropriate scale configuration according to its semantic category and distance to the vehicle, realizing high-resolution attention to key dynamic targets and efficient rough expression to background areas in the automatic driving scene, and balancing expression accuracy and computational efficiency.

[0133] S63: Randomly initialize the rotation vector attribute of each Gaussian ellipsoid, add a rotation vector to each initialized Gaussian anchor point, obtain the initial four-dimensional attributes of the Gaussian ellipsoid, including position, semantics, modified scale and rotation vector.

[0134] Specifically, after the initial anchor point obtains the position and semantic attributes through the self-defined initialization strategy, it obtains the initial scale attribute through the above-mentioned semantic-distance dual adaptive Gaussian scale regulation mechanism, and randomly initializes the rotation vector attribute, as shown in Figure 2 , the final four-dimensional attributes of the initialized Gaussian anchor point are obtained, and the Gaussian ellipsoid can be formed based on the Gaussian anchor point, that is, the initial four-dimensional attributes of the Gaussian ellipsoid are obtained.

[0135] S7: Use the Gaussian attribute iterative model to dynamically adjust the initial four-dimensional attributes of the Gaussian ellipsoid for multiple rounds, project to the target voxel grid, and generate dense 3D semantic occupancy prediction. This step includes the following detailed steps.

[0136] S71: Use the Gaussian attribute iterative model to dynamically adjust the initial four-dimensional attributes of the Gaussian ellipsoid to obtain the four-dimensional iterative attributes of the Gaussian ellipsoid.

[0137] The initialized Gaussian ellipsoid is converted into a sparse voxel structure through voxelization, spatial feature interaction is performed using three-dimensional sparse convolution, and the Gaussian position is projected to the multi-view image feature plane through a deformable attention mechanism to aggregate local image features to enhance Gaussian features. Specifically, the initial Gaussian ellipsoid is taken as the spatial positioning reference with its position coordinates, a discrete spatial point cloud representation is constructed, the dispersed Gaussian nodes are converted into a regularized sparse voxel structure through uniform voxel grid division of the point cloud data, and on the basis of this structured space, a 3D sparse convolutional neural network is used to perform feature aggregation in the non-empty voxel neighborhood through a learnable convolution kernel, adaptively capturing local spatial correlation, realizing implicit feature interaction among three-dimensional Gaussian groups, providing geometric-aware Gaussian representation for subsequent cross-modal fusion, while retaining the efficiency advantage of sparse calculation.

[0138] The multi-view 2D image information is injected into the 3D Gaussian anchor point through deformable attention. Specifically, for the th Gaussian anchor point , whose current attributes are represented as query vector , first generate a set of 3D reference points along random directions according to their covariance , where is a three-dimensional offset, for the nth camera, project these 3D reference points through the camera to the image plane to get 2D projection points , and K represent the intrinsic and extrinsic parameters of the camera respectively. To enhance the capture of local image details and the guiding effect of image pair anchor points, we also perform multi-point sampling in the neighborhood of each projection point to obtain two-dimensional secondary reference points , where is a two-dimensional offset. Aggregate the deformable attention outputs of all cameras, all three-dimensional offsets, and all two-dimensional neighborhood points, and then perform weighted fusion to further enrich the semantic information of the anchor points and improve the optimization guiding ability of sparse anchor points. As shown in the following formula:

[0139]

[0140] where N represents the number of cameras, R represents the number of 3D offset points, J represents the number of 2D offset points, is the first-level image feature map of the nth camera in the current frame, represents the operation of performing deformable attention at the location x of the feature map F with the query Q and returning the weighted feature vector. In this way, each Gaussian anchor point can aggregate the local semantic features of the image at its projection point, thereby updating its representation.

[0141] A multi-layer perceptron neural network is used to decode the position, rotation, scale and semantic attributes of the Gaussian from the Gaussian feature. The Gaussian attribute correction model is used to iteratively optimize each attribute, as shown in Figure 2 , after each round of update, the Gaussian scale parameter is dynamically adjusted as the input for the next iteration based on the new corrected semantic attributes and distance information. The number of iterations can be preset. Specifically, under the guidance of the above attention mechanism, the four attributes of each Gaussian anchor point are optimized. To maintain the coherence of the spatial structure of the anchor points, we use an incremental update strategy to adjust the center position, while the scale, rotation and semantic attributes are updated in a full amount. The specific representation is:

[0142]

[0143] where MLP() represents the multi-layer perceptron decoding, represents the intermediate Gaussian distribution of iteration, represents the updated query vector, respectively represent the position, semantic, scale and rotation attributes of the newly decoded Gaussian ellipsoid from the query vector, represents a new Gaussian distribution.

[0144] S72: Determine the joint gradient norm according to the attributes in the four-dimensional iterative attributes: position, corrected scale and rotation vector.

[0145] Specifically, after obtaining the new attributes of the Gaussian, the Gaussian scale is fine-tuned again as the basis for the next iteration. To this end, a cross-level gradient migration mechanism is designed to dynamically adjust the anchor growth, specifically: the joint gradient norm of the latest position, scale, semantic of each Gaussian is counted :

[0146]

[0147] wherein, represents the gradient of the position attribute of the Gaussian ellipsoid, represents the square of the two-norm of the position gradient of the Gaussian ellipsoid, represents the scale gradient of the Gaussian ellipsoid, represents the square of the two-norm of the scale gradient of the Gaussian ellipsoid, represents the semantic gradient of the Gaussian ellipsoid, represents the square of the two-norm of the semantic gradient of the Gaussian ellipsoid.

[0148] S73: Determine the level migration value of each Gaussian ellipsoid according to the semantic importance factor and the joint gradient norm.

[0149] Specifically, the semantic weighted joint gradient norm is then used to determine the level migration value:

[0150] , represents the square of the two-norm of the semantic gradient of the Gaussian ellipsoid.

[0151] S74: Determine the migration threshold of each Gaussian ellipsoid according to the Euclidean distance corresponding to the initialized Gaussian anchor, the maximum Euclidean distance and the corrected level.

[0152] The migration threshold is , is the reference gradient threshold, is the distance decay factor, is the level suppression factor.

[0153] S75: If the level migration value is greater than the migration threshold, then determine the migrated level according to the migration threshold, the corrected level and the total number of levels.

[0154]

[0155] wherein, is an integer rounding, is the new level after anchor migration, making important objects more easily accessible.

[0156] S76: Input the total number of levels, the level after migration, and the preset basic semantic scale into the scale correction model to obtain the optimized scale of the Gaussian ellipsoid and update the four-dimensional iterative properties of the Gaussian ellipsoid.

[0157] Specifically, the other properties remain unchanged during migration, and only the scale is recalculated This mechanism triggers the strategy through semantic and distance perception gradient, and the semantic priority factor ensures that key targets are more easily refined first, achieving dynamic focusing of key targets at high-resolution levels. While avoiding global calculation redundancy, it significantly improves the geometric restoration accuracy of dynamic objects and complex structures.

[0158] S77: Use the Gaussian property iterative model to cyclically and dynamically adjust the four-dimensional iterative properties of the Gaussian ellipsoid until the iterative termination condition is met, obtaining the four-dimensional iterative properties of the iteratively optimized Gaussian ellipsoid.

[0159] Specifically, the new four-dimensional properties of the Gaussian ellipsoid are input to reiterate the above process. If the migration condition is met, update the level and scale of the Gaussian ellipsoid until the number of iterations reaches the preset number.

[0160] After that, project the finally optimized Gaussian ellipsoid to the target voxel grid and accumulate the semantic contribution value of the Gaussian cell to generate dense 3D semantic occupancy prediction.

[0161] Specifically, to adapt to voxel-level true value supervision, an efficient Gaussian-voxel sputtering mechanism is used, as shown in Figure 2 , which converts the learned 3D semantic Gaussian representation of the scene into dense semantic occupancy prediction and applies multiple losses to ensure the consistency of global structure and local details. Specifically, project the above obtained group of to the required dense voxel grid , and obtain the voxel center occupancy semantic value by aggregating the semantic properties of the neighboring Gaussian cells of each voxel center . First, define the semantic contribution of semantic Gaussian distribution G at a certain voxel center as follows:

[0162]

[0163] where, and respectively, and c denotes the semantic log value of the Gaussian ellipsoid. Further, the semantic occupancy prediction result of the voxel center point v can be represented as the sum of the semantic contributions of the neighboring Gaussian units to it, and the calculation formula is:

[0164]

[0165] wherein, is the index set of the Gaussian neighborhood effectively overlapping with the voxel v, which significantly reduces the traversal overhead of the global k Gaussians, denotes the semantic log value of the i-th Gaussian ellipsoid.

[0166] In addition, by comparing the prediction result with the true value label through the weighted cross-entropy loss and the Lovász-Softmax loss, the model is supervised and trained, and finally the three-dimensional voxel-level occupancy prediction result with geometric structure and semantic information is output, which provides dense environment understanding for automatic driving decision planning.

[0167] Specifically, after obtaining the dense semantic occupancy tensor , we compare it with the real label (including occupancy / free and multi-class semantic labels), and adopt the weighted combination of the cross-entropy loss and the Lovász-Softmax loss as the final supervision signal :

[0168]

[0169] wherein, and are the weight factors of the cross-entropy and the Lovász-Softmax, respectively, which are used to balance the pixel-level accuracy and the global IoU optimization. Through the above process, the method can realize the dense restoration of the semantic occupancy scene while guaranteeing the advantages of sparse representation, and through multiple rounds of iterative optimization, the final Gaussian representation of the scene and the semantic occupancy prediction value of the scene calculated by the final Gaussian attribute are obtained.

[0170] By establishing a Gaussian scale allocation mechanism driven by semantic importance and spatial distance, the collaborative optimization of key target refinement and efficient expression of background area is realized, and at the same time, the key object protection rule and the scale recalibration strategy triggered by dynamic gradient are combined, which significantly improves the geometric structure restoration ability and semantic occupancy prediction accuracy of complex scenes, while ensuring the calculation efficiency of the vehicle-mounted platform, and provides high-confidence three-dimensional environment perception results for automatic driving decision.

[0171] Compared with the prior art, the method has the following beneficial effects:

[0172] 1. The scene element semantic importance category is embedded in the Gaussian scale control system together with the spatial distance information, breaking through the traditional Gaussian method of homogeneous resource allocation, significantly improving the geometric detail retention ability of key targets, and improving the cognitive and decision-making accuracy of complex driving scenes. Through the semantic-distance adaptive dynamic 3D Gaussian scene representation method, a three-dimensional environment model with geometric details and semantic information can be constructed in real time, enabling the automatic driving system to more accurately identify key targets in complex scenes such as intersection blind areas and dense traffic, significantly reducing the risk of perception misjudgment. It can improve the calculation efficiency of three-dimensional scene representation: relying on the semantic-distance coupled Gaussian resource allocation strategy, it realizes intelligent allocation of computing resources, providing efficient and real-time three-dimensional scene understanding capability for vehicle platforms.

[0173] 2. For the degradation problem of small objects at a long distance due to distance penalty, a semantic-driven hierarchical correction rule is established to ensure that key targets obtain maximum geometric expression accuracy at any distance, solving the perception reliability bottleneck of long-tail scenes for autonomous driving.

[0174] 3. After each round of Gaussian attribute iteration optimization, the Gaussian scale is fine-tuned based on the updated semantic attributes and spatial positions, and the representation granularity of key targets is adaptively corrected through a semantic-weighted gradient triggering strategy, effectively maintaining the structural continuity of complex scene reconstruction while improving the topological accuracy of voxel-level semantic occupancy prediction, solving the local geometric distortion problem caused by homogeneous representation in traditional methods, and significantly improving the geometric structure consistency and voxel-level semantic occupancy prediction accuracy of complex scenes.

[0175] As shown in Figure 3 , the scene semantic occupancy prediction system based on semantic-distance adaptive Gaussian provided by the embodiments of the present application comprises:

[0176] A point cloud collection module 10 is configured to perform feature extraction and point cloud collection on the surround view image sequence around the vehicle to obtain a feature map and an original sample point cloud set.

[0177] A sample point analysis module 20 is configured to input the feature map into a double-branch prediction network to obtain semantic information and position information of the original sample points.

[0178] A sample point screening module 30 is configured to screen the original sample point cloud set according to the semantic information and position information of the original sample point cloud set, using the farthest point sampling method and the dynamic semantic perception sampling method in cooperation, as the initial Gaussian anchor points.

[0179] The anchor point level division module 40 is configured to determine the Euclidean distances between all initialized Gaussian anchor points and the vehicle according to the position information of the initialized Gaussian anchor points, wherein the maximum and minimum values of the Euclidean distances are taken as the maximum Euclidean distance and the minimum Euclidean distance respectively; and to determine the level to which each initialized Gaussian anchor point is assigned according to the Euclidean distance between each initialized Gaussian anchor point and the vehicle, the maximum Euclidean distance and the minimum Euclidean distance.

[0180] The Gaussian attribute initialization module 50 is configured to adaptively adjust a preset basic scale according to the level and semantic information of the initialized Gaussian anchor points to obtain a corrected scale, and to obtain initial four-dimensional attributes of a Gaussian ellipsoid based on the initialized Gaussian anchor points after adding a random rotation vector.

[0181] The optimization prediction module 60 is configured to perform multiple rounds of dynamic adjustment on the initial four-dimensional attributes of the Gaussian ellipsoid using a Gaussian attribute iterative model, project the initial four-dimensional attributes onto a target voxel grid, and generate a dense 3D semantic occupancy prediction.

[0182] In this embodiment, the scene semantic occupancy prediction system based on the semantic-distance adaptive Gaussian has similar beneficial effects to the scene semantic occupancy prediction method based on the semantic-distance adaptive Gaussian, and thus repeated description is omitted herein.

[0183] The electronic device provided in the embodiments of the present application includes a memory and a processor; the memory is configured to store a computer program; and the processor is configured to implement the scene semantic occupancy prediction method based on the semantic-distance adaptive Gaussian when the computer program is executed.

[0184] The computer readable storage medium provided in the embodiments of the present application has the computer program stored thereon, and the computer program is configured to implement the scene semantic occupancy prediction method based on the semantic-distance adaptive Gaussian when executed by a processor.

[0185] In this embodiment, the electronic device and the computer readable storage medium have similar beneficial effects to the scene semantic occupancy prediction method based on the semantic-distance adaptive Gaussian, and thus repeated description is omitted herein.

[0186] An electronic device that can be a server or a client of the present application will now be described, which is an example of a hardware device that can be applied to aspects of the present application. The electronic device is intended to represent a wide variety of digital electronic computing devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computing devices. The electronic device can also represent a wide variety of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0187] The electronic device includes a computing unit that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The computing unit, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0188] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM). In the present application, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application. In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0189] It is to be noted that, in the present text, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0190] The above examples are merely intended for describing the technical solutions of the present application, but not for limiting the same; even though the present application has been described in detail with reference to the foregoing examples, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing examples can be modified, or some of the technical features can be replaced equivalently; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian, characterized in that, The application relates to a method for generating a 3D semantic occupancy prediction of a vehicle surrounding, comprising: extracting features and collecting point clouds from a sequence of surround view images of the vehicle surrounding to obtain a feature map and an original sampled point cloud set; inputting the feature map into a double-branch prediction network to obtain semantic information and position information of the original sampled points; screening the original sampled point cloud set according to the semantic information and the position information of the original sampled point cloud set by using a farthest point sampling method and a dynamic semantic perception sampling method to obtain initial Gaussian anchor points; determining the Euclidean distances between all initial Gaussian anchor points and the vehicle according to the position information of the initial Gaussian anchor points, wherein the maximum and minimum values of the Euclidean distances are taken as a maximum Euclidean distance and a minimum Euclidean distance respectively; determining the level to which each initial Gaussian anchor point is assigned according to the Euclidean distance of each initial Gaussian anchor point from the vehicle, the maximum Euclidean distance and the minimum Euclidean distance; adapting a preset basic scale according to the level and the semantic information of the initial Gaussian anchor points to obtain a corrected scale, and adding a random rotation vector to obtain initial four-dimensional attributes of a Gaussian ellipsoid based on the initial Gaussian anchor points; performing multi-round dynamic adjustment on the initial four-dimensional attributes of the Gaussian ellipsoid by using a Gaussian attribute iterative model, projecting the initial four-dimensional attributes onto a target voxel grid, and generating a dense 3D semantic occupancy prediction.

2. The semantic-distance adaptive Gaussian based scene semantic occupancy prediction method of claim 1, wherein, The screening of the original sampled point cloud set according to the semantic information and the position information of the original sampled point cloud set by using the farthest point sampling method and the dynamic semantic perception sampling method to obtain the initial Gaussian anchor points comprises: identifying object types contained in a scene according to the semantic information of the original sampled points, dividing the points in the original sampled point cloud set into foreground sampled points and background sampled points, and removing points with an empty semantic category to obtain an original sampled non-empty point set, wherein the foreground sampled points correspond to dynamic object categories, and the background sampled points correspond to static environmental element categories; taking the proportion of the number of the foreground sampled points in the total number of the original sampled non-empty points as a foreground object proportion, determining the proportion weights of the farthest point sampling and the semantic perception sampling in the total sampling budget according to the foreground object proportion, and determining the sampling quota of the farthest point sampling and the sampling quota of the semantic perception sampling; in the farthest point sampling stage, performing a farthest point sampling algorithm on the original sampled non-empty point set according to the sampling quota of the farthest point sampling to determine a first Gaussian anchor point set; in the semantic perception sampling stage, determining a supplementary sampling quota allocated to each foreground object according to preset priority weights corresponding to the types of the foreground objects, the number of the foreground objects and the sampling quota of the semantic perception sampling; determining a sampling weight of an original sampled non-empty point that is not selected in the farthest point sampling stage according to an occupancy probability prediction value and a local density of the original sampled non-empty point; and determining a Gaussian anchor point set supplemented by each foreground object according to the sampling weight and the sampling quota, wherein the Gaussian anchor point sets supplemented by all the foreground objects constitute a second Gaussian anchor point set; fusing the first Gaussian anchor point set determined in the farthest point sampling stage and the second Gaussian anchor point set determined in the semantic perception sampling stage to form a final initial Gaussian anchor point set.

3. The semantic-distance adaptive Gaussian based scene semantic occupancy prediction method of claim 1, wherein, The method further comprises the following steps after determining the level to which each initialized Gaussian anchor is assigned according to the Euclidean distance of each initialized Gaussian anchor from the vehicle, the maximum Euclidean distance and the minimum Euclidean distance: determining the semantic importance factor of the initialized Gaussian anchor according to the semantic information of the initialized Gaussian anchor and a preset semantic importance division table; adjusting the level of each initialized Gaussian anchor according to the semantic importance factor, the level offset and a level correction model to determine the corrected level of the initialized Gaussian anchor.

4. The semantic-distance adaptive Gaussian based scene semantic occupancy prediction method of claim 3, wherein, The method further comprises the following steps of adaptively adjusting the preset basic scale according to the level and the semantic information of the initialized Gaussian anchor to obtain a corrected scale, and adding a rotation vector to obtain the initial four-dimensional attribute of the Gaussian ellipsoid based on the initialized Gaussian anchor, comprising: calling the corresponding preset basic semantic scale according to the semantic information of the initialized Gaussian anchor; inputting the total number of levels, the corrected level of the initialized Gaussian anchor and the preset basic semantic scale into a scale correction model to obtain the corrected scale of the Gaussian ellipsoid based on the initialized Gaussian anchor; randomly initializing the rotation vector attribute of each Gaussian ellipsoid, adding a rotation vector to each initialized Gaussian anchor to obtain the initial four-dimensional attribute of the optimized Gaussian ellipsoid.

5. The semantic-distance adaptive Gaussian based scene semantic occupancy prediction method of claim 4, wherein, The method further comprises the following steps of using a Gaussian attribute iterative model to perform multi-round dynamic adjustment on the initial four-dimensional attribute of the Gaussian ellipsoid, projecting to a target voxel grid to generate a dense 3D semantic occupancy prediction, comprising: dynamically adjusting the initial four-dimensional attribute of the Gaussian ellipsoid using the Gaussian attribute iterative model to obtain the four-dimensional iterative attribute of the Gaussian ellipsoid; determining the joint gradient norm according to the position, the corrected scale and the rotation vector attribute in the four-dimensional iterative attribute; determining the level migration value of each Gaussian ellipsoid according to the semantic importance factor and the joint gradient norm; determining the migration threshold value of each Gaussian ellipsoid according to the Euclidean distance, the maximum Euclidean distance and the corrected level corresponding to the initialized Gaussian anchor; if the level migration value is greater than the migration threshold value, then determining the level after migration according to the migration threshold value, the corrected level and the total number of levels; inputting the total number of levels, the level after migration and the preset basic semantic scale into the scale correction model to obtain the optimized scale of the Gaussian ellipsoid and update the four-dimensional iterative attribute of the Gaussian ellipsoid; performing cyclic dynamic adjustment on the four-dimensional iterative attribute of the Gaussian ellipsoid using the Gaussian attribute iterative model until the iteration termination condition is met to obtain the four-dimensional iterative attribute of the iteratively optimized Gaussian ellipsoid.

6. The semantic-distance adaptive Gaussian based scene semantic occupancy prediction method of claim 1, wherein, The method further comprises the following steps of extracting features and collecting point clouds from the surround view image sequence around the vehicle to obtain a feature map and an original sampling point cloud set, comprising: extracting features from the surround view image sequence around the vehicle using a feature extraction network to obtain a feature map; emitting a ray from the camera center pixel by pixel on the feature map, and performing interval sampling on the ray at equal sampling intervals within a preset sampling depth range to obtain an original sampling point cloud set.

7. The semantic-distance adaptive Gaussian based scene semantic occupancy prediction method of claim 1, wherein, The method further comprises the following steps before inputting the feature map into the double-branch prediction network to obtain the semantic information and the position information of the original sampling points: inputting the feature map into the double-branch prediction network respectively to obtain the predicted occupancy distribution and the predicted semantic distribution of the sampling points; The real occupancy label and the real semantic label of each sampling point are queried from the 3D voxel occupancy ground truth field, and a joint loss function is used to determine a loss value in combination with a predicted occupancy distribution and a predicted semantic distribution; The dual-branch prediction network is adjusted according to the loss value to determine the trained dual-branch prediction network.

8. A semantic-distance adaptive Gaussian based scene semantic occupancy prediction system, comprising: The method comprises the following steps: The point cloud acquisition module is configured to extract features and collect point clouds from the surround view image sequence around the vehicle to obtain a feature map and an original sampling point cloud set; The sampling point analysis module is configured to input the feature map into the dual-branch prediction network to obtain semantic information and position information of the original sampling points; The sampling point screening module is configured to screen the original sampling point cloud set according to the semantic information and the position information of the original sampling point cloud set, in cooperation with a farthest point sampling method and a dynamic semantic perception sampling method, to obtain initial Gaussian anchor points; The anchor point level division module is configured to determine the Euclidean distance between all initial Gaussian anchor points and the vehicle according to the position information of the initial Gaussian anchor points, wherein the maximum and minimum values of the Euclidean distance are taken as the maximum Euclidean distance and the minimum Euclidean distance, respectively; and the anchor point level division module is further configured to determine the level to which each initial Gaussian anchor point is assigned according to the Euclidean distance of each initial Gaussian anchor point from the vehicle, the maximum Euclidean distance, and the minimum Euclidean distance; The Gaussian attribute initialization module is configured to adaptively adjust a preset basic scale according to the level and semantic information of the initial Gaussian anchor points to obtain a corrected scale, and to obtain initial four-dimensional attributes of a Gaussian ellipsoid based on the initial Gaussian anchor points after adding a random rotation vector; The optimization prediction module is configured to use a Gaussian attribute iterative model to dynamically adjust the initial four-dimensional attributes of the Gaussian ellipsoid for multiple rounds, project the initial four-dimensional attributes onto a target voxel grid, and generate a dense 3D semantic occupancy prediction.

9. An electronic device, comprising: The memory and the processor are included; The memory is configured to store a computer program; The processor is configured to implement the semantic-distance adaptive Gaussian based scene semantic occupancy prediction method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the semantic-distance adaptive Gaussian based scene semantic occupancy prediction method of any one of claims 1 to 7.

Images