Visual occupancy semantic prediction method and device, electronic equipment and storage medium

Abstract

The invention provides a visual occupancy semantic prediction method and device, electronic equipment and a storage medium, and the method comprises the steps: calculating a first fusion feature of first 3D voxel reference points which are uniformly distributed according to a first density in a peripheral 3D detection space according to the image feature information of each visual angle image of the peripheral environment of an automatic driving vehicle; converting the first fusion feature into a first aerial view feature, predicting the visual occupation semantics of each first aerial view grid, and screening out a 3D key detection space needing refined prediction in the 3D detection space; according to the image feature information, calculating a second fusion feature of second 3D voxel reference points uniformly distributed according to a second density in the 3D key detection space, the second density being greater than the first density; and at least according to the second fusion feature, predicting to obtain the visual occupation semantics of the second 3D voxel reference point. Therefore, the sensing coverage capability of a long-distance scene is guaranteed, the fine sensing requirement is met, and the performance efficiency balance is realized.

Description

Technical Field

[0001] This disclosure relates to the field of visual inspection technology, and in particular to a visual occupancy semantic prediction method, apparatus, electronic device, and storage medium. Background Technology

[0002] Visual semantic occupancy technology is a core technology in the field of autonomous driving perception. It predicts the spatial occupancy status of the area around a vehicle through visual occupancy networks, representing the distribution of all common obstacles in the vehicle's surrounding environment. Visual semantic occupancy methods are not limited by target categories, and compared to traditional target detection technologies that can only identify targets of specified categories, they possess more comprehensive environmental perception capabilities, helping vehicles achieve safe decision-making and control.

[0003] However, due to limitations in the computing power of automotive chips, visual semantic occupancy technology faces the challenge of balancing performance and computational efficiency: First, the detection distance is limited. Long-distance occupancy status perception required for high-speed driving requires a large amount of computing resources, and existing models cannot extend the perception range with limited computing power. Second, the detection resolution is limited. Scenarios such as parking and low-speed obstacle avoidance require high occupancy grid resolution, but high resolution will drastically increase the computational load. Third, the detection height is limited. Uneven road surfaces such as slopes require a larger vertical detection range, which also increases the computational load. Summary of the Invention

[0004] In view of this, the present disclosure provides a visual occupancy semantic prediction method, the method comprising: Based on the image feature information of the images of the autonomous vehicle's surrounding environment from various perspectives, the first fusion feature of the first 3D voxel reference points uniformly distributed at the first density in the 3D detection space around the vehicle is calculated. The first fusion feature is converted into a first bird's-eye view feature, and the visual occupancy semantics of each first bird's-eye view grid is predicted based on the first bird's-eye view feature. Then, based on the visual occupancy semantics, the 3D key detection space that needs further refined prediction is selected in the 3D detection space. Based on the image feature information, a second fusion feature of a second 3D voxel reference point uniformly distributed at a second density within the 3D key detection space is calculated; wherein the second density is greater than the first density; Based at least on the second fusion feature, the visual occupancy semantics of the second 3D voxel reference point are predicted.

[0005] Optionally, predicting the visual occupancy semantics of the second 3D voxel reference point based at least on the second fusion feature includes: Based on the grid coordinates of the second 3D voxel reference point located in the first bird's-eye view grid, interpolation calculation is performed on the first bird's-eye view feature to obtain the second bird's-eye view feature of the second 3D voxel reference point; The second fused feature and the second bird's-eye view feature are concatenated to obtain the concatenated feature; The concatenated features are input into a multilayer perceptron to obtain the visual occupancy semantics of the second 3D voxel reference point predicted by the multilayer perceptron.

[0006] Optionally, the occupancy semantics of the bird's-eye view grid includes the occupancy probability of the bird's-eye view grid, and the step of filtering out the key 3D detection spaces that need further refined prediction in the 3D detection space based on the visual occupancy semantics includes: Based on the occupancy semantics, the first bird's-eye view grid with an occupancy probability greater than a preset threshold is selected; Determine the grid coordinates of the selected first bird's-eye view grid within the bird's-eye view plane; Based on the height range of the 3D detection space, the 3D detection space covered by the grid coordinates within the height range is selected as the key 3D detection space that needs further refined prediction within the 3D detection space.

[0007] Optionally, the image feature information includes image feature values ​​and a predicted depth range. The step of calculating the first fusion feature of a first 3D voxel reference point uniformly distributed at a first density within the 3D detection space around the vehicle, based on the image feature information from various perspectives of the autonomous vehicle's surrounding environment, includes: Projecting the first 3D voxel reference points, which are uniformly distributed in the first density within the 3D detection space, onto the image feature information of each viewpoint image, yields the predicted depth range of the projection position of each first 3D voxel reference point. For each viewpoint image, based on the predicted depth range where the projection position is located, the image feature value of the viewpoint image is interpolated to obtain the image feature of each first 3D voxel reference point under the viewpoint image. The image features of each first 3D voxel reference point under all viewpoints are fused to obtain the first fused feature of the first 3D voxel reference point.

[0008] Optionally, the image feature information further includes the prediction probability corresponding to each prediction depth interval. The step of fusing the image features of each first 3D voxel reference point under all viewpoint images to obtain the first fused feature of the first 3D voxel reference point includes: Based on the prediction probability corresponding to each prediction depth interval, the image features of each first 3D voxel reference point under each viewpoint image are subjected to probability weighting to obtain the image features of the first 3D voxel reference point after weighting. The image features of the first 3D voxel reference point after weighted processing under all viewpoints are fused to obtain the first fused feature of the first 3D voxel reference point.

[0009] Optionally, the step of converting the first fused feature into a first bird's-eye view feature and predicting the visual occupancy semantics of each first bird's-eye view grid based on the first bird's-eye view feature includes: The first fused features are grouped according to the grid coordinates of the first bird's-eye view grid; wherein, the first fused features in the same group are located in different height dimensions, but have the same grid coordinates in the planar dimension. The first fused feature is then fused again along the height dimension according to the grid coordinates of the group, to obtain the grid features of each group of grid coordinates after fusion; Based on the grid features of each set of grid coordinates, predict the visual occupancy semantics of each first bird's-eye view grid.

[0010] Optionally, the first fused features are further fused along the height dimension according to the grouped grid coordinates to obtain the grid features of each group of grid coordinates after fusion, including: The first fused features are then fused again along the height dimension according to the grid coordinates of the group; During feature fusion of any set of grid coordinates, if the height dimension does not have at least one first fusion feature indicating that a 3D voxel is occupied, then it is determined that the set of grid coordinates does not have visual occupancy. Otherwise, it is determined that the set of grid coordinates has visual occupancy, and the semantic category of the set of grid coordinates is the semantic category indicated by the first fusion feature with the largest height dimension among the at least one first fusion feature.

[0011] This disclosure also provides a visual occupancy semantic prediction device, the device comprising: The first computing unit is used to calculate the first fusion feature of the first 3D voxel reference points uniformly distributed at the first density in the 3D detection space around the vehicle based on the image feature information of the images of the surrounding environment of the autonomous vehicle from various perspectives. The filtering unit is used to convert the first fused feature into a first bird's-eye view feature, predict the visual occupancy semantics of each first bird's-eye view grid based on the first bird's-eye view feature, and then filter out the 3D key detection space that needs further refined prediction in the 3D detection space based on the visual occupancy semantics. The second calculation unit is used to calculate, based on the image feature information, a second fusion feature of a second 3D voxel reference point uniformly distributed at a second density within the 3D key detection space; wherein the second density is greater than the first density. The prediction unit is used to predict the visual occupancy semantics of the second 3D voxel reference point based at least on the second fusion feature.

[0012] This disclosure also provides an electronic device, including a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the above method by invoking the machine-readable instructions.

[0013] This disclosure also provides a machine-readable storage medium storing machine-readable instructions that, when called and executed by a processor, implement the above-described method.

[0014] Therefore, in the technical solution disclosed herein, firstly, based on the image feature information of images from various perspectives of the vehicle surrounding environment, a first fusion feature of 3D voxel reference points uniformly distributed in the 3D detection space around the vehicle is calculated; secondly, the first fusion feature is converted into a first bird's-eye view feature, and the visual occupancy semantics of each first bird's-eye view grid is predicted accordingly, thereby filtering out the 3D key detection space that needs to be refined for prediction (such as the high-probability occupancy area of ​​the first bird's-eye view grid); then, also based on the image feature information, a second fusion feature of second 3D voxel reference points uniformly distributed and with higher density in the 3D key detection space is calculated, and finally, at least the visual occupancy semantics of the second 3D voxel reference points are predicted based on the second fusion feature.

[0015] Through the above methods, the technical solution disclosed herein "allocates computing power on demand" and adopts a two-stage strategy of "global coarse prediction + local fine prediction" to achieve visual occupancy semantic prediction: firstly, by designing global low-density voxel reference points and converting 3D fusion features into 2D bird's-eye view features, the computational overhead is significantly reduced, ensuring the perception coverage capability of distant scenes; then, by focusing 3D key detection space with local high-density voxels, the detection resolution of 3D key detection space is improved, meeting the needs of fine perception and achieving a balance between performance and computational efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a visual occupancy semantic prediction method as an exemplary embodiment; Figure 2 This is a schematic diagram illustrating a training method for a visual occupancy semantic prediction model, as shown in an exemplary embodiment. Figure 3 This is an exemplary embodiment illustrating reasoning using a visual occupancy semantic prediction model; Figure 4 This is an exemplary embodiment illustrating the hardware structure of an electronic device; Figure 5 This is a block diagram illustrating a visual occupancy semantic prediction device, as shown in an exemplary embodiment. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0019] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this disclosure in other embodiments. In some other embodiments, the methods may include more or fewer steps than those described in this disclosure. Furthermore, a single step described in this disclosure may be broken down into multiple steps in other embodiments; and multiple steps described in this disclosure may be combined into a single step in other embodiments.

[0020] The present disclosure will now be described through specific embodiments and in conjunction with specific application scenarios.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating a visual occupancy semantic prediction method as an exemplary embodiment. The method may perform the following steps: Step 102: Based on the image feature information of the images of the autonomous vehicle's surrounding environment from various perspectives, calculate the first fusion feature of the first 3D voxel reference points that are uniformly distributed at the first density in the 3D detection space around the vehicle.

[0022] For example, in a high-speed driving scenario of an autonomous vehicle, the visual occupancy perception system (hereinafter referred to as the "system") in the autonomous driving domain controller acquires images of the vehicle's surrounding environment from various perspectives captured by onboard surround-view cameras (such as six cameras: front-view, side-view, and rear-view). These images are then input into a backbone network and a feature pyramid network to extract image feature maps (i.e., image feature information), including image feature values ​​and predicted depth. Next, based on the vehicle's coordinate system, the system generates first 3D voxel reference points within a 3D detection space covering 100 meters in front, 25 meters to the left and right, and 5 meters in height, with a point interval of 0.5 meters (i.e., the first density). Then, using the intrinsic and extrinsic parameter matrices of the onboard surround-view cameras, the system projects the first 3D voxel reference points onto the image feature maps of each perspective image, determining the projection depth of each first 3D voxel reference point. Finally, the system uses bilinear interpolation to fuse image features from all perspectives at each first 3D voxel reference point location, obtaining the first fused feature for each first 3D voxel reference point.

[0023] The "first 3D voxel reference point" refers to a regular 3D grid point with low spatial resolution (i.e., low density) set up within the large-scale 3D detection space around the vehicle for preliminary and coarse environmental perception. Its function is to construct a global, low-computational-overhead overview of the environment. The system can obtain several 3D voxels by sampling a regular grid within the cuboid space containing the "vehicle-surrounding 3D detection space" at fixed voxel sizes (e.g., Δx1=0.8m, Δy1=0.8m, Δz1=0.4m), and take the center point of each 3D voxel as the "first 3D voxel reference point," or obtain the "first 3D voxel reference point" through other sampling methods and machine learning algorithms. "Predicted depth" can be a predicted depth range or a predicted depth value. The "first fusion feature" refers to the comprehensive feature vector formed by aggregating 2D image features from multiple camera perspectives to a single 3D voxel reference point through geometric projection and feature sampling methods, which can avoid feature loss caused by single-view occlusion.

[0024] In one embodiment, the image feature information includes image feature values ​​and a predicted depth range. The step of calculating a first fusion feature of a first 3D voxel reference point uniformly distributed at a first density within the 3D detection space around the autonomous vehicle, based on the image feature information of images from various perspectives of the vehicle's surrounding environment, includes: projecting the first 3D voxel reference points uniformly distributed at a first density within the 3D detection space onto the image feature information of each perspective image to obtain the predicted depth range of the projection position of each first 3D voxel reference point; for each perspective image, interpolating the image feature values ​​of that perspective image based on the predicted depth range of the projection position to obtain the image feature of each first 3D voxel reference point under that perspective image; and fusing the image features of each first 3D voxel reference point under all perspective images to obtain the first fusion feature of the first 3D voxel reference point.

[0025] For example, the system inputs images from various viewpoints into the backbone network and the feature pyramid network, extracting image feature information (usually in the form of image feature maps) from each viewpoint, including image feature values ​​and predicted depth ranges. Then, the system projects the first 3D voxel reference point onto the image feature maps of each viewpoint image using the intrinsic and extrinsic parameter matrices of the vehicle-mounted surround-view camera, predicting the projection depth range for each first 3D voxel reference point. For example, predicting that a certain 3D voxel reference point is projected into the 25th projection depth range corresponds to 39.06–40.62 meters. Finally, the system performs interpolation calculations on the 3D voxel reference point within this projection depth range to obtain the first fusion feature of the 3D voxel reference point, where the first fusion feature contains the image features of each viewpoint image at that 3D voxel reference point.

[0026] The depth interval distribution divides the entire predictable depth range into n equally wide predicted depth intervals. Assuming the farthest predicted distance is 100 meters and n=64, the width of each predicted depth interval is 1.5625 meters. "Image feature information" refers to high-dimensional vectors representing the semantic content of local regions, extracted layer by layer from the original image by a deep neural network. These vectors typically exist in the form of multi-channel two-dimensional tensors (i.e., feature maps). Each vector at a spatial location encodes the texture, edges, components, and even object-level semantic information of the corresponding image region. Furthermore, each spatial location of the feature map is bound to a corresponding predicted depth interval, allowing image feature information to simultaneously carry semantic features and depth information, providing a foundation for projection matching and feature interpolation of 3D voxel reference points.

[0027] In one embodiment shown, the image feature information further includes the prediction probability corresponding to each prediction depth interval. The step of fusing the image features of each first 3D voxel reference point under all viewpoint images to obtain the first fused feature of the first 3D voxel reference point includes: performing probability weighting processing on the image features of each first 3D voxel reference point under each viewpoint image according to the prediction probability corresponding to each prediction depth interval to obtain the weighted image features of the first 3D voxel reference point; and performing feature fusion on the weighted image features of the first 3D voxel reference point under all viewpoint images to obtain the first fused feature of the first 3D voxel reference point.

[0028] For example, the system inputs images from various viewpoints into the backbone network and the feature pyramid network, extracting image feature information (usually in the form of image feature maps) from each viewpoint, including image feature values, predicted depth intervals, and the prediction probabilities corresponding to each predicted depth interval. Then, the system projects the first 3D voxel reference point onto the image feature maps of each viewpoint image using the intrinsic and extrinsic parameter matrices of the vehicle-mounted surround-view camera, predicting the projection depth interval and prediction probability for each first 3D voxel reference point. For example, it predicts that a certain 3D voxel reference point has an 80% probability of being projected into the 25th projection depth interval, corresponding to 39.06–40.62 meters, and a 20% probability of being projected into the 26th projection depth interval, corresponding to 40.62–42.18 meters. Finally, the system performs interpolation calculations on the 3D voxel reference point in both projection depth intervals, and weights the interpolation result of each projection depth interval with the predicted probability of that projection depth interval (80% probability for the 25th projection depth interval and 20% probability for the 26th projection depth interval). Then, the weighted results of the two projection depth intervals are fused to obtain the first fused feature of the 3D voxel reference point.

[0029] Here, "predicted probability corresponding to the predicted depth interval" refers to the confidence distribution of the prediction model within the system for each 3D voxel reference point falling within various discrete depth intervals, reflecting the uncertainty of depth estimation. "Probability weighting" refers to using this probability as weight to weight the image features sampled under different depth interval assumptions, thereby generating a soft feature representation that integrates multiple depth possibilities, avoiding feature distortion caused by rigidly selecting a single depth. "Feature fusion" refers to further aggregating the probability-weighted multi-view features (such as summation, convolution after concatenation, or attention weighting) to obtain a more robust comprehensive representation of the same 3D voxel reference point.

[0030] Step 104: Convert the first fusion feature into a first bird's-eye view feature, and predict the visual occupancy semantics of each first bird's-eye view grid based on the first bird's-eye view feature. Then, based on the visual occupancy semantics, select the key 3D detection spaces that need further refined prediction in the 3D detection space.

[0031] For example, the system maps all first fused features to corresponding first bird's-eye view grids according to their XY positions in the vehicle coordinate system. It then aggregates multiple first fused features (from different heights Z) falling into the same grid coordinates along the height dimension (e.g., summation, max pooling, or other aggregation operations). This integrates 3D features at different heights under the same plane grid coordinates into a single 2D grid feature (i.e., the first bird's-eye view feature), which combines feature information from the corresponding plane position across the entire height range. Finally, based on the predicted visual occupancy semantics, the system filters out first bird's-eye view grids with occupancy, and identifies the 3D regions covered by these grids within the full height range of the 3D detection space as the key 3D detection spaces requiring further refined prediction.

[0032] The "first bird's-eye view feature" is a two-dimensional feature map obtained by aggregating the first fused features in the vertical (Z-axis) direction. It simplifies the complex 3D perception problem into a 2D plane problem. This feature has eliminated voxel-level redundant details and has clear spatial semantic localization capabilities, reducing the computational complexity of subsequent processing. The "3D key detection space" is a local 3D region that is dynamically selected based on the coarse prediction results of the first stage and is crucial for autonomous driving decision-making and control. It is usually the area near the vehicle that is predicted to have obstacles.

[0033] In one embodiment shown, the occupancy semantics of the bird's-eye view grid includes the occupancy probability of the bird's-eye view grid. The step of filtering out key 3D detection spaces requiring further refined prediction within the 3D detection space based on the visual occupancy semantics includes: filtering out first bird's-eye view grids whose occupancy probability is greater than a preset threshold based on the occupancy semantics; determining the grid coordinates of the filtered first bird's-eye view grids in the bird's-eye view plane; and, based on the height range of the 3D detection space, using the 3D detection space covered by the grid coordinates within the height range as the key 3D detection spaces requiring further refined prediction within the 3D detection space.

[0034] For example, the system inputs the features of the first bird's-eye view into a pre-trained prediction model. The model then infers and outputs the visual occupancy semantics of each first bird's-eye view grid, clarifying whether the corresponding planar location of the grid is occupied, the occupancy probability (e.g., a value between 0 and 1, with higher values ​​indicating a greater likelihood of an obstacle), and the specific semantic category (obstacle type, such as "vehicle" or "pedestrian"). Subsequently, the system sets a preset threshold (e.g., 0.7) to filter out first bird's-eye view grids with an occupancy probability greater than this threshold—these grids are identified as key areas with potential obstacles. Next, the system determines the specific grid coordinates of the selected key grids within the bird's-eye view plane (XY dimension), such as (X=10, Y=8), (X=11, Y=8), etc. Finally, the system combines the height range of the 3D detection space (e.g., 0-5 meters) and identifies the 3D space covered by these planar grid coordinates within this height range (i.e., X and Y coordinates correspond to key grids, and the Z coordinate covers the complete 3D area from 0 to 5 meters) as the 3D key detection space requiring further refined prediction.

[0035] The "bird's-eye view grid" refers to a representation method that divides the 3D detection space into a series of two-dimensional grid units when viewed from a top perspective, and assigns corresponding occupancy information or semantic information to each unit. Using the bird's-eye view grid can effectively simplify the complexity of 3D space, facilitating subsequent planning and decision-making algorithms. The "occupancy probability of the bird's-eye view grid" refers to a scalar output value corresponding to each first bird's-eye view grid. This value is a probabilistic prediction result normalized by an activation function, ranging from [0,1], representing the probability that at least one 3D voxel is occupied in the vertical direction within the horizontal region of that bird's-eye view grid.

[0036] In some alternative embodiments, the preset threshold is set as a learnable parameter, or is dynamically adjusted according to the real-time environment, such as light intensity and weather type (sunny / rainy / foggy), to adapt to the sensing sensitivity requirements under different working conditions.

[0037] In this embodiment, by filtering the first bird's-eye view grid with an occupancy probability greater than a preset threshold, false detections caused by low-altitude noise (such as ground reflection, raindrop splash, and artifacts caused by sensor jitter) are avoided, making it suitable for unstructured interference scenarios that frequently occur in urban roads.

[0038] In some embodiments, the screening process can also introduce spatial continuity constraints: for example, retaining only sample clusters with a connected region area ≥ 3×3 grids and eliminating isolated noise points, thereby improving the reliability and robustness of the screening results. This connectivity judgment is based on a four-neighbor or eight-neighbor connectivity algorithm, ensuring that the screened regions have a certain degree of spatial continuity and avoiding misjudgments caused by isolated noise. This strategy not only helps reduce unnecessary computational burden but also more accurately identifies important obstacle regions, such as dense pedestrian areas or groups of parked vehicles, thereby enhancing the perception capability of key areas.

[0039] In one embodiment shown, the step of converting the first fused feature into a first bird's-eye view feature and predicting the visual occupancy semantics of each first bird's-eye view grid based on the first bird's-eye view feature includes: grouping the first fused feature according to the grid coordinates of the first bird's-eye view grid; wherein the first fused features in the same group are located in different height dimensions, but have the same grid coordinates in the planar dimension; performing feature fusion again along the height dimension according to the grouped grid coordinates of the first fused feature to obtain the grid features of each group of grid coordinates after fusion; and predicting the visual occupancy semantics of each first bird's-eye view grid based on the grid features of each group of grid coordinates.

[0040] For example, the system first iterates through all the first fused features and groups them according to the planar (XY dimension) grid coordinates of the first bird's-eye view grid. The first fused features within the same group all correspond to the same planar grid coordinates, but come from different heights (Z dimension). That is, first fused features at different heights under the same planar position are grouped together. Subsequently, for each group of first fused features, the system performs feature fusion processing (such as summation, max pooling, or other aggregation operations) along the height dimension, integrating the three-dimensional features at different heights under the same planar grid coordinates into a single two-dimensional grid feature (i.e., the first bird's-eye view feature). This grid feature integrates the feature information of the corresponding planar position across the entire height range. Finally, the system inputs each group of grid features into a pre-trained prediction model, and through the prediction model, outputs the visual occupancy semantics of each first bird's-eye view grid, clarifying whether the planar position corresponding to the grid is occupied and the specific semantic category (obstacle type, such as "vehicle" or "pedestrian").

[0041] The "grouped grid coordinates" refers to classifying the first fused features according to the XY plane coordinates of the first bird's-eye view. Each group corresponds to multiple voxels at the same ground location but different heights (Z-axis). Further fusion along the height dimension aims to aggregate perception results in the vertical direction. The "grid coordinates of the first bird's-eye view grid" refers to the row and column indices of two-dimensional regular grid cells divided on the horizontal plane (XY plane) centered on the autonomous vehicle. Each grid cell has a fixed size (e.g., 0.2m×0.2m, 0.4m×0.4m, or 0.5m×0.5m), covering a preset detection range (e.g., 80m forward, 20m to the left and right, and 20m backward). This grid coordinate system is strictly aligned with the vehicle coordinate system, with the origin located at the center of the vehicle's rear axle, the positive X-axis pointing towards the front of the vehicle, and the positive Y-axis pointing towards the left side of the vehicle.

[0042] In some alternative embodiments, the grid coordinates can be partitioned in a non-uniform manner, for example, using a denser grid (0.2m) in the near-field region (0–10m) and a sparser grid (0.5m) in the far-field region (10–80m) to balance local accuracy and global efficiency. Alternatively, the coordinate system can be replaced with a polar coordinate grid or an adaptive curvature grid to adapt to non-planar road scenarios such as ramps and curves.

[0043] In one embodiment shown, the first fused feature is further fused along the height dimension according to the grouped grid coordinates to obtain the grid features of each group of grid coordinates after fusion. This includes: further fusion of the first fused feature along the height dimension according to the grouped grid coordinates; during the feature fusion process of any group of grid coordinates, if at least one first fused feature does not indicate that a 3D voxel is occupied in the height dimension, then it is determined that the group of grid coordinates does not have visual occupancy; otherwise, it is determined that the group of grid coordinates has visual occupancy, and the semantic category of the group of grid coordinates is the semantic category indicated by the first fused feature with the largest height dimension among the at least one first fused feature.

[0044] For example, the system first traverses all first fused features and groups them according to the planar (XY dimension) grid coordinates of the first bird's-eye view grid. The first fused features in the same group all correspond to the same planar grid coordinates, but come from different heights (Z dimension). That is, first fused features at different heights under the same planar position are grouped together. Subsequently, the system fuses the first fused features of each group again along the height dimension: if none of the first fused features in the height dimension of the group indicate that 3D voxels are occupied, it is directly determined that there is no visual occupation of the grid coordinates in this group; if at least one first fused feature in the group indicates that 3D voxels are occupied, it is determined that there is visual occupation of the grid coordinates in this group, and its semantic category is clearly defined as the semantic category (obstacle type, such as "vehicle" or "pedestrian") indicated by the first fused feature with the largest value in the height dimension among the first fused features indicating occupation. Finally, the grid features corresponding to each group of grid coordinates are obtained.

[0045] In this approach, if none of the voxels within a given bird's-eye view grid group are predicted to be occupied, the grid is considered free. If at least one voxel is predicted to be occupied, the grid is determined to be occupied, and the semantic category corresponding to the occupied voxel at its highest point is used as the final category (e.g., if the highest point is "pedestrian" and the lowest point is "curb," then "pedestrian" is chosen). This strategy effectively avoids low-level noise interference and highlights the most significant obstacles, thus balancing computational efficiency and semantic rationality.

[0046] Step 106: Based on the image feature information, calculate the second fusion feature of the second 3D voxel reference points uniformly distributed at the second density within the 3D key detection space; wherein the second density is greater than the first density.

[0047] For example, the first density is 1 3D voxel reference point per cubic meter (i.e., voxel side length 1 meter), and the second density is set to 4 3D voxel reference points per cubic meter (i.e., voxel side length 0.5 meters). The second density is 4 times the first density. The system first locks the 3D key detection space (such as a three-dimensional region with near-field X-dimensional coordinate range of 0-20 meters, Y-dimensional coordinate range of 0-20 meters, and Z-dimensional coordinate range of 0-5 meters), and uniformly generates second 3D voxel reference points within this region at intervals of 0.5 meters. Then, through the intrinsic and extrinsic parameter matrices of the vehicle-mounted surround-view camera, each second 3D voxel reference point is projected onto the image feature map (including image feature values ​​and predicted depth) of each viewpoint. The predicted depth corresponding to the projection position is matched, and bilinear interpolation is performed on the image feature values ​​to obtain the image features of the second 3D voxel reference point under each viewpoint image. Finally, the image features of the second 3D voxel reference point under all viewpoint images are fused to obtain the second fused feature of the second 3D voxel reference point.

[0048] The "second density" refers to the distribution density of the second 3D voxel reference points within the 3D key detection space. It is usually quantified as "number of voxels / unit volume" or "voxel side length" (the higher the density, the shorter the voxel side length). Technically, the density can be increased by adjusting the voxel resolution parameters, and the second density needs to be flexibly configured according to the vehicle-side computing power redundancy. The definitions of "second 3D voxel reference points" and "second fusion features" are similar to those of "first 3D voxel reference points" and "first fusion features" introduced earlier, but with higher spatial density, aiming to perform refined occupancy semantic prediction for key areas. The system can obtain several 3D voxels by performing regular grid sampling within the cuboid space where the "vehicle perimeter 3D detection space" is located, according to fixed voxel sizes (e.g., Δx2=0.4m, Δy2=0.4m, Δz2=0.2m), and take the center point of each 3D voxel as the "second 3D voxel reference point," or obtain the "second 3D voxel reference point" through other sampling methods and machine learning algorithms.

[0049] Step 108: Based at least on the second fusion feature, predict the visual occupancy semantics of the second 3D voxel reference point.

[0050] For example, the system inputs at least the second fused feature into a multilayer perceptron, which predicts and outputs the binarized occupancy state of each second 3D voxel reference point. The occupancy semantics of each second 3D voxel reference point follows the occupancy semantics of the bird's-eye view grid in which it resides. In the final occupancy result output by the system, the portion of the 3D detection range that does not belong to the 3D key detection space (usually the far-field region) uses the visual occupancy semantics of the bird's-eye view grid, while the portion of the 3D detection range that belongs to the 3D key detection space (usually the near-field key region) is filled with the visual occupancy semantics of the second 3D voxel reference points at a higher density. This achieves efficient environmental modeling that balances long-distance perception range with high near-field accuracy. Finally, the system fuses the visual occupancy semantics of the bird's-eye view grid with the visual occupancy semantics of the second 3D voxel reference points to form a complete occupancy map. The autonomous driving domain controller can then control the vehicle for autonomous driving based on the complete occupancy map.

[0051] The "multilayer perceptron" is a feedforward artificial neural network model that processes data through multiple layers of nonlinear transformations, making it suitable for classification and regression tasks. The second fusion feature fully encodes the geometric texture and image information of the second 3D voxel reference point's location in multi-view images; therefore, a binary prediction of whether the point is occupied can be output based solely on the second fusion feature itself. To improve prediction accuracy, the second fusion feature vector of each second 3D voxel reference point can also be fused with the bird's-eye view feature of that second 3D voxel reference point, and the multilayer perceptron uses the fused result for prediction.

[0052] In one embodiment shown, predicting the visual occupancy semantics of the second 3D voxel reference point based at least on the second fusion feature includes: interpolating the first bird's-eye view feature based on the grid coordinates of the second 3D voxel reference point located in the first bird's-eye view grid to obtain a second bird's-eye view feature of the second 3D voxel reference point; performing feature concatenation between the second fusion feature and the second bird's-eye view feature to obtain a concatenated feature; and inputting the concatenated feature into a multilayer perceptron to obtain the visual occupancy semantics of the second 3D voxel reference point predicted by the multilayer perceptron.

[0053] For example, during low-speed parking, the system generates a bird's-eye view feature with a resolution of 0.4 meters (i.e., the first bird's-eye view feature) in the first stage, and identifies the area within 5 meters around the vehicle containing obstacles as the 3D key detection space. Within this space, the system generates high-density second 3D voxel reference points at 0.1-meter intervals. For one of the second 3D voxel reference points P, its XY coordinates in the vehicle coordinate system are (2.37m, -1.15m), corresponding to a non-integer position in the first bird's-eye view grid. The system performs bilinear interpolation on the first bird's-eye view feature at this position to obtain the second bird's-eye view feature at point P. Simultaneously, the system performs feature extraction and depth prediction on multi-view images captured by the onboard camera, and then calculates the second fused feature at point P, which integrates the features of the images from various perspectives. Subsequently, the system splices the second bird's-eye view feature and the second fused feature along the channel dimension to form an enhanced joint representation, and inputs it into a multilayer perceptron to obtain the binarized probability of whether point P is occupied. Its semantic category is inherited from the semantic category of the bird's-eye view grid where the point is located (such as "vehicle" or "pedestrian").

[0054] Bilinear interpolation is a technique for estimating values ​​at non-integer coordinate locations. It estimates the value of a target point based on the four nearest known data points and their weights. When the second 3D voxel reference point P is mapped back to the first bird's-eye view grid, if it falls on a non-integer coordinate, bilinear interpolation is used to extract information from the surrounding four grid cells and generate the second bird's-eye view features corresponding to point P. This not only improves the accuracy of feature extraction but also ensures the consistency and continuity of features. Feature concatenation is the process of combining information from different sources or levels so that subsequent models can simultaneously utilize this information to make better decisions.

[0055] In this embodiment, by stitching the second bird's-eye view feature and the second fused feature along the channel dimension, the system can create a joint representation that includes local geometric details (provided by the second fused feature) and global contextual information (carried by the second bird's-eye view feature). This design helps to capture more information about the environmental structure and content, thereby enhancing the reliability and accuracy of the final prediction results.

[0056] To implement the above method, a prediction model needs to be trained. Please refer to [link / reference needed]. Figure 2 , Figure 2 This is a schematic diagram illustrating a training method for a visual occupancy semantic prediction model, as shown in an exemplary embodiment. Figure 2 As shown, the training process of the visual occupancy semantic prediction model takes multi-view images as input. First, feature extraction and depth prediction are performed to generate image features and corresponding depth information for each viewpoint. Then, these features are used to construct the first fusion feature in 3D space and converted into bird's-eye view features, thereby inferring the occupancy semantics of each bird's-eye view grid. Based on this, the system selects a key 3D detection space according to the occupancy semantics and generates a second 3D voxel reference point with higher density within this region. Combining this with the original image features, the second fusion feature is calculated, ultimately predicting a refined second 3D voxel occupancy semantics. Throughout the training process, the model simultaneously receives the bird's-eye view grid occupancy semantic ground truth value converted from the first 3D voxel occupancy semantic ground truth value, and the second 3D voxel occupancy semantic ground truth value as supervision signals. By jointly optimizing the comprehensive loss between coarse-grained and fine-grained prediction results and their corresponding ground truth values, the model parameters are iteratively updated until convergence, thus enabling the model to achieve a visual occupancy semantic prediction capability that balances perceptual coverage and local detail accuracy.

[0057] In some embodiments, considering the severe sample imbalance problem between occupied and unoccupied areas in the bird's-eye view grid, the loss function for occupancy semantics can be jointly calculated using cross-entropy loss and Dice loss (also known as "dice loss"). Cross-entropy loss is one of the most commonly used loss functions in multi-class classification tasks, used to measure the difference between the predicted probability distribution and the true label; while Dice loss is more sensitive to the foreground (i.e., occupied area) and can effectively alleviate the optimization bias caused by class imbalance.

[0058] Please see Figure 3 , Figure 3 This is an exemplary embodiment illustrating reasoning using a visual occupancy semantic prediction model. For example... Figure 3As shown, the system first receives multi-view images of the vehicle's surrounding environment collected by the autonomous vehicle, enters the first-stage model inference, extracts image features, predicts depth information, and determines the features after fusing the images from each viewpoint based on low-density voxel reference points. Finally, it converts these features into bird's-eye view features, thereby completing the bird's-eye view grid occupancy semantic prediction for the entire 3D detection space. Subsequently, based on the prediction results, the system filters out 3D spaces with potential obstacles and identifies them as key 3D detection spaces requiring further refined prediction. Next, the system inputs the high-density voxel reference points in this space, along with the original image features, depth prediction results, and corresponding bird's-eye view features, into the second-stage model for inference to obtain higher-resolution local occupancy semantic output. Finally, the global occupancy semantic prediction results from the first stage are fused with the refined local prediction results from the second stage to generate the final visual occupancy semantic map, achieving comprehensive and accurate perception modeling of the vehicle's surrounding environment and providing reliable support for the decision-making and control of the autonomous driving system.

[0059] Corresponding to the embodiments of the visual occupancy semantic prediction method described above, this disclosure also provides an embodiment of a visual occupancy semantic prediction device.

[0060] Please see Figure 4 , Figure 4 This is an exemplary embodiment illustrating the hardware structure of an electronic device. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, and non-volatile memory 410, and may also include other necessary hardware. One or more embodiments of this disclosure can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 410 into memory 408 and then runs it. Of course, besides software implementation, one or more embodiments of this disclosure do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0061] Please see Figure 5 , Figure 5 This is a block diagram illustrating a visual occupancy semantic prediction device as an exemplary embodiment. The visual occupancy semantic prediction device 500 can be applied to, for example... Figure 4 The illustrated electronic device is used to implement the technical solution of this disclosure. The device includes: The first computing unit 502 is used to calculate the first fusion feature of the first 3D voxel reference points uniformly distributed in the vehicle perimeter 3D detection space according to the image feature information of the images of the surrounding environment of the autonomous vehicle from various perspectives. The filtering unit 504 is used to convert the first fused feature into a first bird's-eye view feature, predict the visual occupancy semantics of each first bird's-eye view grid based on the first bird's-eye view feature, and then filter out the 3D key detection space that needs further refined prediction in the 3D detection space based on the visual occupancy semantics. The second calculation unit 506 is used to calculate, based on the image feature information, a second fusion feature of a second 3D voxel reference point uniformly distributed at a second density within the 3D key detection space; wherein the second density is greater than the first density. The prediction unit 508 is used to predict the visual occupancy semantics of the second 3D voxel reference point based at least on the second fusion feature.

[0062] In some embodiments, predicting the visual occupancy semantics of the second 3D voxel reference point based at least on the second fusion feature includes: Based on the grid coordinates of the second 3D voxel reference point located in the first bird's-eye view grid, interpolation calculation is performed on the first bird's-eye view feature to obtain the second bird's-eye view feature of the second 3D voxel reference point; The second fused feature and the second bird's-eye view feature are concatenated to obtain the concatenated feature; The concatenated features are input into a multilayer perceptron to obtain the visual occupancy semantics of the second 3D voxel reference point predicted by the multilayer perceptron.

[0063] In some embodiments, the occupancy semantics of the bird's-eye view grid includes the occupancy probability of the bird's-eye view grid, and the step of filtering out key 3D detection spaces requiring further refined prediction within the 3D detection space based on the visual occupancy semantics includes: Based on the occupancy semantics, the first bird's-eye view grid with an occupancy probability greater than a preset threshold is selected; Determine the grid coordinates of the selected first bird's-eye view grid within the bird's-eye view plane; Based on the height range of the 3D detection space, the 3D detection space covered by the grid coordinates within the height range is selected as the key 3D detection space that needs further refined prediction within the 3D detection space.

[0064] In some embodiments, the image feature information includes image feature values ​​and a predicted depth range. The step of calculating the first fusion feature of a first 3D voxel reference point uniformly distributed at a first density within the vehicle's 3D detection space based on the image feature information from various perspectives of the autonomous vehicle's surrounding environment includes: Projecting the first 3D voxel reference points, which are uniformly distributed in the first density within the 3D detection space, onto the image feature information of each viewpoint image, yields the predicted depth range of the projection position of each first 3D voxel reference point. For each viewpoint image, based on the predicted depth range where the projection position is located, the image feature value of the viewpoint image is interpolated to obtain the image feature of each first 3D voxel reference point under the viewpoint image. The image features of each first 3D voxel reference point under all viewpoints are fused to obtain the first fused feature of the first 3D voxel reference point.

[0065] In some embodiments, the image feature information further includes the prediction probability corresponding to each prediction depth interval, and the fusion of image features of each first 3D voxel reference point under all viewpoint images to obtain the first fused feature of the first 3D voxel reference point includes: Based on the prediction probability corresponding to each prediction depth interval, the image features of each first 3D voxel reference point under each viewpoint image are subjected to probability weighting to obtain the image features of the first 3D voxel reference point after weighting. The image features of the first 3D voxel reference point after weighted processing under all viewpoints are fused to obtain the first fused feature of the first 3D voxel reference point.

[0066] In some embodiments, converting the first fused feature into a first bird's-eye view feature and predicting the visual occupancy semantics of each first bird's-eye view grid based on the first bird's-eye view feature includes: The first fused features are grouped according to the grid coordinates of the first bird's-eye view grid; wherein, the first fused features in the same group are located in different height dimensions, but have the same grid coordinates in the planar dimension. The first fused feature is then fused again along the height dimension according to the grid coordinates of the group, to obtain the grid features of each group of grid coordinates after fusion; Based on the grid features of each set of grid coordinates, predict the visual occupancy semantics of each first bird's-eye view grid.

[0067] In some embodiments, the first fused features are fused again along the height dimension according to the grouped grid coordinates to obtain the grid features of each group of grid coordinates after fusion, including: The first fused features are then fused again along the height dimension according to the grid coordinates of the group; During feature fusion of any set of grid coordinates, if the height dimension does not have at least one first fusion feature indicating that a 3D voxel is occupied, then it is determined that the set of grid coordinates does not have visual occupancy. Otherwise, it is determined that the set of grid coordinates has visual occupancy, and the semantic category of the set of grid coordinates is the semantic category indicated by the first fusion feature with the largest height dimension among the at least one first fusion feature.

[0068] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0069] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0070] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0071] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0072] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0073] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0074] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0075] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this disclosure. The singular forms “a,” “the,” and “the” as used in one or more embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0078] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this disclosure, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0079] The above description is merely a preferred embodiment of one or more embodiments of this disclosure and is not intended to limit the scope of one or more embodiments of this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this disclosure should be included within the scope of protection of one or more embodiments of this disclosure.

Claims

1. A method of visual occupancy semantic prediction, the method comprising: The method comprises: According to the image feature information of each perspective image of the automatic driving vehicle, a first fusion feature of a first 3D voxel reference point uniformly distributed in a 3D detection space around the vehicle at a first density is calculated; The first fusion feature is converted into a first bird's eye view feature, and the visual occupancy semantics of each first bird's eye view grid is predicted according to the first bird's eye view feature, and then the 3D key detection space that needs further refined prediction in the 3D detection space is screened according to the visual occupancy semantics; According to the image feature information, a second fusion feature of a second 3D voxel reference point uniformly distributed in the 3D key detection space at a second density is calculated; wherein the second density is greater than the first density; At least according to the second fusion feature, the visual occupancy semantics of the second 3D voxel reference point is predicted.

2. The method of claim 1, wherein, The at least according to the second fusion feature, the visual occupancy semantics of the second 3D voxel reference point is predicted, comprising: According to the grid coordinates of the second 3D voxel reference point in the first bird's eye view grid, the first bird's eye view feature is calculated by interpolation to obtain the second bird's eye view feature of the second 3D voxel reference point; The second fusion feature and the second bird's eye view feature are connected to obtain the connected feature; The connected feature is input into a multi-layer perceptron to obtain the visual occupancy semantics of the second 3D voxel reference point predicted by the multi-layer perceptron.

3. The method of claim 1, wherein, The occupancy semantics of the bird's eye view grid includes the occupancy probability of the bird's eye view grid, and the 3D key detection space that needs further refined prediction in the 3D detection space is screened according to the visual occupancy semantics, comprising: According to the occupancy semantics, the first bird's eye view grid with the occupancy probability greater than a preset threshold is screened; The grid coordinates of the screened first bird's eye view grid in the bird's eye view plane are determined; According to the height range of the 3D detection space, the 3D detection space covered by the grid coordinates in the height range is taken as the 3D key detection space screened in the 3D detection space that needs further refined prediction.

4. The method of claim 1, wherein, The image feature information includes image feature value and predicted depth interval, and the first fusion feature of the first 3D voxel reference point uniformly distributed in the 3D detection space at the first density is calculated according to the image feature information of each perspective image of the automatic driving vehicle, comprising: The first 3D voxel reference point uniformly distributed in the 3D detection space at the first density is projected into the image feature information of each perspective image to obtain the predicted depth interval where each first 3D voxel reference point is located; For each perspective image, the image feature value of the perspective image is calculated by interpolation according to the predicted depth interval where the projection position is located, to obtain the image feature of each first 3D voxel reference point under the perspective image; The image features of each first 3D voxel reference point under all perspective images are fused to obtain the first fusion feature of the first 3D voxel reference point.

5. The method of claim 4, wherein, The image feature information further includes a prediction probability corresponding to each prediction depth interval, and the image features of each first 3D voxel reference point under each view image are fused to obtain first fusion features of the first 3D voxel reference point, including: According to the prediction probability corresponding to each prediction depth interval, the image features of each first 3D voxel reference point under each view image are subjected to probability weighting processing to obtain the image features of the first 3D voxel reference point after weighting processing; The image features of the first 3D voxel reference point after weighting processing under all view images are subjected to feature fusion to obtain the first fusion features of the first 3D voxel reference point.

6. The method of claim 1, wherein, The first fusion features are converted into first bird's-eye view features, and the visual occupancy semantics of each first bird's-eye view grid is predicted according to the first bird's-eye view features, including: The first fusion features are grouped according to the grid coordinates of the first bird's-eye view grid; wherein the first fusion features in the same group are different in height dimension but have the same grid coordinates in plane dimension; The first fusion features are subjected to feature fusion again along the height dimension according to the grouped grid coordinates to obtain grid features of the fused each group of grid coordinates; The visual occupancy semantics of each first bird's-eye view grid is predicted according to the grid features of the each group of grid coordinates.

7. The method of claim 6, wherein, The first fusion features are subjected to feature fusion again along the height dimension according to the grouped grid coordinates to obtain grid features of the fused each group of grid coordinates, including: The first fusion features are subjected to feature fusion again along the height dimension according to the grouped grid coordinates; In the feature fusion process of any group of grid coordinates, if there is no at least one first fusion feature indicating that the 3D voxel exists in the occupancy in the height dimension, it is determined that the group of grid coordinates does not exist in the visual occupancy; Otherwise, it is determined that the group of grid coordinates exists in the visual occupancy, and the semantic category of the group of grid coordinates is the semantic category indicated by the first fusion feature with the largest height dimension in the at least one first fusion feature.

8. A visual occupancy semantic prediction apparatus, characterized by, The device includes: A first computing unit configured to calculate first fusion features of first 3D voxel reference points uniformly distributed in a first density in a 3D detection space around an autonomous vehicle according to image feature information of view images of an environment around the autonomous vehicle; A screening unit configured to convert the first fusion features into first bird's-eye view features, predict visual occupancy semantics of each first bird's-eye view grid according to the first bird's-eye view features, and screen a 3D key detection space requiring further refined prediction in the 3D detection space according to the visual occupancy semantics; A second computing unit configured to calculate second fusion features of second 3D voxel reference points uniformly distributed in a second density in the 3D key detection space according to the image feature information; wherein the second density is greater than the first density; A prediction unit configured to predict visual occupancy semantics of the second 3D voxel reference points according to at least the second fusion features.

9. An electronic device, comprising: comprising a communication interface, a processor, a memory and a bus, the communication interface, the processor and the memory are connected with each other through the bus; the memory stores machine readable instructions, the processor executes the method of any one of claims 1 to 7 by calling the machine readable instructions.

10. A machine-readable storage medium, characterized in that, the machine readable storage medium stores machine readable instructions, the machine readable instructions realize the method of any one of claims 1 to 7 when called and executed by the processor.

Images