3D occupancy grid perception method and apparatus, vehicle and readable storage medium

Abstract

The application relates to a 3D occupancy grid perception method, device, vehicle and readable storage medium. The method comprises the following steps: acquiring space image features of a space image collected by a target space and image estimated depth; generating a preset number of anchor frames for the target space, fusing initialization anchor frame information of each anchor frame and the image estimated depth to obtain anchor frame pretreatment features of each anchor frame; obtaining first space information and first occupancy category information of each anchor frame according to the space image features, the anchor frame pretreatment features and anchor frame embedding features; obtaining occupancy probabilities of each occupancy grid in each effective grid in a plurality of grids generated for the target space according to second space information of the effective grid and the first space information; and obtaining second occupancy category information of each occupancy grid in each effective grid according to the occupancy probabilities, the first space information and the first occupancy category information. The method can improve the calculation efficiency of 3D occupancy grid perception.

Description

Technical Field

[0001] This application relates to the field of computer vision perception technology, and in particular to a 3D occupancy grid perception method, apparatus, vehicle, and readable storage medium. Background Technology

[0002] 3D occupancy grid perception is a key technology in the field of robotics, used to construct a three-dimensional geometric and semantic representation of the environment.

[0003] Traditional 3D occupancy grid sensing technologies mainly include methods based on LiDAR, vision (RGB / RGB-D), and laser-vision fusion. However, LiDAR-based 3D occupancy grid sensing methods suffer from sparse data in distant or occluded areas during single-frame LiDAR scans. Voxelization using laser point cloud data requires significant computational resources and results in incomplete occupancy grids, leading to poor prediction of grid semantic information. Vision-based 3D occupancy grid sensing algorithms use monocular / multi-view cameras or RGB-D sensors for depth estimation before constructing the 3D occupancy grid, but monocular / multi-view depth estimation accuracy is low, resulting in low spatial estimation accuracy for the 3D occupancy grid. Laser-vision fusion methods require simultaneous data acquisition from both LiDAR and cameras for 3D occupancy grid prediction, but processing both LiDAR and image data simultaneously consumes substantial memory and computational resources, leading to low computational efficiency.

[0004] It is evident that traditional 3D occupancy grid sensing technology suffers from high computational resource consumption and low computational efficiency in 3D occupancy grid recognition and prediction. Summary of the Invention

[0005] Therefore, it is necessary to provide a 3D occupancy grid sensing method, apparatus, vehicle, readable storage medium, and computer program product that can reduce the computational resource consumption required for 3D occupancy grid sensing and improve the computational efficiency of 3D occupancy grid sensing, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a 3D occupancy grid sensing method, comprising the following steps:

[0007] Acquire spatial image features and image depth estimation of spatial images acquired in the target space;

[0008] A preset number of anchor boxes are generated in the target space, the initial anchor box information of each anchor box is obtained, and the initial anchor box information and the estimated depth of the image are fused to obtain the anchor box preprocessing features of each anchor box.

[0009] Based on the spatial image features, the preprocessed features of each anchor frame, and the anchor frame embedding features corresponding to the initial anchor frame information, the first spatial information and the first occupancy category information of each anchor frame are obtained.

[0010] Multiple grids are generated for the target space. Based on the second spatial information of each effective grid and the first spatial information of each grid, the occupancy probability of each effective grid being occupied by each anchor frame is obtained; the number of grids is greater than the preset number.

[0011] Based on the occupancy probability, the first spatial information, and the first occupancy category information, the second occupancy category information of each occupied grid in each effective grid is obtained.

[0012] In one embodiment, obtaining the first spatial information and first occupancy category information of each anchor frame based on the spatial image features, the preprocessed features of each anchor frame, and the anchor frame embedding features corresponding to the initialized anchor frame information includes:

[0013] Feature aggregation is performed on the spatial image features and the preprocessed features of each anchor frame to obtain the anchor frame fusion features of each anchor frame;

[0014] The initial anchor frame information is used as the basic information of each anchor frame, and the anchor frame fusion feature and the anchor frame embedding feature are fused. The fused features are refined by a preset multilayer perceptron model to obtain the incremental information of each anchor frame.

[0015] Based on the data types of the basic information and the incremental information, element-level operations are performed on the basic information and the incremental information to obtain the first spatial information and the first occupancy category information of each anchor frame.

[0016] In one embodiment, the feature aggregation of the spatial image features and each of the preprocessed anchor frames to obtain the anchor frame fusion features of each anchor frame includes:

[0017] Multiple key points within a preset neighborhood of each anchor frame are obtained, and each key point is projected onto the spatial image to obtain the key point pixel coordinates of each key point.

[0018] Based on the pixel coordinates of each key point and the spatial image features, extract the key point image features corresponding to each key point;

[0019] The key point image features and the anchor frame preprocessing features are fused to obtain the anchor frame fused features.

[0020] In one embodiment, obtaining the occupancy probability of each effective grid being occupied by each anchor frame based on the second spatial information of each effective grid and the first spatial information of each grid includes:

[0021] Obtain the pixel position of each of the grids in the target pixel coordinate system; the target pixel coordinate system is the pixel coordinate system of the spatial image;

[0022] Obtain the pixel range of the spatial image in the target pixel coordinate system, and take the grid cells whose pixel positions are within the pixel range as the valid grid cells;

[0023] Based on the second spatial information and the first spatial information, the distance between each grid and each anchor frame is obtained, and the occupancy probability of each effective grid being occupied by each anchor frame is determined based on the distance.

[0024] In one embodiment, obtaining the second occupancy category information of each occupied grid in each of the effective grids based on each of the occupancy probabilities, each of the first spatial information, and each of the first occupancy category information includes:

[0025] The occupied grid is obtained from the effective grid according to each of the occupancy probabilities;

[0026] Obtain the opacity of the anchor frame in each of the first spatial information, and determine the category attribute quantization value of each anchor frame based on the anchor frame opacity and the first occupancy category information;

[0027] Based on the occupancy probability and the anchor frame opacity, obtain the contribution weight of the occupancy grid being occupied by each anchor frame;

[0028] Based on the contribution weights and the category attribute quantification values ​​of each anchor box, the semantic expectation of the anchor box is obtained; the semantic expectation is the average value of the anchor box belonging to the target object category in the target space.

[0029] Based on the occupancy probability and the semantic expectation, the category attribute quantization value of each occupied grid is obtained. If the category attribute quantization value of the occupied grid is greater than a preset category threshold, the second occupancy category information of each occupied grid is determined.

[0030] In one embodiment, obtaining the occupied grid from the valid grid according to each of the occupancy probabilities includes:

[0031] The effective grid cells whose occupancy probability is greater than a preset occupancy threshold are designated as the occupied grid cells.

[0032] In one embodiment, obtaining the contribution weight of the occupied grid being occupied by each of the anchor frames based on each of the occupancy probabilities and the anchor frame opacity includes:

[0033] Based on the occupancy probabilities, the conditional probabilities of each grid cell under a given anchor frame are obtained.

[0034] The anchor frame opacity is used as the prior probability. Based on the prior probability and the conditional probability, the posterior probability of each anchor frame under the given grid condition is obtained. The posterior probability is used as the contribution weight.

[0035] In one embodiment, fusing the initial anchor frame information and the estimated image depth to obtain the anchor frame preprocessing features of each anchor frame includes:

[0036] Based on the initial anchor frame information, extract the anchor frame features of each anchor frame; and project each anchor frame onto the pixel coordinate system of the spatial image to obtain the pixel position of each anchor frame;

[0037] Based on the pixel position of each anchor frame and the estimated depth of the image, the estimated depth of each anchor frame is obtained;

[0038] The spatial location information in the initial anchor frame information and the estimated depth of the anchor frame are fused, and the fused features are convolved to obtain the anchor frame depth features of each anchor frame.

[0039] The anchor frame's own features and anchor frame depth features are fused to obtain the anchor frame preprocessing features.

[0040] Secondly, this application provides a 3D occupancy grid sensing device, the device comprising:

[0041] The image processing module is used to acquire spatial image features and image depth estimation of spatial images acquired from the target space;

[0042] An anchor frame preprocessing module is used to generate a preset number of anchor frames in the target space, obtain the initial anchor frame information of each anchor frame, fuse the initial anchor frame information and the estimated depth of the image to obtain the anchor frame preprocessing features of each anchor frame.

[0043] The feature encoding module is used to obtain the first spatial information and the first occupancy category information of each anchor frame based on the spatial image features, the preprocessed features of each anchor frame, and the anchor frame embedding features corresponding to the initial anchor frame information.

[0044] An occupancy probability calculation module is used to generate multiple grids in the target space, and to obtain the occupancy probability of each effective grid being occupied by each anchor frame based on the second spatial information of each effective grid and the first spatial information of each grid; the number of grids is greater than the preset number;

[0045] The occupancy category calculation module is used to obtain the second occupancy category information of each occupied grid in each of the effective grids based on the occupancy probability, the first spatial information, and the first occupancy category information.

[0046] Thirdly, this application also provides a vehicle, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described 3D occupancy grid perception method.

[0047] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0048] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0049] The aforementioned 3D occupancy grid perception method, device, vehicle, readable storage medium, and computer program product acquire spatial image features and image depth estimation of a spatial image of the target space; generate a preset number of anchor frames in the target space; fuse the initial anchor frame information and image depth estimation of each anchor frame to obtain the anchor frame preprocessing features of each anchor frame; utilize anchor frames carrying initial Gaussian spatial information and semantic category information to reduce the consumption of computational resources in spatial recognition perception; and then, based on the spatial image features, the preprocessing features of each anchor frame, and the anchor frame embedding features, achieve iterative correction of Gaussian parameters (position / scale / rotation / transparency) through dual-path optimization using basic and incremental information of the anchor frames. The first spatial information and first occupancy category information of each anchor frame are obtained to obtain accurate Gaussian information and semantic information of each anchor frame. Then, for multiple grids generated in the target space, the occupancy probability of each effective grid being occupied by each anchor frame is obtained based on the second spatial information of each effective grid and the first spatial information of each grid. Based on the occupancy probability, the first spatial information and the first occupancy category information of each effective grid, the second occupancy category information of each occupied grid is obtained. Thus, the spatial information and category information of a large number of occupied grids in the spatial scene can be efficiently and quickly identified based on a small amount of anchor frame information, thereby reducing the consumption of computing resources and effectively improving the computational efficiency of 3D occupied grid recognition and perception in space. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is an application environment diagram of a 3D occupancy grid sensing method in one embodiment;

[0052] Figure 2 This is a flowchart illustrating a 3D occupancy grid sensing method in one embodiment;

[0053] Figure 3 This is a schematic diagram of the generation of a target spatial raster in one embodiment;

[0054] Figure 4 This is a schematic diagram illustrating the acquisition of Gaussian and semantic information of anchor frames in one embodiment.

[0055] Figure 5 This is a schematic diagram illustrating the acquisition of 3D occupancy grid information in one embodiment;

[0056] Figure 6 This is a schematic diagram of anchor frame initialization and preprocessing in one embodiment;

[0057] Figure 7 This is a flowchart illustrating a 3D occupancy grid sensing method in another embodiment;

[0058] Figure 8 This is a structural block diagram of a 3D occupancy grid sensing device in one embodiment;

[0059] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] The 3D occupancy grid sensing method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 can acquire spatial image features and estimated depth of spatial images captured from the target space; terminal 102 can generate a preset number of anchor frames in the target space, fuse the initial anchor frame information and estimated depth of each anchor frame to obtain anchor frame preprocessing features; terminal 102 can obtain first spatial information and first occupancy category information of each anchor frame based on spatial image features, anchor frame preprocessing features, and anchor frame embedding features; terminal 102 can obtain the occupancy probability of each effective grid being occupied by each anchor frame based on second spatial information of effective grids among multiple grids generated in the target space and each first spatial information; terminal 102 can obtain second occupancy category information of each occupied grid in each effective grid based on each occupancy probability, each first spatial information, and each first occupancy category information. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, etc. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0062] In one exemplary embodiment, such as Figure 2 As shown, a 3D occupancy grid sensing method is provided. This embodiment applies this method to... Figure 1 The method is illustrated using terminal 102 as an example. In this embodiment, the method includes the following steps S202 to S210. Wherein:

[0063] Step S202: Obtain the spatial image features and image depth estimation of the spatial image acquired from the target space.

[0064] In practical applications, a spatial image can be a single-frame, single-view image captured from a target space (e.g., an indoor space), assuming the image size is (H, W, C). Terminal 102 can use relatively mature image feature extraction models (e.g., the EfficientNet model, a convolutional neural network architecture that can be used for image classification) and depth estimation models (e.g., the DepthAnyThingV2 model, a monocular depth estimation model) to process the spatial image, obtaining the multi-scale image features (i.e., spatial image features) Feat_img and the estimated image depth Depth.

[0065] Step S204: Generate a preset number of anchor boxes in the target space, obtain the initial anchor box information of each anchor box, fuse the initial anchor box information and the image estimated depth to obtain the anchor box preprocessing features of each anchor box.

[0066] In practical implementation, considering the size of common spaces and the resource consumption of computing devices, the terminal 102 can generate a preset number of anchor frames for the target space through a mature anchor frame generation model, so as to cover the modeling requirements of the target space (e.g., indoor space).

[0067] This preset number can be determined based on the space size, the preset grid size, and computing resources. The specific value of this preset number can be adjusted according to the specific situation in actual application. For example, if the current size of the indoor space environment is limited to (-2.4, +2.4) in the X direction, (-2.4, +2.4) in the Y direction, and (0, +2.88) in the Z direction, that is, the target space size is [4.8m x 4.8m x 2.88m], assuming the grid size is [0.08m x 0.08m x 0.08m], and considering the scaling ratio determined by computing resource consumption (for example, 1 / 8), the initial number of anchors can be set to 16200.

[0068] When generating anchor boxes, each anchor carries information with a fixed dimension, namely the initial anchor box information D_a. This can include spatial position information (X, Y, Z), spatial scaling information (scale_x, scale_y, scale_z), spatial orientation quaternions (w, x, y, z), opacity information (opacities), and semantic category information (assuming there are 12 categories of objects in the target space, there are 12 categories of probability information, so the semantic category information has 12 dimensions). Therefore, the initial anchor data can be data with a shape of [16200, 23]. It is important to note that the initial anchor spatial position information (X, Y, Z) is randomly generated based on the size of the current scene, and the coordinate system is the camera coordinate system. The spatial position information coordinates (X, Y, Z) of the initially generated anchor will not exceed the limit range of the target space size.

[0069] For the target space scene, the mutual transformation relationship between the world coordinate system, camera coordinate system, and pixel coordinate system can be pre-calibrated so that the terminal 102 can use the feature extraction module in the pre-trained occupancy grid perception model to fuse the initial anchor box information D_a and the image estimated depth Depth to obtain the anchor box preprocessing feature Feat_anchor_depth of each anchor box.

[0070] Step S206: Based on the spatial image features, the preprocessed features of each anchor frame, and the anchor frame embedding features corresponding to each initialized anchor frame information, the first spatial information and the first occupancy category information of each anchor frame are obtained.

[0071] Among them, spatial information can also be called Gaussian information. The first spatial information is the Gaussian information of the anchor frame, which includes spatial position information (X, Y, Z), spatial scaling size information (scale_x, scale_y, scale_z), spatial orientation quaternion (w,x, y, z), and transparency information (opacities), that is, Gaussian parameters (position / scale / rotation / transparency).

[0072] Occupation category information, also known as semantic category information or semantic information, is the category information of the object occupied by the anchor box in the target space. For example, if the objects in the target space are assumed to be of 12 categories, there are 12 categories of probability information.

[0073] In practical applications, during the initialization phase, the anchor boxes are encoded to obtain the anchor embedding feature `anchor_embed`. This can be understood as mapping the anchor features to a high-dimensional space to facilitate subsequent processing by the occupancy grid perception model. Terminal 102 can refine the anchor boxes using the encoder module in the occupancy grid perception model, based on the spatial image feature `Feat_img`, the preprocessed anchor box feature `Feat_anchor_depth`, and the anchor embedding feature `anchor_embed` corresponding to the initial anchor box information `D_a`, thereby obtaining more accurate first spatial information and first occupancy category information for each anchor box.

[0074] Step S208: Generate multiple grids in the target space. Based on the second spatial information and the first spatial information of each effective grid, obtain the occupancy probability of each effective grid being occupied by each anchor frame.

[0075] The second spatial information is Gaussian information of the grid, including three-dimensional spatial position information (D_grid).

[0076] In specific implementations, such as Figure 3The diagram shown illustrates the generation of the target space grid. Terminal 102 can generate multiple basic 3D grids for the target space based on the scene size and preset grid size. Currently, these grids are empty and unoccupied. For example, assuming the grid size is [0.08m x 0.08m x 0.08m], then for an indoor space scene with X direction (-2.4, +2.4), Y direction (-2.4, +2.4), and Z direction (0, +2.88), a total of 129,600 3D voxel grids [60, 60, 36] can be generated. The number of generated grids is greater than the preset number of anchor frames.

[0077] For the generated voxel grid, terminal 102 can filter out the valid grids from each grid, and determine the distance between each grid and each anchor frame based on the second spatial information of each valid grid and the first spatial information of each anchor frame, so as to calculate the occupancy probability of each valid grid being occupied by each anchor frame. .

[0078] Step S210: Based on each occupancy probability, each first spatial information, and each first occupancy category information, obtain the second occupancy category information of each occupied grid in each effective grid.

[0079] Among them, the second occupancy category information, also known as the semantic information of the occupancy grid, is the category information of the object occupied by the occupancy grid in the target space.

[0080] In practical applications, terminal 102 can adjust the occupancy probability according to the occupancy probability. The first spatial information and the first occupation category information are used to calculate the second occupation category information of each occupied grid in each effective grid after Gaussian probability calculation.

[0081] The aforementioned 3D occupancy grid perception method acquires spatial image features and image depth estimation from a spatial image of the target space. It generates a preset number of anchor boxes in the target space, fuses the initial anchor box information and image depth estimation of each anchor box to obtain preprocessed anchor box features. This allows the use of anchor boxes carrying initial Gaussian spatial information and semantic category information, reducing computational resource consumption in spatial recognition perception. Then, based on the spatial image features, preprocessed anchor box features, and anchor box embedding features, it iterative correction of Gaussian parameters (position / scale / rotation / transparency) is achieved through a dual-path optimization of "basic information + incremental information" for the anchor boxes, resulting in the first spatial feature of each anchor box. Information and first occupancy category information are used to obtain accurate Gaussian information and semantic information for each anchor frame. Then, for multiple grids generated in the target space, the occupancy probability of each effective grid being occupied by each anchor frame is obtained based on the second spatial information and the first spatial information of each effective grid. Based on each occupancy probability, each first spatial information and each first occupancy category information, the second occupancy category information of each occupied grid in each effective grid is obtained. Thus, the spatial information and category information of a large number of occupied grids in a spatial scene can be efficiently and quickly identified based on a small amount of anchor frame information, thereby reducing the consumption of computing resources and effectively improving the computational efficiency of 3D occupied grid recognition and perception in space.

[0082] In an exemplary embodiment, the first spatial information and first occupancy category information of each anchor frame are obtained based on spatial image features, preprocessed features of each anchor frame, and anchor frame embedding features corresponding to initial anchor frame information. This includes: performing feature aggregation on spatial image features and preprocessed features of each anchor frame to obtain anchor frame fusion features of each anchor frame; using initial anchor frame information as basic information of each anchor frame, fusing the anchor frame fusion features and anchor frame embedding features, refining the fused features through a preset multilayer perceptron model to obtain incremental information of each anchor frame; and performing element-level operations on the basic information and incremental information according to their data types to obtain the first spatial information and first occupancy category information of each anchor frame.

[0083] In specific implementations, such as Figure 4 The diagram shown illustrates the acquisition of Gaussian and semantic information of anchor frames. Terminal 102 can perform feature aggregation on spatial image features Feat_img and preprocessed anchor frame features Feat_anchor_depth based on neighborhood key points of spatial image features to obtain anchor frame fusion features Feat_anchor_fusion for each anchor frame.

[0084] Figure 4The feedforward network, normalization, and sparse convolution shown are relatively mature feature processing methods, and this application does not specifically limit them. The refinement module in the occupancy grid perception model of this application is the core module for ultimately obtaining the Gaussian information (spatial information) and semantic information (category information) required for prediction. As in the previous example, the anchor box input to the refinement module itself carries 23 dimensions of required information, i.e., the initial anchor box information D_a. However, this information is not accurate enough. The complete 23-dimensional information can be divided into basic information and incremental information, which are then restricted and refined in the refinement module.

[0085] Basic information is stored in the initialized anchor data, i.e., the initialized anchor box information D_a. The terminal 102 can use each initialized anchor box information D_a as the basic information of each anchor box. Incremental information needs to be obtained through the refinement module. The terminal 102 can add the anchor box fusion feature Feat_anchor_fusion and the anchor box embedding feature anchor_embed, and input the fused feature into the preset multilayer perceptron model (MLP). Through the linear transformation and nonlinear activation of multiple hidden layers of the MLP, the fused feature is accurately corrected, thereby obtaining the incremental information D_delta of each anchor box. The data size of the incremental information D_delta is also [16200, 23].

[0086] Furthermore, terminal 102 can perform element-wise operations on each basic information D_a and each incremental information D_delta according to the data types of each basic information and each incremental information, using the following formula of the refined anchor frame information prediction model, to obtain the final refined anchor frame information D_a_final. This refined anchor frame information D_a_final includes the first spatial information and the first occupancy category information of each anchor frame:

[0087]

[0088] in, This represents different element-level operation processing methods. For example, for XYZ spatial location information, This indicates element-wise addition, meaning that the X in the final 3D information of the anchor is the X in the base D_a plus the corresponding X in D_delta; while for the rotation quaternion in the anchor frame, This indicates element-wise multiplication.

[0089] Finally, through the occupancy grid perception model refinement module, the precise Gaussian information and semantic information of the anchor boxes required for perception can be obtained. In other words, the refined anchor box information D_a_final includes the required Gaussian information, namely the first spatial information, including spatial position information (X, Y, Z), spatial scaling size information (scale_x, scale_y, scale_z), spatial orientation quaternion (w, x, y, z), transparency information (opacities), and semantic information, namely the first occupancy category information (assuming 12 categories, then there is probability information for 12 categories).

[0090] The technical solution of this embodiment aggregates spatial image features and preprocessed features of each anchor frame to obtain anchor frame fusion features, which can provide important input for subsequent feature refinement to extract accurate information. Then, the initial anchor frame information is used as the basic information of each anchor frame, and the anchor frame fusion features and anchor frame embedding features are fused. The fused features are refined through a preset multilayer perceptron model to obtain incremental information of each anchor frame. According to the data types of each basic information and each incremental information, element-level operations are performed on each basic information and each incremental information to obtain the first spatial information and the first occupancy category information of each anchor frame. Thus, dual-path optimization based on basic information and incremental information can provide accurate spatial information and category information of anchor frames.

[0091] In an exemplary embodiment, feature aggregation is performed on spatial image features and preprocessed features of each anchor frame to obtain anchor frame fusion features for each anchor frame. This includes: acquiring multiple key points within a preset neighborhood of each anchor frame; projecting each key point onto the spatial image to obtain the key point pixel coordinates of each key point; extracting key point image features corresponding to each key point based on the pixel coordinates of each key point and the spatial image features; and fusing the key point image features and the preprocessed features of the anchor frames to obtain anchor frame fusion features.

[0092] In practical applications, for each anchor, terminal 102 can learn to acquire multiple key points within a preset neighborhood of each anchor frame. For example, seven key points within the neighborhood of each anchor frame can be obtained for fusing image features. For spatial location information X, Y, and Z, three offsets are learned in each direction, ensuring the offsets are within 0.5 meters to avoid excessive offsetting. In this way, the model can automatically adjust the required neighborhood points during training to sample better features. Thus, for a preset number (e.g., 16200) of initial anchor frames, the neighborhood key point spatial data D_a_neighbour can be obtained, with a data size of [16200, 7, 3].

[0093] like Figure 4As shown, before the deformable feature aggregation stage, each keypoint is projected onto the spatial image to obtain the keypoint pixel coordinates. Then, in the deformable feature aggregation stage, based on the pixel coordinates of each keypoint and the multi-scale image features (spatial image features Feat_img), the image features corresponding to each keypoint are extracted and superimposed by convolution to increase the dimension to 96, resulting in the keypoint image features Feat_img_keypoints. The keypoint image features Feat_img_keypoints and the anchor box preprocessing features Feat_anchor_depth can be superimposed and fused using the concat() function to obtain the anchor box fusion features Feat_anchor_fusion.

[0094] The technical solution of this embodiment obtains the key point pixel coordinates of each key point by projecting multiple key points in the preset neighborhood of each anchor frame onto a spatial image, thereby providing key point reference information for anchor frame feature aggregation; then, based on the pixel coordinates of each key point and the spatial image features, the key point image features corresponding to each key point are extracted, and the key point image features and anchor frame preprocessing features are fused to obtain anchor frame fused features, which provides important input for the subsequent accurate calculation of Gaussian information and semantic information of the anchor frame.

[0095] In an exemplary embodiment, the occupancy probability of each effective grid being occupied by each anchor frame is obtained based on the second spatial information and the first spatial information of each effective grid, including: obtaining the pixel position of each grid in the target pixel coordinate system; obtaining the pixel range of the spatial image in the target pixel coordinate system, and taking the grid whose pixel position is within the pixel range as the effective grid; obtaining the distance between each grid and each anchor frame based on the second spatial information and the first spatial information, and determining the occupancy probability of each effective grid being occupied by each anchor frame based on the distance.

[0096] The target pixel coordinate system is the pixel coordinate system of the spatial image.

[0097] In specific implementations, such as Figure 5 The diagram shown illustrates the acquisition of 3D occupancy grid information. Terminal 102 can project the grid onto the pixel coordinate system of the spatial image through coordinate transformation, obtaining the pixel position of each grid in the target pixel coordinate system; it also obtains the pixel range of the spatial image in the target pixel coordinate system, and identifies grids whose pixel positions fall within this range as valid grids. Alternatively, grids that cannot be projected onto the target pixel coordinate system are considered invalid grids. For each valid grid, its three-dimensional spatial position information can be recorded as the second spatial information D_grid, with a data size of [N, 3], where N is the number of valid grids.

[0098] Assuming that the Gaussian information of different anchors is independent, for each valid grid cell, the distance between each grid cell and each anchor frame can be obtained based on the second spatial information and the first spatial information. The occupancy probability of each valid grid cell being occupied by each anchor frame can then be determined based on the distance, as shown in the following formula:

[0099]

[0100] in, Represents a grid Spatial location, Represents a grid At the given location, what is the overall occupancy probability of all anchors (assuming there are P anchors in total)? This indicates that the i-th anchor is in the grid. The probability of occupancy at a given location is calculated using the following formula:

[0101]

[0102] in, It represents the mean center of the i-th anchor, which is the three-dimensional spatial information of the first spatial information in the refined anchor frame information D_a_final; The covariance matrix can be calculated using scale and rotation information. The single-unit occupancy probability is derived from the above equation. It can be seen that when the grid The closer the position is to the center of the i-th anchor, the larger the grid. The higher the probability of being occupied by the i-th anchor, the greater the likelihood of it being occupied. This is relevant when calculating the overall occupancy probability. At that time, if grid If it is close enough to a certain anchor, then If the value approaches 1, it can be considered that the effective grid is occupied by the anchor.

[0103] The technical solution of this embodiment defines effective grids as those whose pixel positions in the target pixel coordinate system are within the pixel range of the target pixel coordinate system. This filters out invalid grids that cannot be projected into the pixel coordinate system, laying a key foundation for identifying occupied grids from the effective grids. Furthermore, based on the second spatial information and the first spatial information, the distance between each grid and each anchor frame is obtained. The occupancy probability of each effective grid being occupied by each anchor frame is determined based on the distance. Thus, based on the spatial positional relationship between the grid and the anchor frame, the grid occupied by the anchor frame can be accurately identified.

[0104] In an exemplary embodiment, obtaining the second occupation category information of each occupied grid in each effective grid based on each occupation probability, each first spatial information, and each first occupation category information includes: obtaining occupied grids from the effective grids based on each occupation probability; obtaining the anchor frame opacity in each first spatial information, and determining the category attribute quantization value of each anchor frame based on the anchor frame opacity and the first occupation category information; obtaining the contribution weight of each anchor frame occupying the occupied grid based on each occupation probability and the anchor frame opacity; obtaining the semantic expectation of the anchor frame based on each contribution weight and the category attribute quantization value of each anchor frame; obtaining the category attribute quantization value of each occupied grid based on the occupation probability and the semantic expectation; and determining the second occupation category information of each occupied grid if the category attribute quantization value of the occupied grid is greater than a preset category threshold.

[0105] In practical applications, terminal 102 can determine based on thresholds and various occupancy probabilities. It retrieves occupied grid cells from the valid grid cells. As an example, this can be handled by setting a threshold to control the occupancy probability. Valid grid cells larger than a preset occupancy threshold are designated as occupancy grid cells, i.e., when... A grid can be identified if it exceeds a certain threshold. It is occupied. The threshold is usually set based on statistical experience, for example... If the value is greater than 0.5, it is considered occupied; if it is lower than this value, it is considered unoccupied.

[0106] The overall occupancy probability is obtained by calculating using Gaussian probability. After extracting the occupied grid from the valid grid, the terminal 102 can calculate the semantic information to obtain the semantic attributes of the occupied grid. The calculation formula is as follows:

[0107]

[0108] in, This represents the quantized value of the category attribute that occupies the grid. The semantic expectation of the anchor box can be the average value of the target object category in the target space to which the anchor box belongs.

[0109] Based on each occupy probability and anchor frame opacity Get the occupied grid by each anchor frame Contribution weight Then, based on the weight of each contribution... Quantified values ​​of the category attributes of each anchor box To obtain the semantic expectation of the anchor box. :

[0110]

[0111] in, This represents the quantized value of the category attribute for each anchor box. The Gaussian information of all anchors in the refined anchor box information D_a_final can be considered as a Gaussian mixture model, allowing us to obtain the anchor box opacity (denoted as ) in the first spatial information. ), and based on the anchor frame opacity Based on the first category information, determine the quantified value of the category attribute for each anchor box. As an example, the opacity attribute (denoted as opacity) in the anchor box information D_a_final can be refined. As a prior distribution of the Gaussian distribution, it is L1 normalized (i.e., summing the absolute values ​​of each element in the vector and then dividing each element by the sum). The conditional probability is a Gaussian probability distribution parameterized by the mean m, scale information s, and rotation information r. Then, the first occupied category information (i.e., the semantic information of the anchor box) is normalized using the softmax function to obtain the quantized value of the category attribute of each anchor box. .

[0112] Furthermore, terminal 102 can adjust the occupancy probability accordingly. and semantic expectations This yields the quantized values ​​of the category attributes for each occupied grid cell. Finally, the semantic attributes of the occupied grid can still be processed by setting a threshold, based on the quantified value of the category attribute of the occupied grid. If the number of occupants exceeds a preset category threshold, determine the second occupancy category information for each occupied grid.

[0113] The technical solution of this embodiment obtains occupied grids from the effective grids based on each occupancy probability, laying a crucial foundation for subsequent identification and calculation of the spatial and category information of the occupied grids. Then, based on the anchor frame opacity and first occupancy category information in each first spatial information, the category attribute quantization value of each anchor frame is determined. Based on each occupancy probability and anchor frame opacity, the contribution weight of each anchor frame occupying the occupied grid is obtained, thereby clarifying the importance of the grid occupied by each anchor frame. Based on each contribution weight and the category attribute quantization value of each anchor frame, the semantic expectation of the anchor frame is obtained. Based on the occupancy probability and semantic expectation, the category attribute quantization value of each occupied grid is obtained. If the category attribute quantization value of the occupied grid is greater than a preset category threshold, the second occupancy category information of each occupied grid is determined. Therefore, based on Gaussian probability, the category attribute of each occupied grid can be accurately identified.

[0114] In an exemplary embodiment, the contribution weight of an occupied grid being occupied by each anchor frame is obtained based on each occupancy probability and the anchor frame opacity, including: obtaining the conditional probability of each grid under a given anchor frame based on each occupancy probability; using the anchor frame opacity as a prior probability; obtaining the posterior probability of each anchor frame under a given grid based on the prior probability and the conditional probability; and using the posterior probability as the contribution weight.

[0115] In a specific implementation, terminal 102 can adjust the occupancy probability according to the occupancy probability. The conditional probability of each grid cell under a given anchor frame can be obtained using the following formula. :

[0116]

[0117] Adjust the opacity of the anchor frame As a prior probability, based on the prior probability and conditional probability This yields the posterior probability of each anchor frame given a specific grid. The posterior probability is used as the contribution weight.

[0118]

[0119] It can be seen that in the grid The contribution weight of the i-th anchor (which can be considered as a Gaussian distribution) Opacity and likelihood probability Two parts determine the final semantic expectation. It is a weighted average of the semantic attributes of all anchor frames in a Gaussian distribution.

[0120] The technical solution of this embodiment can obtain the conditional probability of each grid under a given anchor frame by based on the occupancy probability; then, by using the anchor frame opacity as the prior probability, the posterior probability of each anchor frame under a given grid can be obtained based on the Gaussian probability, thereby clarifying the contribution weight of each anchor frame to the occupied grid.

[0121] In an exemplary embodiment, the initial anchor frame information and the estimated image depth are fused to obtain the anchor frame preprocessing features of each anchor frame, including: extracting the anchor frame features of each anchor frame based on the initial anchor frame information; projecting each anchor frame onto the pixel coordinate system of the spatial image to obtain the pixel position of each anchor frame; obtaining the estimated anchor frame depth of each anchor frame based on the pixel position of each anchor frame and the estimated image depth; fusing the spatial position information and the estimated anchor frame depth in the initial anchor frame information, and convolving the fused features to obtain the anchor frame depth features of each anchor frame; and fusing the anchor frame features and the anchor frame depth features to obtain the anchor frame preprocessing features.

[0122] In specific implementations, such as Figure 6 The diagram shown illustrates the anchor frame initialization and preprocessing. For each anchor's initial anchor frame information D_a in the camera coordinate system, terminal 102 can use a fully connected layer to extract features from the anchor itself, obtaining the anchor frame's own features Feat_anchor, with a data size of [16200, 96]. Each anchor frame is then projected onto the pixel coordinate system of the spatial image to obtain the corresponding pixel position (u, ...). v); then, based on the image depth estimation using monocular depth estimation and the pixel positions of each anchor box, the estimated anchor box depth D_a_depth of the initialized anchor can be obtained, with a data size of [16200, 1]; at the same time, the anchor box itself also carries spatial position Z information, with a data size of [16200, 1]. After superimposing the spatial position Z information and the estimated anchor box depth D_a_depth from the initialized anchor box information, the anchor box depth feature Feat_depth, which represents the high-dimensional depth features, is extracted through convolution, with a data size of [16200, 96]; finally, the concat() function can be used to superimpose and connect the anchor box's own feature Feat_anchor and the anchor box depth feature Feat_depth to obtain the preprocessed anchor box feature Feat_anchor_depth.

[0123] The technical solution of this embodiment projects each anchor frame onto the pixel coordinate system of the spatial image to fuse the pixel position of each anchor frame and the estimated depth of the image, thereby obtaining the estimated depth of each anchor frame. Then, it fuses the spatial position information and the estimated depth of the anchor frame in the initial anchor frame information, and performs convolution on the fused features to obtain the anchor frame depth features of each anchor frame. Finally, it fuses the anchor frame's own features and the anchor frame depth features to obtain the anchor frame preprocessing features, thus providing an important foundation for subsequent feature aggregation and refinement to extract more accurate Gaussian and semantic information.

[0124] In one possible example, such as Figure 7As shown, this application provides a 3D occupancy grid sensing method based on probabilistic Gaussian, specifically including:

[0125] Single-frame image feature extraction and depth estimation. Multi-level feature extraction and depth estimation are performed on single-frame images using image feature extraction and depth estimation models, yielding multi-level image features (spatial image features) and estimated image depth. This allows for the combination of monocular depth estimation and image features, improving perception accuracy and enhancing the overall perception accuracy of the robot system in various scenarios.

[0126] The feature extraction module performs feature preprocessing. Based on the proposed feature processing module, anchors with Gaussian and semantic information are pre-generated. Anchor box depth estimates (anchor box depth features) are obtained through projection. The depth estimates are then superimposed with the spatial location information Z-values ​​from the initialized anchor box information to obtain fused anchor box preprocessed features, providing fused depth information for subsequent model processing.

[0127] The encoder module performs feature aggregation and Gaussian and semantic information computation. Based on the design, the encoder module processes and fuses depth information, multi-scale image features, and Gaussian anchor information. Through projective learnable keypoints and operations such as deformable feature aggregation and sparse convolution, it performs high-dimensional feature information processing, decomposing anchor information into two types: basic information and incremental information. The incremental information is optimized through model optimization, and finally, complete Gaussian and semantic information is obtained, improving the model's perceptual accuracy. Thus, a fixed number of anchors cover the indoor space scene. Through dual-path optimization of "basic information + incremental information," it achieves iterative correction of Gaussian parameters (position / scale / rotation / transparency) while reducing computational resource consumption, thereby both reducing computational resource consumption and improving computational efficiency.

[0128] Based on probabilistic Gaussian calculation of 3D occupied grid attributes, the perception results of 3D occupied grids are obtained. Invalid grids are filtered out using projection, and then the probability of a grid being occupied is calculated using Gaussian probability. Finally, the semantic attributes of the occupied grids are calculated using the concept of Gaussian mixture model to perform geometric probability prediction and semantic probability prediction of 3D occupied grids, thereby obtaining the spatial location and semantic information of 3D occupied grids and improving the accuracy of perception.

[0129] Indoor robot scenarios are quite complex. The perception method proposed in this application can efficiently and quickly detect the 3D occupancy semantic grid of the scene, providing good prior conditions for robot indoor navigation or other tasks, and playing a role in improving the quality of planning, control and decision-making.

[0130] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a 3D occupancy grid perception method described above.

[0131] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0132] Based on the same inventive concept, this application also provides a 3D occupancy grid sensing device for implementing the 3D occupancy grid sensing method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more 3D occupancy grid sensing device embodiments provided below can be found in the limitations of the 3D occupancy grid sensing method described above, and will not be repeated here.

[0133] In one exemplary embodiment, such as Figure 8 As shown, a 3D occupancy grid sensing device is provided, comprising:

[0134] The image processing module 810 is used to acquire spatial image features and image depth estimation of spatial images acquired from the target space.

[0135] Anchor frame preprocessing module 820 is used to generate a preset number of anchor frames in the target space, obtain the initial anchor frame information of each anchor frame, fuse the initial anchor frame information and the image estimated depth to obtain the anchor frame preprocessing features of each anchor frame.

[0136] The feature encoding module 830 is used to obtain the first spatial information and the first occupancy category information of each anchor frame based on the spatial image features, the preprocessed features of each anchor frame, and the anchor frame embedding features corresponding to the initial anchor frame information.

[0137] The occupancy probability calculation module 840 is used to generate multiple grids in the target space and obtain the occupancy probability of each effective grid being occupied by each anchor frame based on the second spatial information and the first spatial information of each effective grid; the number of grids is greater than the preset number.

[0138] The occupancy category calculation module 850 is used to obtain the second occupancy category information of each occupied grid in each effective grid based on each occupancy probability, each first spatial information and each first occupancy category information.

[0139] In one embodiment, the feature encoding module 830 is further configured to perform feature aggregation on spatial image features and preprocessed features of each anchor frame to obtain anchor frame fusion features of each anchor frame; use the initial anchor frame information as the basic information of each anchor frame, and fuse the anchor frame fusion features and anchor frame embedding features; refine the fused features through a preset multilayer perceptron model to obtain incremental information of each anchor frame; and perform element-level operations on the basic information and incremental information according to the data types of each basic information and each incremental information to obtain the first spatial information and the first occupancy category information of each anchor frame.

[0140] In one embodiment, the feature encoding module 830 is further configured to acquire multiple key points in a preset neighborhood of each anchor frame, project each key point onto a spatial image to obtain the key point pixel coordinates of each key point; extract the key point image features corresponding to each key point based on the key point pixel coordinates and spatial image features; and fuse the key point image features and anchor frame preprocessing features to obtain anchor frame fused features.

[0141] In one embodiment, the occupancy probability calculation module 840 is further configured to obtain the pixel position of each grid in the target pixel coordinate system; the target pixel coordinate system is the pixel coordinate system of the spatial image; obtain the pixel range of the spatial image in the target pixel coordinate system, and take the grid whose pixel position is within the pixel range as the effective grid; obtain the distance between each grid and each anchor frame according to each second spatial information and each first spatial information, and determine the occupancy probability of each effective grid being occupied by each anchor frame according to the distance.

[0142] In one embodiment, the occupancy category calculation module 850 is further configured to: obtain occupancy grids from the effective grids according to each occupancy probability; obtain the anchor frame opacity in each first spatial information, and determine the category attribute quantization value of each anchor frame according to the anchor frame opacity and the first occupancy category information; obtain the contribution weight of each anchor frame occupying the occupancy grid according to each occupancy probability and the anchor frame opacity; obtain the semantic expectation of the anchor frame according to each contribution weight and the category attribute quantization value of each anchor frame; the semantic expectation is the average value of the target object category in the target space to which the anchor frame belongs; obtain the category attribute quantization value of each occupancy grid according to the occupancy probability and the semantic expectation; and determine the second occupancy category information of each occupancy grid if the category attribute quantization value of the occupancy grid is greater than a preset category threshold.

[0143] In one embodiment, the occupancy category calculation module 850 is further configured to use valid grids with an occupancy probability greater than a preset occupancy threshold as occupancy grids.

[0144] In one embodiment, the occupancy category calculation module 850 is further configured to obtain the conditional probability of each grid under a given anchor frame based on each occupancy probability; use the anchor frame opacity as the prior probability; and obtain the posterior probability of each anchor frame under a given grid based on the prior probability and the conditional probability, and use the posterior probability as the contribution weight.

[0145] In one embodiment, the anchor frame preprocessing module 820 is further configured to: extract the anchor frame features of each anchor frame based on the initial anchor frame information; project each anchor frame onto the pixel coordinate system of the spatial image to obtain the pixel position of each anchor frame; obtain the estimated anchor frame depth of each anchor frame based on the pixel position of each anchor frame and the estimated depth of the image; fuse the spatial position information and the estimated anchor frame depth in the initial anchor frame information, and convolve the fused features to obtain the anchor frame depth features of each anchor frame; and fuse the anchor frame features and the anchor frame depth features to obtain the anchor frame preprocessing features.

[0146] Each module in the aforementioned 3D occupancy grid sensing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0147] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, input / output interfaces (I / O), a display unit, and input devices. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface, display unit, and input devices are also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a 3D occupancy grid perception method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device.

[0148] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0149] In one embodiment, a vehicle is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps described in the embodiments of the 3D occupancy grid perception method.

[0150] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps described in the embodiments of the 3D occupancy grid sensing method.

[0151] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps described in various embodiments of the 3D occupancy grid perception method.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0153] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic resistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence processors, etc., and are not limited to these.

[0154] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0155] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A 3D occupancy grid sensing method, characterized in that, The method includes: Acquire spatial image features and image depth estimation of spatial images acquired in the target space; A preset number of anchor boxes are generated in the target space. Initial anchor box information for each anchor box is obtained. The initial anchor box information and the estimated depth of the image are fused to obtain anchor box preprocessing features for each anchor box. This includes: extracting anchor box features of each anchor box based on the initial anchor box information; projecting each anchor box onto the pixel coordinate system of the spatial image to obtain the pixel position of each anchor box; obtaining the estimated depth of each anchor box based on the pixel position and the estimated depth of the image; fusing the spatial position information in the initial anchor box information and the estimated depth of the anchor box, and convolving the fused features to obtain the anchor box depth features for each anchor box; and fusing the anchor box features and the anchor box depth features to obtain the anchor box preprocessing features. Based on the spatial image features, the preprocessed features of each anchor frame, and the anchor frame embedding features corresponding to the initial anchor frame information, the first spatial information and the first occupancy category information of each anchor frame are obtained. Multiple grids are generated for the target space. Based on the second spatial information of each effective grid and the first spatial information of each grid, the occupancy probability of each effective grid being occupied by each anchor frame is obtained. The occupancy probability is determined by the distance between each grid and each anchor frame. The number of grids is greater than the preset number. Based on the occupancy probabilities, the first spatial information, and the first occupancy category information, the second occupancy category information of each occupied grid in each effective grid is obtained; the second occupancy category information is determined by the category attribute quantization value of each occupied grid obtained based on the occupancy probabilities and the category attribute quantization value of each anchor frame, and the category attribute quantization value of the anchor frame is determined by the anchor frame opacity in each of the first spatial information and the first occupancy category information.

2. The method according to claim 1, characterized in that, The step of obtaining the first spatial information and first occupancy category information of each anchor frame based on the spatial image features, the preprocessed features of each anchor frame, and the anchor frame embedding features corresponding to the initialized anchor frame information includes: Feature aggregation is performed on the spatial image features and the preprocessed features of each anchor frame to obtain the anchor frame fusion features of each anchor frame; The initial anchor frame information is used as the basic information of each anchor frame, and the anchor frame fusion feature and the anchor frame embedding feature are fused. The fused features are refined by a preset multilayer perceptron model to obtain the incremental information of each anchor frame. Based on the data types of the basic information and the incremental information, element-level operations are performed on the basic information and the incremental information to obtain the first spatial information and the first occupancy category information of each anchor frame.

3. The method according to claim 2, characterized in that, The step of aggregating the spatial image features and the preprocessed features of each anchor frame to obtain the anchor frame fusion features of each anchor frame includes: Multiple key points within a preset neighborhood of each anchor frame are obtained, and each key point is projected onto the spatial image to obtain the key point pixel coordinates of each key point. Based on the pixel coordinates of each key point and the spatial image features, extract the key point image features corresponding to each key point; The key point image features and the anchor frame preprocessing features are fused to obtain the anchor frame fused features.

4. The method according to claim 1, characterized in that, The step of obtaining the occupancy probability of each effective grid being occupied by each anchor frame based on the second spatial information of each effective grid and the first spatial information of each grid includes: Obtain the pixel position of each of the grids in the target pixel coordinate system; the target pixel coordinate system is the pixel coordinate system of the spatial image; Obtain the pixel range of the spatial image in the target pixel coordinate system, and take the grid cells whose pixel positions are within the pixel range as the valid grid cells; Based on each of the second spatial information and each of the first spatial information, the distance between each grid and each of the anchor frames is obtained, and the occupancy probability of each effective grid being occupied by each of the anchor frames is determined based on the distance.

5. The method according to claim 1, characterized in that, The step of obtaining the second occupancy category information of each occupied grid in each effective grid based on each occupancy probability, each first spatial information, and each first occupancy category information includes: The occupied grid is obtained from the effective grid according to each of the occupancy probabilities; Obtain the opacity of the anchor frame in each of the first spatial information, and determine the category attribute quantization value of each anchor frame based on the anchor frame opacity and the first occupancy category information; Based on the occupancy probability and the anchor frame opacity, obtain the contribution weight of the occupancy grid being occupied by each anchor frame; Based on the contribution weights and the category attribute quantification values ​​of each anchor box, the semantic expectation of the anchor box is obtained; the semantic expectation is the average value of the anchor box belonging to the target object category in the target space. Based on the occupancy probability and the semantic expectation, the category attribute quantization value of each occupied grid is obtained. If the category attribute quantization value of the occupied grid is greater than a preset category threshold, the second occupancy category information of each occupied grid is determined.

6. The method according to claim 5, characterized in that, The step of obtaining the occupied grid from the valid grid according to each of the occupancy probabilities includes: The effective grid cells whose occupancy probability is greater than the preset occupancy threshold are designated as the occupied grid cells.

7. The method according to claim 5, characterized in that, The step of obtaining the contribution weight of the occupied grid being occupied by each of the anchor frames based on each of the occupancy probabilities and the anchor frame opacity includes: Based on the occupancy probabilities, the conditional probabilities of each grid cell under a given anchor frame are obtained. The anchor frame opacity is used as the prior probability. Based on the prior probability and the conditional probability, the posterior probability of each anchor frame under the given grid condition is obtained. The posterior probability is used as the contribution weight.

8. A 3D occupancy grid sensing device, characterized in that, The device includes: The image processing module is used to acquire spatial image features and image depth estimation of spatial images acquired from the target space; An anchor frame preprocessing module is used to generate a preset number of anchor frames in the target space, obtain the initial anchor frame information of each anchor frame, fuse the initial anchor frame information and the estimated depth of the image to obtain the anchor frame preprocessing features of each anchor frame. The feature encoding module is used to obtain the first spatial information and the first occupancy category information of each anchor frame based on the spatial image features, the preprocessed features of each anchor frame, and the anchor frame embedding features corresponding to the initial anchor frame information. An occupancy probability calculation module is used to generate multiple grids in the target space, and obtain the occupancy probability of each effective grid being occupied by each anchor frame based on the second spatial information of each effective grid and the first spatial information of each grid. The occupancy probability is determined by the distance between each grid and each anchor frame. The number of grids is greater than the preset number. The occupancy category calculation module is used to obtain second occupancy category information for each occupied grid in each effective grid based on each occupancy probability, each first spatial information, and each first occupancy category information; the second occupancy category information is determined by the category attribute quantization value of each occupied grid obtained based on the occupancy probability and the category attribute quantization value of each anchor frame, and the category attribute quantization value of the anchor frame is determined by the anchor frame opacity in each first spatial information and each first occupancy category information.

9. A vehicle comprising a memory and a processor, said memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images