Method for determining occupancy information, method and apparatus for training occupancy prediction model

Abstract

A method for determining occupancy information, a method and device for training an occupancy prediction model are disclosed. The method for determining occupancy information comprises determining a first image and a second image; determining a depth probability distribution corresponding to the first image based on the first image and the second image; determining a view volume feature corresponding to the first image based on the first image and the depth probability distribution; projecting and aggregating the view volume feature to obtain a two-dimensional bird's eye view feature corresponding to the view volume feature; and determining occupancy information of a target scene based on the two-dimensional bird's eye view feature. The technical solution of the disclosure determines the depth probability distribution of the first image in the binocular image first, and then determines the view volume feature based on the first image and the depth probability distribution. Then, the two-dimensional bird's eye view feature is determined based on the view volume feature, so as to determine the occupancy information of the target scene. In this way, the three-dimensional voxel feature can be avoided to be explicitly constructed in the intermediate stage, thereby solving the problems of high calculation complexity and large memory occupation of the existing three-dimensional space occupancy prediction.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a method for determining occupancy information, a method for training an occupancy prediction model, and an apparatus. Background Technology

[0002] Occupancy Network (OCC) provides intelligent agents (such as intelligent driving vehicles or robots) with refined 3D environmental geometry and semantic information by predicting the occupancy state of voxels in 3D space. It is a key technology in fields such as embodied intelligent robots, autonomous driving and drones.

[0003] However, in pursuit of ultimate perception accuracy and range, spatial occupancy prediction algorithms in related technologies typically employ extremely complex network structures, such as containing a large number of 3D (Three-Dimensional) convolutional or Transformer (a neural network model based on self-attention mechanism) layers. This results in extremely high hardware computing power requirements during actual deployment and makes it difficult to balance perception accuracy and inference efficiency. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a method for determining occupancy information, a method and apparatus for training an occupancy prediction model, which can solve the technical problem in related technologies where it is difficult to balance perception accuracy and reasoning efficiency.

[0005] A first aspect of this disclosure provides a method for determining occupancy information. The method includes: determining a first image and a second image of a target scene acquired by an image sensor at the same time; determining a depth probability distribution corresponding to the first image based on the first image and the second image; determining a frustum feature corresponding to the first image based on the first image and the depth probability distribution; performing projection aggregation processing on the frustum feature to obtain a two-dimensional bird's-eye view feature corresponding to the frustum feature; and determining the occupancy information of the target scene based on the two-dimensional bird's-eye view feature.

[0006] A second aspect of this disclosure provides a training method for an occupancy prediction model. The training method includes: determining a disparity depth estimation network and an initial occupancy network; determining at least one set of sample images and label information; wherein the set of sample images includes a third image and a fourth image acquired by an image sensor at the same time; processing the third image and the fourth image using the disparity depth estimation network to obtain an initial depth probability distribution corresponding to the third image; processing the third image and the initial depth probability distribution using the initial occupancy network to obtain initial frustum features corresponding to the third image, and performing projection aggregation processing on the initial frustum features to obtain two-dimensional bird's-eye view features; determining initial occupancy information based on the two-dimensional bird's-eye view features; iteratively training the disparity depth estimation network and the initial occupancy network based on the initial occupancy information and label information to obtain a trained disparity depth estimation network and a trained occupancy network; and obtaining a trained occupancy prediction model based on the trained disparity depth estimation network and the trained occupancy network.

[0007] A third aspect of this disclosure provides an occupancy information determination device, comprising: an image determination module for determining a first image and a second image of a target scene acquired by an image sensor at the same time; a depth determination module for determining a depth probability distribution corresponding to the first image based on the first image and the second image; a frustum feature determination module for determining a frustum feature corresponding to the first image based on the first image and the depth probability distribution; a bird's-eye view feature determination module for performing projection aggregation processing on the frustum feature to obtain a two-dimensional bird's-eye view feature corresponding to the frustum feature; and an occupancy information determination module for determining the occupancy information of the target scene based on the two-dimensional bird's-eye view feature.

[0008] A fourth aspect of this disclosure provides a training apparatus for an occupancy prediction model. The training apparatus includes: a network determination module for determining a disparity depth estimation network and an initial occupancy network; a sample determination module for determining at least one set of sample images and label information; wherein the set of sample images includes a third image and a fourth image acquired by an image sensor at the same time; a disparity depth estimation module for processing the third image and the fourth image using the disparity depth estimation network to obtain an initial depth probability distribution corresponding to the third image; an occupancy prediction module for processing the third image and the initial depth probability distribution using the initial occupancy network to obtain initial frustum features corresponding to the third image, and performing projection aggregation processing on the initial frustum features to obtain two-dimensional bird's-eye view features; determining initial occupancy information based on the two-dimensional bird's-eye view features; a training module for iteratively training the disparity depth estimation network and the initial occupancy network based on the initial occupancy information and label information to obtain a trained disparity depth estimation network and a trained occupancy network; and a processing module for obtaining a trained occupancy prediction model based on the trained disparity depth estimation network and the trained occupancy network.

[0009] A fifth aspect of this disclosure provides a computer-readable storage medium storing a computer program for performing the occupancy information determination method provided in the first aspect embodiment, or the occupancy prediction model training method provided in the second aspect embodiment.

[0010] A sixth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor configured to read executable instructions from the memory and execute the instructions to implement the occupancy information determination method provided in the first aspect embodiment above, or the occupancy prediction model training method provided in the second aspect embodiment above.

[0011] A seventh aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs the occupancy information determination method provided in the first aspect embodiment above, or the occupancy prediction model training method provided in the second aspect embodiment above.

[0012] Based on the occupancy information determination method provided in this disclosure, a first image and a second image of the target scene acquired by an image sensor at the same time are determined; based on the first image and the second image, the depth probability distribution corresponding to the first image is determined; based on the first image and the depth probability distribution, the view frustum feature corresponding to the first image is determined; the view frustum feature is subjected to projection aggregation processing to obtain the two-dimensional bird's-eye view feature corresponding to the view frustum feature; based on the two-dimensional bird's-eye view feature, the occupancy information of the target scene is determined. Thus, by projecting and aggregating the view frustum features to obtain two-dimensional bird's-eye view features, the explicit construction of three-dimensional voxel features in the intermediate stage of OCC three-dimensional occupancy prediction can be avoided, thereby solving the problems of large memory consumption and high memory bandwidth requirements of existing solutions. Furthermore, the entire process relies on the first and second images to generate depth probability distributions and convert them layer by layer into three-dimensional view frustum features. The three-dimensional view frustum features are then converted into two-dimensional bird's-eye view features, and the occupancy information is determined through the two-dimensional bird's-eye view features. This eliminates the need to perform a large number of three-dimensional convolution and three-dimensional Transformer operations, thereby reducing the overall computational complexity. In addition, the simultaneous reduction in computational load and memory overhead can reduce inference latency and reduce the continuous power consumption of hardware, thereby solving the problems of poor inference real-time performance and excessive power consumption of existing solutions. At the same time, since the depth probability distribution obtained by jointly solving the first and second images can solve the uncertainty of the modeling space depth and completely preserve the three-dimensional spatial information of the scene, the perception accuracy of occupancy prediction is guaranteed. Therefore, this disclosure can achieve spatial occupancy prediction of the target scene while taking into account both perception accuracy and inference speed. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of an intelligent vehicle in a driving scenario provided for an exemplary embodiment of this disclosure.

[0014] Figure 2 A flowchart illustrating a method for determining occupancy information provided as an exemplary embodiment of this disclosure.

[0015] Figure 3 A flowchart illustrating a method for determining occupancy information provided as another exemplary embodiment of this disclosure.

[0016] Figure 4 A flowchart illustrating a method for determining occupancy information, which is yet another exemplary embodiment of this disclosure.

[0017] Figure 5 A flowchart illustrating a method for determining occupancy information, provided as another exemplary embodiment of this disclosure.

[0018] Figure 6 A flowchart illustrating a method for determining occupancy information, which is yet another exemplary embodiment of this disclosure.

[0019] Figure 7A flowchart illustrating a method for determining occupancy information, which is yet another exemplary embodiment of this disclosure.

[0020] Figure 8 A flowchart illustrating a method for determining occupancy information, which is yet another exemplary embodiment of this disclosure.

[0021] Figure 9 A flowchart illustrating a method for determining occupancy information, which is yet another exemplary embodiment of this disclosure.

[0022] Figure 10 This is a flowchart illustrating a training method for an occupancy prediction model provided as an exemplary embodiment of the present disclosure.

[0023] Figure 11 A schematic diagram of the structure of an occupancy information determination device provided for an exemplary embodiment of this disclosure.

[0024] Figure 12 A schematic diagram of the structure of a device for determining occupancy information provided as another exemplary embodiment of this disclosure.

[0025] Figure 13 A schematic diagram of the structure of a device for determining occupancy information provided as another exemplary embodiment of this disclosure.

[0026] Figure 14 A schematic diagram of the structure of a device for determining occupancy information provided as another exemplary embodiment of this disclosure.

[0027] Figure 15 A schematic diagram of the structure of a training apparatus for an occupation prediction model provided as an exemplary embodiment of the present disclosure.

[0028] Figure 16 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0029] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0030] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0031] Application Overview First, the application scenarios of this disclosure will be introduced. The method for determining occupancy information provided in the embodiments of this disclosure can be applied to multiple technical fields such as intelligent vehicles, drones, and mobile robots, covering scenarios such as intelligent driving, assisted driving, autonomous flight navigation of drones, robot environmental perception and obstacle avoidance, as well as any other feasible environmental perception scenarios.

[0032] like Figure 1 As shown, the intelligent device 10 (such as an intelligent vehicle) is equipped with at least one set of binocular image sensors 11. The binocular image sensors 11 can be sensors corresponding to the forward-looking binocular acquisition module of the device 10, the side-looking binocular acquisition module of the intelligent device 10, or the rear-looking binocular acquisition module of the intelligent device 10, etc. Of course, for drones or robots, the binocular image sensors 11 can be sensors corresponding to the downward-looking binocular module of the drone, the forward obstacle avoidance binocular module of the mobile robot, etc. Furthermore, the target scene around the device can be synchronously acquired through the binocular image sensors 11 in the same set, obtaining two frames of images of the target scene at the same time, namely the first image and the second image, often referred to as the left and right images.

[0033] Furthermore, the occupancy prediction model can be used to process the first and second images acquired by the binocular image sensor 11 to obtain the three-dimensional occupancy information of the target scene. It should be noted that... Figure 1 This disclosure is merely an illustrative example and does not represent the actual situation. Figure 1 The number, location, and type of image sensors 11 are not limited.

[0034] In pursuit of ultimate perception accuracy and range, OCC occupancy prediction methods in related technologies typically employ extremely complex network structures and massive computational processes, which can lead to the following drawbacks: (1) High computational complexity: Existing OCC models have a large number of 3D convolutions, Transformer layers or dense multi-view feature fusion, which leads to a huge amount of computation.

[0035] (2) High memory and bandwidth requirements: To achieve three-dimensional spatial modeling, the existing OCC model needs to construct a large-scale dense three-dimensional voxel representation and perform a large number of 3D convolution and three-dimensional Transformer operations based on the voxel. The three-dimensional voxel representation and the intermediate feature map in the form of three-dimensional voxels will occupy a huge amount of memory. At the same time, the huge amount of computation will lead to frequent data reading and writing, which has extremely high requirements for memory bandwidth.

[0036] (3) High latency and power consumption: The huge amount of computation and memory access directly leads to excessively high inference latency, which cannot meet the needs of real-time decision-making by intelligent agents. At the same time, continuous high-load operation is very unfriendly to devices such as mobile robots that rely on battery power.

[0037] To address the aforementioned issues, this disclosure provides a method for determining occupancy information. This method involves determining a first image and a second image of a target scene acquired simultaneously by an image sensor; determining a depth probability distribution corresponding to the first image based on the first and second images; determining a frustum feature corresponding to the first image based on the first image and the depth probability distribution; performing projection aggregation processing on the frustum feature to obtain a two-dimensional bird's-eye view feature corresponding to the frustum feature; and determining the occupancy information of the target scene based on the two-dimensional bird's-eye view feature. Thus, by projecting and aggregating the view frustum features to obtain two-dimensional bird's-eye view features, the explicit construction of three-dimensional voxel features in the intermediate stage of OCC three-dimensional occupancy prediction can be avoided, thereby solving the problems of large memory consumption and high memory bandwidth requirements of existing solutions. Furthermore, the entire process relies on the first and second images to generate depth probability distributions and convert them layer by layer into three-dimensional view frustum features. The three-dimensional view frustum features are then converted into two-dimensional bird's-eye view features, and the occupancy information is determined through the two-dimensional bird's-eye view features. This eliminates the need to perform a large number of three-dimensional convolution and three-dimensional Transformer operations, thereby reducing the overall computational complexity. In addition, the simultaneous reduction in computational load and memory overhead can reduce inference latency and reduce the continuous power consumption of hardware, thereby solving the problems of poor inference real-time performance and excessive power consumption of existing solutions. At the same time, since the depth probability distribution obtained by jointly solving the first and second images can solve the uncertainty of the modeling space depth and completely preserve the three-dimensional spatial information of the scene, the perception accuracy of occupancy prediction is guaranteed. Therefore, this disclosure can achieve spatial occupancy prediction of the target scene while taking into account both perception accuracy and inference speed.

[0038] Exemplary methods Figure 2 This is a flowchart illustrating a method for determining occupancy information provided as an exemplary embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices, such as... Figure 2 As shown, the method includes steps S21-S25.

[0039] S21. Determine the first and second images of the target scene acquired by the image sensor at the same time.

[0040] For example, such as Figure 1As shown, a first image and a second image can be acquired simultaneously by the first imaging unit and the second imaging unit in at least one binocular image sensor 11 on the smart device 10. These first and second images constitute a binocular image, both of which are images of the target scene acquired at a first moment. The target scene refers to the three-dimensional spatial scene surrounding the smart device 10 within the field of view of the binocular image sensor 11, where environmental perception and occupancy prediction are required. For example, the first image and the second image can be the left image acquired by the first imaging unit and the right image acquired by the second imaging unit in the binocular image sensor 11, respectively. Alternatively, the first image can be a reference image in the binocular image, and the second image can be a matching image in the binocular image.

[0041] In this context, the reference image generally refers to the image used as a spatial coordinate reference to generate pixel query positions. The disparity, depth, and 3D coordinates of all pixels are based on the camera coordinate system of this image. The matching image generally refers to another image used to calculate disparity, where each pixel in the reference image is used as a reference to search for corresponding matching points laterally. The matching image is only used to provide matching clues and is not used as the final geometric output reference.

[0042] S22. Based on the first image and the second image, determine the depth probability distribution corresponding to the first image.

[0043] For example, the depth probability distribution corresponding to the first image refers to the probability that each pixel in the first image corresponds to a different depth value. The depth probability distribution can more accurately express the uncertainty of depth estimation, and compared with directly outputting a definite depth value, it can provide a richer and more accurate feature basis for the subsequent determination of occupancy information.

[0044] S23. Based on the first image and the depth probability distribution, determine the view frustum features corresponding to the first image.

[0045] For example, the first image features and depth probability distribution of the first image can be extended firstly to obtain the corresponding extended first image features and extended depth probability distribution; then the extended first image features and extended depth probability distribution can be weighted and fused to obtain the view frustum features corresponding to the first image.

[0046] For example, the first image features corresponding to the first image and the original depth probability distribution (which is the depth probability distribution obtained through S22) can be dimensionally expanded to obtain the first expanded image features and the depth expanded probability distribution. Then, the first expanded image features are spatially unfolded along the ray direction of the image sensor, and a distribution weighting coefficient is assigned to each depth position in combination with the depth expanded probability distribution to form a frustum feature with the distribution weighting coefficient. This frustum feature is the frustum feature corresponding to the first image. Here, the ray direction of the image sensor refers to the ray direction starting from the optical center of the image sensor and pointing to each pixel in the first image, corresponding to different depth positions in three-dimensional space. That is to say, the embodiments of this disclosure can expand the expanded first image features according to the depth expanded probability distribution to construct a frustum feature with binocular geometric probability constraints.

[0047] S24. Perform projection aggregation processing on the view frustum features to obtain the two-dimensional bird's-eye view features corresponding to the view frustum features.

[0048] S25. Based on the features of the two-dimensional bird's-eye view, determine the occupancy information of the target scene.

[0049] For example, 2D bird's-eye view features can be encoded using a BEV Encoder (BEV encoding), and a lightweight channel-to-height decoding method can be employed to convert the encoded 2D bird's-eye view features into 3D occupancy prediction results. The channel-to-height lightweight decoding method can convert the feature channels of the 2D bird's-eye view features into height and occupancy class dimensions through 2D convolution or channel mapping. Compared to traditional OCC occupancy prediction schemes that rely on 3D convolution to construct complete 3D voxels before decoding and outputting occupancy results, this scheme eliminates the need to construct dense 3D intermediate representations, thereby significantly reducing the computational load, memory consumption, and inference latency of OCC occupancy prediction during the decoding stage.

[0050] The method for determining occupancy information provided in this embodiment of the present disclosure involves determining a first image and a second image of a target scene acquired by an image sensor at the same time; determining a depth probability distribution corresponding to the first image based on the first image and the second image; determining a frustum feature corresponding to the first image based on the first image and the depth probability distribution; performing projection aggregation processing on the frustum feature to obtain a two-dimensional bird's-eye view feature corresponding to the frustum feature; and determining the occupancy information of the target scene based on the two-dimensional bird's-eye view feature. Thus, by projecting and aggregating the view frustum features to obtain two-dimensional bird's-eye view features, the explicit construction of three-dimensional voxel features in the intermediate stage of OCC three-dimensional occupancy prediction can be avoided, thereby solving the problems of large memory consumption and high memory bandwidth requirements of existing solutions. Furthermore, the entire process relies on the first and second images to generate depth probability distributions and convert them layer by layer into three-dimensional view frustum features. The three-dimensional view frustum features are then converted into two-dimensional bird's-eye view features, and the occupancy information is determined through the two-dimensional bird's-eye view features. This eliminates the need to perform a large number of three-dimensional convolution and three-dimensional Transformer operations, thereby reducing the overall computational complexity. In addition, the simultaneous reduction in computational load and memory overhead can reduce inference latency and reduce the continuous power consumption of hardware, thereby solving the problems of poor inference real-time performance and excessive power consumption of existing solutions. At the same time, since the depth probability distribution obtained by jointly solving the first and second images can solve the uncertainty of the modeling space depth and completely preserve the three-dimensional spatial information of the scene, the perception accuracy of occupancy prediction is guaranteed. Therefore, this disclosure can achieve spatial occupancy prediction of the target scene while taking into account both perception accuracy and inference speed.

[0051] like Figure 3 As shown above, in the above Figure 2 Based on the illustrated embodiment, step S24 may include steps S241-S244.

[0052] S241. Determine the preset space configuration parameters.

[0053] For example, the preset spatial configuration parameters in this embodiment include, but are not limited to, preset BEV (Bird's Eye View) spatial range and grid resolution. The BEV spatial range refers to the spatial range that the plane corresponding to the features of the two-dimensional bird's-eye view needs to cover in the carrier coordinate system. It can be set according to the perception requirements of the actual application scenario. For example, for intelligent driving scenarios, the BEV spatial range can be set to a planar area centered on the intelligent vehicle, with X meters in front and behind, and Y meters to the left and right. The grid resolution refers to the actual spatial size corresponding to each grid unit on the bird's-eye view plane. The higher the grid resolution, the higher the accuracy of the final occupancy information, but the corresponding computational load will also increase slightly. It can be balanced according to actual needs.

[0054] S242. Perform coordinate transformation on the view frustum features to obtain the view frustum features in the carrier coordinate system.

[0055] For example, the carrier coordinate system in this embodiment refers to a three-dimensional coordinate system established based on the smart device, which can unify the spatial coordinates of image features acquired from different positions and angles for subsequent unified projection aggregation processing. The view frustum feature determined in step S23 is located in the view frustum space, and the shape of the view frustum feature is [B, C_f, N_z, H_f, W_f]. Among them, N_z is the depth candidate dimension, which is used to characterize the multiple candidate spatial positions of the first image feature along the ray direction of the image sensor. The image sensor ray refers to a virtual ray emitted from the optical center (lens center) of the image sensor to each pixel on the image plane. This straight line from the optical center through the pixel and extending into the three-dimensional space is the sensor ray corresponding to that pixel. B is the batch size, that is, the number of samples sent to the network for model inference / training at the same time during one model inference / training. C_f is the number of image feature channels of the first image feature, and H_f and W_f are the dimensions of the downsampled image features when extracting features from the first and second images.

[0056] In some examples, for a position (u, v) and a depth candidate z in the first image feature, the frustum feature can be back-projected to a 3D spatial position in the camera coordinate system based on the parameters of the image sensor (such as camera intrinsic parameters). Then, based on the parameters of the image sensor (such as camera extrinsic parameters), the back-projected feature is transformed to the carrier coordinate system (such as the vehicle coordinate system or robot coordinate system), thereby obtaining the frustum feature in the carrier coordinate system in step S242. The camera intrinsic parameters include parameters such as the camera's focal length and principal point position, used to characterize the projection geometric relationship between the camera's optical center and the image plane pixels; the camera extrinsic parameters include the camera's rotation matrix and translation vector relative to the carrier coordinate system, used to characterize the spatial transformation relationship between the camera coordinate system and the carrier coordinate system.

[0057] S243. Based on the spatial configuration parameters, the view frustum features in the carrier coordinate system are projected onto the grid positions corresponding to the bird's-eye view feature plane to obtain the projected bird's-eye view grid features.

[0058] For example, for each view frustum feature point in the carrier coordinate system, the corresponding two-dimensional bird's-eye view grid coordinates can be determined based on its coordinate position in the carrier coordinate system and a preset spatial configuration parameter. Then, the feature information of the view frustum feature point is assigned to the corresponding grid position. In this way, after projecting all view frustum feature points, the projected bird's-eye view grid feature is obtained. If multiple view frustum feature points are projected to the same bird's-eye view grid position, the feature information of these feature points can be temporarily retained for subsequent aggregation processing. Here, the bird's-eye view feature plane refers to the horizontal xy-plane in the carrier coordinate system, corresponding to the horizontal plane of the scene viewed from above in three-dimensional space. All view frustum features will be projected and aggregated onto this plane to obtain the two-dimensional bird's-eye view feature.

[0059] In some examples, the frustum features corresponding to the 3D frustum spatial location can be assigned to a grid position (i,j) in the BEV plane according to a preset BEV spatial range and grid resolution. During projection, frustum features from different image pixels and candidates at different depths may fall into the same BEV grid.

[0060] S244. Perform feature aggregation processing on the bird's-eye view grid features to obtain two-dimensional bird's-eye view features.

[0061] For example, for multiple view frustum features falling into the same BEV grid, a preset aggregation method can be used to perform feature aggregation processing on these features, so that each BEV grid outputs only one aggregated feature. By stitching together the aggregated features of all grids, a complete two-dimensional bird's-eye view feature can be obtained. The feature aggregation methods in this embodiment are, for example, but not limited to, max pooling, average pooling, or weighted summation, etc., and this embodiment does not limit these methods. In this embodiment, feature aggregation integrates and compresses all view frustum features into grid features on the two-dimensional bird's-eye view plane, thereby completing the entire projection aggregation process without storing dense three-dimensional voxel features, thus effectively reducing memory usage.

[0062] In some examples, for multiple view frustum features falling within the same BEV grid, aggregation methods such as summation, averaging, max pooling, or learning weighting can be used to aggregate these features, resulting in a 2D bird's-eye view feature. If the shape of the view frustum feature is [B, C_f, N_z, H_f, W_f], then the shape of the projected aggregated 2D bird's-eye view feature is [B, C_bev, X, Y]. Here, C_bev is the number of feature channels in the 2D bird's-eye view feature, and X and Y are the number of grids obtained by dividing the preset BEV spatial range along the x-axis and y-axis of the carrier coordinate system, respectively, representing the planar dimensions of the 2D bird's-eye view feature.

[0063] After transforming each feature point in the view frustum feature to the carrier coordinate system, each feature point corresponds to a determined bird's-eye view plane coordinate in the carrier coordinate system. By mapping this coordinate to the preset bird's-eye view grid position and aggregating multiple features within the same grid, a complete two-dimensional bird's-eye view feature can be obtained. The entire process does not require explicit maintenance of large-scale three-dimensional voxel features in the intermediate stage. Feature organization is completed only on the two-dimensional plane through projection aggregation, which greatly reduces memory usage and computation.

[0064] View frustum features from different image pixels and different depth candidates may fall into the same BEV grid. It should be noted that in this embodiment, explicit 3D voxel features do not need to be retained after step S244 because the depth candidate dimension N_z is mainly used to determine the projection position of the first image feature in space. After BEV projection and aggregation of the 3D view frustum features, the 3D view frustum feature representation is compressed into a 2D BEV feature. This avoids maintaining large-scale 3D voxel features in the intermediate stage and also avoids subsequent high-overhead 3D convolution or 3D Transformer calculations on the 3D voxel features.

[0065] The method for determining occupancy information provided in this disclosure can ensure projection accuracy by utilizing binocular probabilistic depth priors, while controlling computational load and memory usage through two-dimensional BEV aggregation. Therefore, it can obtain highly accurate three-dimensional occupancy information without explicitly maintaining large-scale three-dimensional voxel features in the intermediate stage.

[0066] like Figure 4 As shown above, in the above Figure 2 Based on the illustrated embodiment, step S25 may include steps S251-S253.

[0067] S251. Perform feature encoding on the two-dimensional bird's-eye view features to obtain the encoded two-dimensional features.

[0068] For example, a BEV encoder based on a convolutional neural network can be used to perform feature encoding on the features of a two-dimensional bird's-eye view, extracting global contextual information under different receptive fields to obtain encoded two-dimensional features with more complete semantic information. For example, stacked residual convolutional modules or Transformer modules can be used to complete the feature encoding of the two-dimensional bird's-eye view features, and this disclosure does not limit this.

[0069] In some examples, the 2D bird's-eye view features can be input into the BEV Encoder module for 2D spatial feature encoding to obtain encoded 2D features. The BEV Encoder module refers to a neural network module used to extract contextual features from the grid features on the 2D bird's-eye view plane. This module further fuses local spatial information and contextual information within the BEV plane. Since both the input and output of this module are 2D BEV feature maps, it can be implemented using 2D convolution, 2D residual modules, or other 2D feature encoding structures. Furthermore, unlike feature encoding directly in 3D voxel space, the BEV encoding in this embodiment always operates on the 2D BEV plane, thereby significantly reducing intermediate feature storage requirements and computational complexity.

[0070] In some examples, if the shape of the 2D bird's-eye view feature is [B, C_bev, X, Y], then the shape of the encoded 2D feature is [B, C_e, X, Y]. Here, C_e represents the number of channels in the encoded 2D feature.

[0071] S252. Perform channel mapping processing on the encoded two-dimensional features to obtain two-dimensional output features.

[0072] like Figure 5 As shown above, in the above Figure 4 Based on the illustrated embodiment, step S252 may include steps S2521-S2522.

[0073] S2521. Determine the target number of channels; wherein, the target number of channels is determined by the occupancy category dimension and the height dimension. The height dimension is the dimension obtained by dividing the three-dimensional space where the target scene is located along the vertical direction, and the occupancy category dimension is the dimension used to represent the category corresponding to the voxels in the three-dimensional space.

[0074] For example, the specific values ​​of the occupancy category dimension and height dimension can be preset according to the actual task requirements, and multiplying the two yields the target number of channels in step S2521. For instance, for intelligent driving scenarios on open roads, the number of voxel layers Z in the vertical direction and the number of occupancy semantic categories C_occ to be identified can be set according to the perception range and accuracy requirements. Then, multiplying the two yields the target number of channels C_out.

[0075] S2522. Perform channel mapping processing on the encoded two-dimensional features to map the original number of channels of the two-dimensional features to the target number of channels, thereby obtaining the two-dimensional output features.

[0076] For example, channel mapping processing includes, but is not limited to, two-dimensional convolution operations and channel mapping operations. The role of channel mapping processing is to pre-divide the channel dimensions of BEV plane features into height dimension and occupancy class dimension, laying the foundation for subsequent generation of three-dimensional occupancy prediction results. At the same time, the entire mapping process is only performed on two-dimensional features, without introducing additional three-dimensional computational overhead.

[0077] In some examples, a 1×1 2D convolutional layer can be used to process the encoded 2D features. During processing, only the channel dimension is transformed, without changing the spatial dimensions of the features in the X and Y axes. This converts the original number of channels to the preset target number, resulting in 2D output features that meet the dimensionality requirements. This operation is only a 2D computation, which significantly reduces computational complexity and memory usage compared to 3D convolution operations, facilitating deployment on edge devices. Here, the X and Y axes refer to the X and Y axes in the vehicle coordinate system. Taking the vehicle's VCS coordinate system as an example, the VCS coordinate system is a vehicle-centric coordinate system. Its origin is a specific reference point on the vehicle (such as the center of the rear axle), the X-axis points in the vehicle's forward direction, the Y-axis points to the left or right side of the vehicle, and the Z-axis points upwards towards the vehicle and is perpendicular to the plane formed by the X and Y axes.

[0078] By obtaining two-dimensional output features through steps S2521 to S2522, the dimensional structure required for three-dimensional occupying information can be directly constructed from the two-dimensional encoded features without introducing high-complexity three-dimensional convolution calculations. This ensures the dimensional integrity of the final occupied information while maintaining low computational overhead, thus improving the deployment adaptability of the method.

[0079] In some examples, step S252 generates a 3D occupancy prediction result from the 2D bird's-eye view features without explicitly constructing 3D voxel features and performing 3D convolution. If the shape of the encoded 2D feature is [B, C_e, X, Y], then the number of channels C_e of the 2D feature can be mapped to C_out = C_occ × Z through 2D convolution or channel mapping operations to obtain the 2D output feature with the shape [B, C_occ × Z, X, Y]. Here, Z is the number of voxel layers in the height direction, C_occ is the number of occupied classes, and C_out is the product of the number of voxel layers in the height direction and the number of occupied classes. Therefore, through channel mapping processing, the encoded 2D feature with the shape [B, C_e, X, Y] can be converted into a 2D output feature with the shape [B, C_occ × Z, X, Y].

[0080] S253. Perform dimensional rearrangement on the two-dimensional output features to obtain the occupancy information of the target scene.

[0081] For example, the channel dimension C_out can be split into an occupancy category dimension and a height dimension through a dimension rearrangement operation, thereby obtaining three-dimensional occupancy information. The shape of the two-dimensional output feature after dimension rearrangement is [B, C_occ, Z, X, Y]. Furthermore, the number of category channels C_occ can be flexibly adapted for different prediction tasks: if the occupancy prediction task is binary classification occupancy prediction, only distinguishing whether space is occupied, then C_occ can be set to 2, corresponding to the idle and occupied states respectively; alternatively, a single-channel output can be used, with the channel value directly representing the occupancy probability of the voxel. If the occupancy prediction task is semantic occupancy prediction, requiring the distinction of different objects in space, then C_occ corresponds to the total number of all object categories that can be recognized. Each channel corresponds to a category of object labels (such as vehicles, pedestrians, roads, etc.), and the prediction score within a channel represents the confidence that the voxel is occupied by an object of the category corresponding to that channel. By using the channel with the highest prediction score among all channels of C_occ, the specific category of the object occupying the voxel can be determined. For example, if the model can recognize five semantic categories: idle, vehicle, pedestrian, vegetation, and building, then C_occ is configured to 5, with each channel corresponding to one of the semantic categories, and the channel with the highest prediction score is taken as the category of the object occupying that voxel.

[0082] In the method for determining occupancy information provided in this embodiment, the decoding method shown in steps S252-S253 is not the only geometric source in this disclosure, but is used in conjunction with binocular probabilistic prior-driven frustum feature construction and two-dimensional bird's-eye view mesh feature aggregation. Steps S21-S24 are responsible for placing the first image features into the BEV space as accurately as possible, while steps S252-S253 are responsible for recovering the three-dimensional occupancy output from the two-dimensional bird's-eye view features with low computational overhead. In this way, this embodiment can balance the spatial positioning capabilities provided by binocular geometric constraints with the end-side deployment efficiency provided by two-dimensional BEV representation.

[0083] like Figure 6 As shown above, in the above Figure 4 Based on the illustrated embodiment, step S253 may include steps S2531-S2533.

[0084] S2531. Perform dimensional splitting on the two-dimensional output features along the channel dimension to obtain multiple independent dimensions, including at least the height dimension and the category dimension.

[0085] For example, C_out = C_occ × Z, which is merged in the channel dimension, can be split into two independent dimensions: the occupation category dimension C_occ corresponding to the occupation category, and the height dimension Z corresponding to the vertical layer. After the split, the original spatial dimensions in the X and Y axes remain unchanged.

[0086] S2532. Based on the multiple independent dimensions obtained from the splitting, the two-dimensional output features are reorganized to obtain three-dimensional occupancy features.

[0087] For example, the split height dimension Z is reorganized into a spatial dimension, which together with the original X-axis and Y-axis dimensions of the BEV plane constitutes a three-dimensional spatial dimension, ultimately resulting in a three-dimensional occupied feature with the shape [B, C_occ, Z, X, Y]. The entire reorganization process only requires rearranging the feature dimensions, without the need for additional convolution calculations, and does not introduce additional computational overhead.

[0088] S2533. Based on the three-dimensional occupancy features, determine the occupancy information of the target scene.

[0089] For example, since the shape of the 3D occupancy feature is [B, C_occ, Z, X, Y], which is the original 3D occupancy feature tensor, basic occupancy prediction information such as raster-level category and occupancy confidence can be directly extracted from the 3D occupancy feature. Based on this, the complete 3D occupancy result is obtained through operations such as coordinate mapping, thresholding, and post-processing.

[0090] The method for determining occupancy information provided in this disclosure can obtain the occupancy features of the three-dimensional spatial dimension by simply rearranging the dimensions, without the need for explicit feature storage and convolution calculation of the three-dimensional voxels. Thus, while retaining the advantage of the computational efficiency of two-dimensional BEV, it can also output three-dimensional occupancy information that meets the task requirements, thus balancing prediction accuracy and inference speed.

[0091] In some embodiments, S2532, based on the multiple independent dimensions obtained from the splitting, the two-dimensional output features are reorganized to obtain three-dimensional occupancy features, including: using the height dimension obtained from the splitting as the vertical voxel dimension of the three-dimensional occupancy space; using the occupancy category dimension obtained from the splitting as the category prediction dimension corresponding to the three-dimensional voxel position in the three-dimensional occupancy space; and performing tensor dimension rearrangement processing on the two-dimensional output features after dimension splitting based on the original horizontal space dimension, vertical voxel dimension and category prediction dimension of the two-dimensional output features to obtain three-dimensional occupancy features.

[0092] For example, the height dimension Z obtained after dimensionality splitting can be used as a voxel layer in the vertical direction of the 3D occupied space, and the occupied category dimension C_occ obtained after dimensionality splitting can be used as the category prediction dimension corresponding to each voxel position in the 3D occupied space. Furthermore, by concatenating these two dimensions with the X and Y plane dimensions natively carried by the 2D output features, a 3D occupied feature containing both complete 3D spatial dimensions and category prediction dimensions can be generated. It can be seen that this dimensional arrangement transformation is merely a tensor dimension recombination operation, requiring no additional learnable parameters and not performing convolution operations, thus maximizing model computational efficiency.

[0093] like Figure 7 As shown above, in the above Figure 2 Based on the illustrated embodiment, step S23 may include steps S231-S232.

[0094] S231. Perform feature expansion processing on the first image features and depth probability distribution of the first image to obtain the corresponding first expanded image features and depth expanded probability distribution.

[0095] For example, suppose the shape of the first image feature is [B, C_f, H_f, W_f], and the shape of the depth probability distribution is [B, N_z, H_f, W_f]. In order to elevate the two-dimensional first image feature into a three-dimensional view frustum, embodiments of this disclosure may expand the first image feature along the depth candidate dimension to expand the first image feature into a first expanded image feature, the shape of which is [B, C_f, 1, H_f, W_f]; and expand the depth probability distribution into a depth expanded probability distribution, the shape of which is [B, 1, N_z, H_f, W_f].

[0096] S232. The first extended image features are weighted and fused with the depth extended probability distribution to obtain the view frustum features corresponding to the first image.

[0097] For example, the first extended image features can be multiplied element-wise with the depth extended probability distribution and weighted and fused to obtain the view frustum features in step S232. The shape of the view frustum features is [B, C_f, N_z, H_f, W_f]. Thus, in this embodiment of the present disclosure, the first image features are not simply copied to all depth candidates, nor are they assigned to a fixed depth position. Instead, they are weighted and assigned according to the depth probability distribution obtained by binocular parallax conversion: depth candidate positions with higher probabilities receive stronger feature responses, and depth candidate positions with lower probabilities receive weaker feature responses.

[0098] Therefore, the view frustum feature in this embodiment is a Frustum feature constrained by binocular probability prior. The view frustum feature contains both image semantic information and binocular geometric probability information, thereby providing a more reliable spatial positioning basis for subsequent projection into the BEV space.

[0099] In some embodiments, S231, performing feature expansion processing on the first image features and depth probability distribution of the first image to obtain corresponding first expanded image features and depth expanded probability distribution, includes: determining a depth candidate dimension based on the depth probability distribution; and performing feature expansion processing on the first image features and depth probability distribution based on the depth candidate dimension to obtain corresponding first expanded image features and depth expanded probability distribution.

[0100] For example, the shape of the depth probability distribution is [B, N_z, H_f, W_f], where N_z is the depth candidate dimension. This depth probability distribution is based on the multi-depth candidate probability output obtained from binocular disparity estimation, where the depth candidate dimension corresponds to the predicted probability of multiple different depth intervals, and each depth candidate corresponds to a sampling position at a different distance in three-dimensional space. Based on this depth candidate dimension N_z, the tensor arrangement structure is determined, adding placeholder dimensions to the two-dimensional first image features and the depth probability distribution to complete dimension alignment and realize the feature expansion operation. This expansion operation is designed entirely around the depth layering requirements represented by N_z, providing a dimensional basis for subsequently constructing view frustum features containing depth information of the N_z layer. The placeholder dimension is a temporary dimension with a value of 1 added to achieve tensor broadcast multiplication and dimension matching; it is only used to complete the tensor shape and does not carry any valid data information such as image features or depth probabilities.

[0101] like Figure 8 As shown above, in the above Figure 2 Based on the illustrated embodiment, step S22 may include steps S221-S223.

[0102] S221. Perform feature extraction on the first image and the second image respectively to obtain the first image features corresponding to the first image and the second image features corresponding to the second image.

[0103] For example, embodiments of this disclosure can perform feature extraction on a first image to obtain first image features corresponding to the first image; simultaneously, feature extraction can be performed on a second image to obtain second image features corresponding to the second image. Feature extraction can be based on traditional algorithms, such as SIFT (Scale-invariant Feature Transform), SURF (Speed ​​Up Robust Feature), and HOG (Histogram of Oriented Gradient). It can also be based on deep learning algorithms, such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), and autoencoders. Furthermore, the feature extraction network used to extract image features can be a lightweight binocular feature extraction network.

[0104] In other words, the specific implementation method of feature extraction is not limited in the embodiments disclosed herein.

[0105] In some examples, the tensor shape of the first or second image is [B, 3, H, W]. Here, 3 represents the number of RGB image channels, and H and W represent the height and width of the image, respectively. Furthermore, the shape of the extracted first or second image features can be represented as [B, C_f, H_f, W_f]. Here, C_f represents the number of image feature channels, and H_f and W_f represent the dimensions of the downsampled feature map. The first and second image features determined in step S221 serve two main purposes: one is to be used together for subsequent binocular disparity probability estimation, and the other is to use the first image features as the source of image features when constructing subsequent view frustum features. Therefore, embodiments of this disclosure can use the matching relationship between binocular features to generate geometric probability priors, rather than relying solely on monocular images to estimate depth, thereby providing a more stable geometric basis for subsequent spatial projection.

[0106] In some embodiments, the first and second images can be preprocessed, including distortion correction, stereo correction, and resizing, to ensure that the binocular images (i.e., the first and second images) meet the geometric alignment requirements for subsequent binocular matching. The tensor shapes of the preprocessed first and second images remain [B, 3, H, W]. Distortion correction, stereo correction, and resizing operations can be accelerated by dedicated hardware modules to reduce the load on CPUs (Central Processing Units), GPUs (Graphics Processing Units), or other general-purpose computing units. Furthermore, feature extraction can be performed on the preprocessed first and second images to obtain first image features corresponding to the first image and second image features corresponding to the second image.

[0107] S222. Based on the features of the first image and the features of the second image, perform disparity matching on the first image and the second image to obtain the disparity probability distribution corresponding to the first image.

[0108] For example, in this step, a disparity probability distribution can be generated based on the binocular matching relationship. The binocular matching relationship refers to the mapping and matching association in which the same real three-dimensional space point appears one-to-one between the pixel coordinates of the left image and the pixel coordinates of the right image in two binocular images (i.e., the first image and the second image) captured simultaneously.

[0109] In some examples, the matching cost between the first image features and the second image features can be determined first under multiple preset disparities; then, the multiple matching costs are stacked to obtain a disparity cost volume. Furthermore, the disparity cost volume is normalized to obtain the disparity probability distribution corresponding to the first image. Here, the multiple preset disparities are manually pre-set model hyperparameters. The disparity matching in this embodiment refers to measuring the feature matching similarity at each pixel location within a preset disparity value range based on the image features of the first image and the image features of the second image, establishing a feature matching cost relationship, and obtaining a pixel-by-pixel disparity probability distribution based on the first image through cost normalization, thereby completing the stereo matching association between the two images and solving the pixel-level disparity information stereo matching process.

[0110] In this embodiment, instead of directly predicting a predetermined disparity value, the probability distribution corresponding to multiple preset disparities is predicted, i.e., a disparity probability distribution. This probability distribution can retain more uncertainty information, providing richer geometric constraints for subsequent depth probability calculation. The disparity probability distribution refers to the probability distribution of each pixel in the first image under different preset disparities. Each preset disparity corresponds to a probability value; a higher probability value indicates a greater likelihood that the true disparity corresponding to that pixel is the preset disparity. After obtaining the disparity probability distribution, it can be converted into the aforementioned depth probability distribution based on the inverse relationship between disparity and depth, thus completing the probabilistic estimation of depth information.

[0111] In some examples, the first and second image features can be used as inputs to estimate the probability distribution of each pixel in the first image across multiple preset disparities by constructing a binocular matching cost. For example, a disparity cost body can be constructed based on the matching relationship between the first and second image features. Assuming the number of preset disparities is D, the cost body can be represented as [B, D, H_f, W_f]. Furthermore, for each position (u, v) in the first image features, the cost body represents the matching cost between that position and the corresponding candidate position in the second image across D preset disparities. Here, the cost body is a four-dimensional fractional cube, which is generated by traversing all potential disparities in the second image for each pixel in the first image and calculating the matching score one by one. Finally, all matching scores together constitute the matching score set. Further, the disparity dimension can be normalized to obtain the disparity probability distribution corresponding to the first image, which can be represented as [B, D, H_f, W_f]. Where P_d(b, d, v, u) represents the probability that position (u, v) in the first image feature of the b-th sample corresponds to the d-th preset disparity.

[0112] S223. Determine the depth probability distribution based on the disparity probability distribution and the parameters of the image sensor.

[0113] For example, in this embodiment of the present disclosure, the mapping relationship between disparity and depth can be determined first based on the parameters of the image sensor (such as focal length, binocular baseline, and other calibration parameters); and then the depth probability distribution can be determined based on the disparity probability distribution and the mapping relationship between disparity and depth.

[0114] Thus, since the disparity probability distribution determined in this embodiment retains uncertainty information on different preset disparities, the disparity probability distribution can be converted into a depth probability distribution using camera geometry and used to construct the view frustum feature.

[0115] like Figure 9 As shown above, in the above Figure 8Based on the illustrated embodiment, step S222 may include steps S2221-S2223.

[0116] S2221. Under multiple preset parallaxes, determine the matching cost between the first image feature and the second image feature.

[0117] For example, for each pixel position within the first image feature, based on binocular stereo matching constraints, the matching coordinates corresponding to all preset disparities can be traversed on the second image feature to determine the feature similarity between the current position of the first image feature and the corresponding matching position of the second image feature, and this similarity can be converted into a matching cost under the current preset disparity. Furthermore, the matching costs corresponding to all preset disparities together constitute a four-dimensional disparity cost body.

[0118] S2222: Stack multiple matching costs to obtain the disparity cost volume.

[0119] For example, the matching costs obtained under each preset disparity can be stacked sequentially along the disparity dimension to generate a complete four-dimensional disparity cost volume. This four-dimensional disparity cost volume can store the matching cost data corresponding to each pixel position in the first image feature under all preset disparities, laying the foundation for subsequent normalization operations along the disparity dimension and outputting the disparity probability distribution.

[0120] S2223. Normalize the disparity cost volume to obtain the disparity probability distribution corresponding to the first image.

[0121] For example, a normalization operation can be performed on the disparity cost volume along the disparity dimension to convert the matching cost corresponding to each preset disparity into the corresponding disparity probability, thereby obtaining the disparity probability distribution. In this way, the probability information of different preset disparities can be preserved more completely, avoiding the problem of geometric information loss caused by directly predicting and determining the disparity, providing accurate and reliable input for the subsequent transformation of the depth probability distribution, and ensuring the robustness of the overall depth estimation.

[0122] In some embodiments, step S223, determining the depth probability distribution based on the disparity probability distribution and the parameters of the image sensor, includes: determining the mapping relationship between disparity and depth based on the parameters of the image sensor; and determining the depth probability distribution based on the disparity probability distribution and the mapping relationship.

[0123] For example, since the construction of subsequent frustum features requires unfolding image features in the depth direction, the disparity probability distribution is converted into a depth probability distribution in this embodiment. For a binocular image sensor (such as a binocular camera), there is a definite geometric relationship between disparity d and depth z: z = f·b / d. Here, f is the camera focal length, and b is the baseline length of the binocular camera. Furthermore, for each preset disparity d_i, its corresponding depth candidate z_i can be calculated based on the camera calibration parameters. And through this mapping relationship, the disparity probability distribution [B, D, H_f, W_f] is converted into a depth probability distribution [B, N_z, H_f, W_f]. Here, N_z is the number of depth candidates.

[0124] In some examples, if there is a one-to-one correspondence between the preset disparity and depth candidates, the probabilities can be directly rearranged based on the mapping relationship between disparity and depth. If the preset disparity and depth candidates use different discrete partitioning methods, the probabilities on the preset disparity can be assigned to the corresponding depth candidates by looking up a table, interpolation, or probability accumulation.

[0125] The main function of step S223 is to transform the disparity spatial probability obtained from binocular matching into a depth spatial probability suitable for 3D spatial projection. Compared to directly using monocular depth probability, the depth probability in this embodiment of the disclosure is derived from binocular disparity geometric constraints, thus having a more defined physical scale; compared to directly using a single depth value, the scheme in this embodiment of the disclosure can retain the uncertainty of depth candidates, which is beneficial for subsequent weighted expansion of features.

[0126] Figure 10 This is a flowchart illustrating a training method for an occupancy prediction model provided as an exemplary embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices, such as... Figure 10 As shown, the training method includes steps S101-S106.

[0127] S101. Determine the disparity depth estimation network and the initial occupancy network.

[0128] For example, the occupancy prediction model provided in this disclosure includes at least a disparity depth estimation network and an occupancy network.

[0129] The disparity depth estimation network can be an initial disparity depth estimation network, or, in this embodiment of the disclosure, the disparity depth estimation network can be pre-trained to obtain a pre-trained disparity depth estimation network. Then, the pre-trained disparity depth estimation network and the initial occupancy network are trained together to obtain a trained occupancy prediction model.

[0130] Thus, by pre-training the disparity depth estimation network and reusing its modules and weights, high-quality initial parameters can be provided for the occupancy prediction model, quickly endowing the model with stereo geometric perception capabilities and shortening the overall training convergence cycle. At the same time, reusing mature binocular matching priors can output a more accurate disparity probability distribution, providing reliable geometric constraints for the construction of Frustum frustum features, thereby improving the accuracy of occupancy prediction.

[0131] S102. Determine at least one set of sample images and label information; wherein, the set of sample images includes a third image and a fourth image acquired by an image sensor at the same time.

[0132] For example, images can be acquired based on multiple binocular image sensors set on a smart device to determine at least one set of sample images, or at least one set of sample images can be obtained from a general training sample database. The method of determining at least one set of sample images is not limited in this embodiment of the disclosure.

[0133] In some examples, at least one set of sample images has corresponding label information, which is the annotation result of the actual occupancy state. This label information is used to calculate the prediction loss during model training and to constrain the update direction of the model parameters. In one feasible design, this label information can be used to indicate whether each voxel corresponding to the target scene is occupied; in another feasible design, this label information can not only be used to indicate whether each voxel is occupied, but also to indicate the category of the object occupying that voxel when it is occupied.

[0134] The tag information can be obtained by collecting point cloud data of the target scene through high-precision 3D reconstruction equipment and then discretizing the occupancy probability, or it can be obtained by converting existing 3D scene semantic annotation data. This disclosure does not limit the specific method of obtaining the tag information.

[0135] In some examples, embodiments of this disclosure include at least one set of sample images, each set of sample images including a third image and a fourth image acquired at the same time; wherein the third image and the fourth image are binocular images.

[0136] S103. Use a disparity depth estimation network to process the third and fourth images to obtain the initial depth probability distribution corresponding to the third image.

[0137] It is understandable that the specific implementation of using the disparity depth estimation network to process the third and fourth images to obtain the initial depth probability distribution corresponding to the third image can refer to the implementation of step S22 mentioned above, and will not be repeated here.

[0138] S104. The initial occupancy network is used to process the third image and the initial depth probability distribution to obtain the initial view frustum features corresponding to the third image. The initial view frustum features are then subjected to projection aggregation to obtain two-dimensional bird's-eye view features. Based on the two-dimensional bird's-eye view features, the initial occupancy information is determined.

[0139] It is understandable that the specific implementation of using the initial occupancy network to process the third image and the initial depth probability distribution to obtain the initial view frustum features corresponding to the third image can be referred to the implementation of step S23 mentioned above, and will not be repeated here.

[0140] It is understandable that the specific implementation method of projecting and aggregating the initial view frustum features to obtain the two-dimensional bird's-eye view features can refer to the implementation method of the aforementioned step S24, and will not be repeated here.

[0141] It is understandable that the specific implementation method for determining the initial occupancy information based on the features of the two-dimensional bird's-eye view can refer to the implementation method of the aforementioned step S25, and will not be repeated here.

[0142] S105. Based on the initial occupancy information and label information, iteratively train the disparity depth estimation network and the initial occupancy network to obtain the trained disparity depth estimation network and the trained occupancy network.

[0143] For example, a loss value can be determined based on initial occupancy and label information. This loss value is then used to update the network parameters in the disparity depth estimation network (the initial disparity depth estimation network or a pre-trained disparity depth estimation network) and the network parameters in the initial occupancy network until the updated network satisfies the convergence condition. If the updated network satisfies the convergence condition, the trained disparity depth estimation network and the trained occupancy network are obtained.

[0144] The updated network satisfying the convergence condition can include: the updated occupancy network satisfying the convergence condition; or, both the updated disparity depth estimation network and the updated occupancy network satisfying the convergence condition.

[0145] In some examples, the updated network satisfies the convergence condition, which can be achieved in three ways: first, the loss of the network output is less than a certain preset value; second, the change in weights between two iterations is less than a certain preset value; and third, the number of iterations reaches a preset number.

[0146] S106. Based on the trained disparity depth estimation network and the trained occupancy network, the trained occupancy prediction model is obtained.

[0147] For example, the occupancy prediction model in this embodiment includes a disparity depth estimation network and an occupancy network. Therefore, a trained occupancy prediction model can be obtained based on the trained disparity depth estimation network and the trained occupancy network.

[0148] In other words, during the training phase of the occupancy prediction model, the embodiments of this disclosure can reuse the network structure or key modules and weights of the pre-trained disparity depth estimation network. At the start of training, the weights of the pre-trained disparity depth estimation network can be loaded, thereby giving the occupancy prediction model better initial geometric perception capabilities. Then, based on the 3D occupancy supervision signal (i.e., label information), the pre-trained disparity depth estimation network and the initial occupancy network are jointly trained end-to-end. Through joint training, the disparity depth estimation network can not only retain its original binocular matching capability but also adjust its output disparity probability distribution according to the occupancy prediction task, making this probability distribution more suitable for subsequent construction of view frustum features and prediction of 3D spatial occupancy.

[0149] That is, the embodiments of this disclosure do not use an independent binocular depth estimation result as post-processing input, but use the disparity probability estimation process as the geometric prior generation link in the occupancy prediction model, and jointly optimize it with subsequent view transformation and decoding under the supervision of the final 3D occupancy. This can keep the binocular geometric prior consistent with the 3D occupancy task objective.

[0150] It should be noted that the occupancy prediction model (including the disparity depth estimation network and the occupancy network) used in the occupancy information determination method in this disclosure embodiment can be obtained based on any embodiment corresponding to the above-mentioned occupancy prediction model training method.

[0151] Exemplary device Figure 11 An apparatus for determining occupancy information provided in an embodiment of this disclosure, such as Figure 11 As shown, the occupancy information determination device 110 includes an image determination module 111, a depth determination module 112, a view frustum feature determination module 113, a bird's-eye view feature determination module 114, and an occupancy information determination module 115.

[0152] Image determination module 111 is used to determine the first image and the second image of the target scene acquired by the image sensor at the same time. The depth determination module 112 is used to determine the depth probability distribution corresponding to the first image based on the first image and the second image; The frustum feature determination module 113 is used to determine the frustum features corresponding to the first image based on the first image and the depth probability distribution; The bird's-eye view feature determination module 114 is used to perform projection aggregation processing on the view frustum features to obtain the two-dimensional bird's-eye view features corresponding to the view frustum features; The occupancy information determination module 115 is used to determine the occupancy information of the target scene based on the features of the two-dimensional bird's-eye view.

[0153] In some embodiments, such as Figure 12 As shown, in Figure 11 Based on this, the bird's-eye view feature determination module 114 includes a parameter determination unit 1141, a coordinate transformation unit 1142, a feature projection unit 1143, and a feature aggregation unit 1144.

[0154] The parameter determination unit 1141 is used to determine preset spatial configuration parameters; The coordinate transformation unit 1142 is used to perform coordinate transformation processing on the view frustum features to obtain the view frustum features in the carrier coordinate system. The feature projection unit 1143 is used to project the view frustum feature in the carrier coordinate system onto the grid position corresponding to the bird's-eye view feature plane based on the spatial configuration parameters, so as to obtain the projected bird's-eye view grid feature. Feature aggregation unit 1144 is used to perform feature aggregation processing on the bird's-eye view grid features to obtain two-dimensional bird's-eye view features.

[0155] In some embodiments, such as Figure 13 As shown, in Figure 11 Based on this, the occupation information determination module 115 includes a feature encoding unit 1151, a channel mapping unit 1152, and a dimension rearrangement unit 1153.

[0156] Feature encoding unit 1151 is used to perform feature encoding processing on the features of the two-dimensional bird's-eye view to obtain the encoded two-dimensional features; The channel mapping unit 1152 is used to perform channel mapping processing on the encoded two-dimensional features to obtain two-dimensional output features; The dimension rearrangement unit 1153 is used to perform dimension rearrangement processing on the two-dimensional output features to obtain the occupancy information of the target scene.

[0157] In some embodiments, the channel mapping unit 1152 is specifically used to determine the target number of channels; wherein, the target number of channels is determined by the occupancy category dimension and the height dimension, the height dimension is the dimension obtained by dividing the three-dimensional space where the target scene is located along the vertical direction, and the occupancy category dimension is the dimension used to characterize the category corresponding to the voxel in the three-dimensional space; the encoded two-dimensional features are processed by channel mapping to map the original number of channels of the two-dimensional features to the target number of channels, thereby obtaining two-dimensional output features.

[0158] In some embodiments, the dimension rearrangement unit 1153 is specifically used to perform dimension splitting processing on the two-dimensional output features along the channel dimension to obtain multiple independent dimensions including at least the height dimension and the occupancy category dimension; to perform dimension reorganization on the two-dimensional output features based on the multiple independent dimensions obtained by splitting to obtain three-dimensional occupancy features; and to determine the occupancy information of the target scene based on the three-dimensional occupancy features.

[0159] In some embodiments, the dimension rearrangement unit 1153 is specifically used to perform dimension splitting processing on the two-dimensional output features along the channel dimension to obtain multiple independent dimensions including at least a height dimension and an occupancy category dimension; to use the split height dimension as the vertical voxel dimension of the three-dimensional occupancy space; to use the split occupancy category dimension as the category prediction dimension corresponding to the three-dimensional voxel position in the three-dimensional occupancy space; to perform tensor dimension rearrangement processing on the dimension-split two-dimensional output features based on the original horizontal space dimension, vertical voxel dimension and category prediction dimension of the two-dimensional output features to obtain three-dimensional occupancy features; and to determine the occupancy information of the target scene based on the three-dimensional occupancy features.

[0160] In some embodiments, the frustum feature determination module 113 is specifically used to perform feature expansion processing on the first image features and depth probability distribution of the first image to obtain corresponding first expanded image features and depth expanded probability distribution; and to perform weighted fusion of the first expanded image features and depth expanded probability distribution to obtain the frustum features corresponding to the first image.

[0161] In some embodiments, the frustum feature determination module 113 is specifically used to determine a depth candidate dimension based on a depth probability distribution; perform feature expansion processing on the first image features and the depth probability distribution based on the depth candidate dimension to obtain corresponding first expanded image features and depth expanded probability distribution; and perform weighted fusion of the first expanded image features and the depth expanded probability distribution to obtain the frustum features corresponding to the first image.

[0162] In some embodiments, such as Figure 14 As shown, in Figure 11 Based on this, the depth determination module 112 includes a feature extraction unit 1121, a disparity matching unit 1122, and a depth determination unit 1123.

[0163] The feature extraction unit 1121 is used to extract features from the first image and the second image respectively, to obtain the first image features corresponding to the first image and the second image features corresponding to the second image. The disparity matching unit 1122 is used to perform disparity matching on the first image and the second image based on the first image features and the second image features to obtain the disparity probability distribution corresponding to the first image; The depth determination unit 1123 is used to determine the depth probability distribution based on the disparity probability distribution and the parameters of the image sensor.

[0164] In some embodiments, the disparity matching unit 1122 is specifically used to determine the matching cost between the first image feature and the second image feature under multiple preset disparities; stack the multiple matching costs to obtain a disparity cost body; and normalize the disparity cost body to obtain the disparity probability distribution corresponding to the first image.

[0165] In some embodiments, the depth determination unit 1123 is specifically used to determine the mapping relationship between disparity and depth based on the parameters of the image sensor; and to determine the depth probability distribution based on the disparity probability distribution and the mapping relationship.

[0166] The beneficial technical effects of the above-described embodiment of the occupancy information determination device 110 can be found in the corresponding beneficial technical effects of the above-described exemplary method for determining occupancy information, and will not be repeated here.

[0167] Figure 15 A training apparatus for an occupancy prediction model provided in this disclosure embodiment, such as Figure 15 As shown, the training device 150 includes a network determination module 151, a sample determination module 152, a disparity depth estimation module 153, an occupancy prediction module 154, a training module 155, and a processing module 156.

[0168] Network determination module 151 is used to determine the disparity depth estimation network and the initial occupancy network; The sample determination module 152 is used to determine at least one set of sample images and label information; wherein, the set of sample images includes a third image and a fourth image acquired by an image sensor at the same time. The disparity depth estimation module 153 is used to process the third image and the fourth image using the disparity depth estimation network to obtain the initial depth probability distribution corresponding to the third image. Occupation prediction module 154 is used to process the third image and the initial depth probability distribution using the initial occupancy network to obtain the initial view frustum features corresponding to the third image, and to perform projection aggregation processing on the initial view frustum features to obtain two-dimensional bird's-eye view features; based on the two-dimensional bird's-eye view features, the initial occupancy information is determined. Training module 155 is used to iteratively train the disparity depth estimation network and the initial occupancy network based on the initial occupancy information and label information, so as to obtain the trained disparity depth estimation network and the trained occupancy network. Processing module 156 is used to obtain a trained occupancy prediction model based on the trained disparity depth estimation network and the trained occupancy network.

[0169] The beneficial technical effects of the above-described exemplary embodiment of the occupancy prediction model training device 150 can be found in the corresponding beneficial technical effects of the occupancy prediction model training method in the above-described exemplary method section, and will not be repeated here.

[0170] Exemplary electronic devices Figure 16 A structural diagram of an electronic device 160 provided in this disclosure embodiment includes at least one processor 161 and a memory 162.

[0171] The processor 161 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 160 to perform desired functions.

[0172] The memory 162 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 161 may execute one or more computer program instructions to implement the methods for determining occupancy information and / or training occupancy prediction models and / or other desired functions of the various embodiments of this disclosure described above.

[0173] In one example, the electronic device 160 may also include an input device 163 and an output device 164, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0174] The input device 163 may also include, for example, a keyboard, a mouse, etc.

[0175] The output device 164 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0176] Of course, for the sake of simplicity, Figure 16 Only some of the components of the electronic device 160 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 160 may include any other suitable components depending on the specific application.

[0177] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform steps in the method for determining occupancy information and / or steps in the method for training an occupancy prediction model, as described in the "Exemplary Methods" section of this disclosure.

[0178] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0179] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform steps in the method for determining occupancy information and / or steps in the method for training an occupancy prediction model, as described in the "Exemplary Methods" section above.

[0180] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0181] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0182] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for determining occupancy information, comprising: Determine the first and second images of the target scene acquired by the image sensors at the same time; Based on the first image and the second image, determine the depth probability distribution corresponding to the first image; Based on the first image and the depth probability distribution, determine the view frustum features corresponding to the first image; The view frustum feature is subjected to projection aggregation processing to obtain the two-dimensional bird's-eye view feature corresponding to the view frustum feature; Based on the features of the two-dimensional bird's-eye view, the occupancy information of the target scene is determined.

2. The method according to claim 1, wherein, The projection aggregation process performed on the frustum features to obtain the corresponding two-dimensional bird's-eye view features includes: Determine the preset space configuration parameters; The view frustum feature is subjected to coordinate transformation to obtain the view frustum feature in the carrier coordinate system; Based on the spatial configuration parameters, the view frustum feature in the carrier coordinate system is projected onto the grid position corresponding to the bird's-eye view feature plane to obtain the projected bird's-eye view grid feature. The bird's-eye view grid features are subjected to feature aggregation processing to obtain the two-dimensional bird's-eye view features.

3. The method according to claim 1, wherein, The determination of the occupancy information of the target scene based on the features of the two-dimensional bird's-eye view includes: The two-dimensional bird's-eye view features are subjected to feature encoding processing to obtain encoded two-dimensional features; The encoded two-dimensional features are subjected to channel mapping processing to obtain two-dimensional output features; The dimensional rearrangement of the two-dimensional output features is performed to obtain the occupancy information of the target scene.

4. The method according to claim 3, wherein, The process of performing channel mapping on the encoded two-dimensional features to obtain two-dimensional output features includes: Determine the target number of channels; wherein the target number of channels is determined by the category dimension and the height dimension, the height dimension is the dimension obtained by dividing the three-dimensional space where the target scene is located along the vertical direction, and the category dimension is the dimension used to characterize the category corresponding to the voxels in the three-dimensional space; The encoded two-dimensional features are subjected to channel mapping processing to map the original number of channels of the two-dimensional features to the target number of channels, thereby obtaining the two-dimensional output features.

5. The method according to claim 3, wherein, The step of performing dimensional rearrangement processing on the two-dimensional output features to obtain the occupancy information of the target scene includes: The two-dimensional output feature is split along the channel dimension to obtain multiple independent dimensions, including at least the height dimension and the category dimension. Based on the multiple independent dimensions obtained from the splitting, the two-dimensional output feature is reorganized to obtain the three-dimensional occupancy feature; Based on the three-dimensional occupancy features, the occupancy information of the target scene is determined.

6. The method according to claim 5, wherein, The step of reorganizing the two-dimensional output features based on the multiple independent dimensions obtained from the splitting to obtain three-dimensional occupancy features includes: The height dimension obtained from the split is used as the vertical voxel dimension of the three-dimensional occupied space; The occupancy category dimension obtained from the split is used as the category prediction dimension corresponding to the three-dimensional voxel position in the three-dimensional occupancy space; Based on the original horizontal spatial dimension of the two-dimensional output feature, as well as the vertical voxel dimension and the category prediction dimension, the two-dimensional output feature after dimension splitting is subjected to tensor dimension rearrangement processing to obtain the three-dimensional occupancy feature.

7. The method according to claim 1, wherein, The step of determining the frustum features corresponding to the first image based on the first image and the depth probability distribution includes: The first image features and the depth probability distribution of the first image are respectively subjected to feature expansion processing to obtain the corresponding first expanded image features and depth expanded probability distribution; The first extended image features are weighted and fused with the depth extended probability distribution to obtain the view frustum features corresponding to the first image.

8. The method according to claim 7, wherein, The step of performing feature expansion processing on the first image features and the depth probability distribution of the first image to obtain corresponding first expanded image features and depth expanded probability distribution includes: Based on the depth probability distribution, candidate depth dimensions are determined; Based on the depth candidate dimension, feature expansion processing is performed on the first image feature and the depth probability distribution to obtain the corresponding first expanded image feature and depth expanded probability distribution.

9. The method according to any one of claims 1-8, wherein, The step of determining the depth probability distribution corresponding to the first image based on the first image and the second image includes: Feature extraction is performed on the first image and the second image respectively to obtain the first image feature corresponding to the first image and the second image feature corresponding to the second image; Based on the first image features and the second image features, disparity matching is performed on the first image and the second image to obtain the disparity probability distribution corresponding to the first image; The depth probability distribution is determined based on the disparity probability distribution and the parameters of the image sensor.

10. The method according to claim 9, wherein, The step of performing disparity matching on the first image and the second image based on the first image features and the second image features to obtain the disparity probability distribution corresponding to the first image includes: Under multiple preset parallaxes, determine the matching cost between the first image feature and the second image feature; The multiple matching costs are stacked to obtain a disparity cost volume; The disparity cost volume is normalized to obtain the disparity probability distribution corresponding to the first image.

11. The method according to claim 9, wherein, Determining the depth probability distribution based on the disparity probability distribution and the parameters of the image sensor includes: Based on the parameters of the image sensor, the mapping relationship between parallax and depth is determined; The depth probability distribution is determined based on the disparity probability distribution and the mapping relationship.

12. A method for training an occupation prediction model, comprising: Determine the disparity depth estimation network and the initial occupancy network; Determine at least one set of sample images and label information; wherein, the set of sample images includes a third image and a fourth image acquired by an image sensor at the same time; The disparity depth estimation network is used to process the third image and the fourth image to obtain the initial depth probability distribution corresponding to the third image; The initial occupancy network is used to process the third image and the initial depth probability distribution to obtain the initial frustum features corresponding to the third image. The initial frustum features are then subjected to projection aggregation to obtain two-dimensional bird's-eye view features. Based on the two-dimensional bird's-eye view features, the initial occupancy information is determined. Based on the initial occupancy information and the label information, the disparity depth estimation network and the initial occupancy network are iteratively trained to obtain the trained disparity depth estimation network and the trained occupancy network. Based on the trained disparity depth estimation network and the trained occupancy network, a trained occupancy prediction model is obtained.

13. A device for determining occupancy information, comprising: An image determination module is used to determine the first and second images of the target scene acquired by the image sensor at the same time. A depth determination module is used to determine the depth probability distribution corresponding to the first image based on the first image and the second image; A frustum feature determination module is used to determine the frustum features corresponding to the first image based on the first image and the depth probability distribution; The bird's-eye view feature determination module is used to perform projection aggregation processing on the view frustum features to obtain the two-dimensional bird's-eye view features corresponding to the view frustum features; The occupancy information determination module is used to determine the occupancy information of the target scene based on the features of the two-dimensional bird's-eye view.

14. A training device for a prediction model, comprising: The network determination module is used to determine the disparity depth estimation network and the initial occupancy network; A sample determination module is used to determine at least one set of sample images and label information; wherein, the set of sample images includes a third image and a fourth image acquired by an image sensor at the same time; The disparity depth estimation module is used to process the third image and the fourth image using the disparity depth estimation network to obtain the initial depth probability distribution corresponding to the third image; The occupancy prediction module is used to process the third image and the initial depth probability distribution using the initial occupancy network to obtain the initial view frustum features corresponding to the third image, and to perform projection aggregation processing on the initial view frustum features to obtain two-dimensional bird's-eye view features; based on the two-dimensional bird's-eye view features, the initial occupancy information is determined. The training module is used to iteratively train the disparity depth estimation network and the initial occupancy network based on the initial occupancy information and the label information, so as to obtain the trained disparity depth estimation network and the trained occupancy network. The processing module is used to obtain a trained occupancy prediction model based on the trained disparity depth estimation network and the trained occupancy network.

15. A computer-readable storage medium storing a computer program for performing the method for determining occupancy information according to any one of claims 1-11, or the method for training an occupancy prediction model according to claim 12.

16. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for determining occupancy information as described in any one of claims 1-11, or the method for training the occupancy prediction model as described in claim 12.

Images