Shielding area three-dimensional voxel reasoning method and system

Abstract

The invention belongs to the technical field of three-dimensional mapping, and particularly relates to an occlusion area three-dimensional voxel reasoning method and system. The method comprises the following steps: S10, acquiring an RGB image containing a semantic segmentation result of an occlusion region and a corresponding depth map, and converting the depth map into an initial sparse three-dimensional voxel; and S20, constructing a three-dimensional voxel inference network, and inferring a three-dimensional voxel occupancy probability graph from the initial sparse three-dimensional voxels through the three-dimensional voxel inference network. The occlusion area can be reconstructed based on the monocular image; multi-format output and navigation interface butt joint are supported, and the method can be integrated to SLAM, 3D mapping or path search systems to serve as spatial feasibility constraints; the method can be used for querying whether any spatial position is passable; the method can be used for extracting a trafficability sub-graph from a designated area for path planning.

Description

Technical Field

[0001] This application belongs to the field of 3D mapping technology, and specifically relates to a method and system for 3D voxel inference in occluded areas. Background Technology

[0002] In traditional intelligent transportation and mobile robot visual perception, occlusion structures pose a significant challenge to the identification of passable areas. Because the ground information in these areas cannot be directly observed through ordinary images, passable area identification systems based solely on semantic segmentation and 2D depth estimation are prone to incorrectly classifying areas below occlusions as impassable.

[0003] Existing methods often rely on multi-view structures or lidar equipment, which increases hardware costs and deployment complexity.

[0004] Therefore, there is an urgent need for a technical solution to overcome or mitigate at least one of the aforementioned defects in the existing technology. Summary of the Invention

[0005] The purpose of this application is to provide a three-dimensional voxel inference method and system for occluded regions, so as to solve the problems of high hardware cost and deployment complexity in the existing technology.

[0006] The technical solution of this application is:

[0007] The first aspect of this application provides a method for three-dimensional voxel inference of occluded regions, including:

[0008] S10. Obtain an RGB image containing the semantic segmentation results of the occluded region and its corresponding depth map, and convert the depth map into an initial sparse three-dimensional voxel;

[0009] S20. Construct a three-dimensional voxel inference network, and infer a three-dimensional voxel occupancy probability map from the initial sparse three-dimensional voxels through the three-dimensional voxel inference network.

[0010] In at least one embodiment of this application, in S10, obtaining an RGB image containing semantic segmentation results of the occluded region and its corresponding depth map, and converting the depth map into an initial sparse three-dimensional voxel, includes:

[0011] S11. Obtain the RGB image containing the semantic segmentation results of the occluded region and its corresponding depth map;

[0012] S12. Extract the occluded region from the depth map based on the semantic segmentation result of the occluded region;

[0013] S13. Convert the depth map into an initial sparse three-dimensional voxel:

[0014]

[0015] Where (x,y,z) are the initial sparse 3D voxel coordinates, (u,v) are the depth map coordinates, and f u f is the horizontal focal length, an intrinsic parameter of the camera. v The vertical focal length is the camera's intrinsic parameter, (c u ,c v () represents the principal point coordinates of the camera's intrinsic parameters;

[0016] S14. Convert the initial sparse three-dimensional voxels into a binary voxel mesh diagram, wherein the voxel mesh V of the binary voxel mesh diagram is... t ∈{c0,c1} H×W×Z Where c0 represents an empty voxel, c1 represents an occupied voxel, H represents the length of the voxel grid, W represents the width of the voxel grid, and Z represents the height of the voxel grid.

[0017] In at least one embodiment of this application, in step S20, a three-dimensional voxel inference network is constructed, and a three-dimensional voxel occupancy probability map is inferred from the initial sparse three-dimensional voxels through the three-dimensional voxel inference network, including:

[0018] S21. Extract the cavity contour within the occluded area and generate a geometric prior;

[0019] S22. Construct a three-dimensional voxel reasoning network and integrate geometric priors;

[0020] S23. Construct the joint loss function;

[0021] S24. Construct the dataset;

[0022] S25. Train and optimize the three-dimensional voxel inference network based on the joint loss function and the dataset;

[0023] S26. The three-dimensional voxel occupancy probability map is inferred from the initial sparse three-dimensional voxels through the three-dimensional voxel inference network.

[0024] In at least one embodiment of this application, in step S21, extracting the cavity contour within the occluded area and generating a geometric prior includes:

[0025] From the initial sparse three-dimensional voxels, obtain unit-by-unit voxel slices along the X-axis. Each slice includes voxel S on the YZ plane. x (y,z), where, y∈[0,Y-1], z∈[0,Z-1];

[0026] For each slice, scan along the y-column from bottom to top to find the voxel with the smallest z-value. min (x, y), where each column y corresponds to a voxel z. minThe curve bounded by (x, y) and the lines X = x, Z = 0 is taken as the contour curve of the slice, where the set of points on the contour curve of the slice is P(x) = {(y, z)}. min (x,y))};

[0027] Based on the point set on the contour curve of each slice, the cavity contour curve in the parameterized polar coordinate system is obtained by Fourier series as a geometric prior.

[0028] The cavity profile curve is mapped onto voxel space to obtain the geometric prior voxel.

[0029] In at least one embodiment of this application, in step S22, a three-dimensional voxel inference network is constructed and geometric priors are fused, including:

[0030] An encoder, decoder, and a secondary voxel occupancy probability prediction module are constructed for a 3D voxel inference network.

[0031] The encoder is used to convert the height dimension into the feature dimension and output three-dimensional features;

[0032] The decoder is used to restore spatial resolution and to fit the cavity profile curve using a Fourier series.

[0033] The secondary voxel occupancy probability prediction module is used to predict the secondary voxel occupancy probability of voxels around the cavity contour curve, wherein the secondary occupancy probability of each voxel is:

[0034]

[0035] Among them, P occ (x,y,z) represents the second-order occupancy probability of voxel (x,y,z), Bernoulli(p) is the prediction of random perturbation in the p-Beta(α,β) distribution, where p-Beta(α,β) is the random perturbation probability. The voxel position (y, z) is the 8-neighborhood around the cavity profile curve.

[0036] In at least one embodiment of this application, in step S23, constructing the joint loss function includes:

[0037] S231. Construct the cavity edge voxel constraint loss function:

[0038]

[0039] in, The loss is the voxel constraint loss at the cavity edge, γ is the robustness threshold, and L is the number of slices. For each slice x, the Fourier coefficients of the current contour curve are obtained through a contour fitting process. Here, δ represents the true coefficients of the corresponding contours in the training data, and δ represents the contour fitting residuals.

[0040] S232. Constructing the voxel reconstruction loss function:

[0041]

[0042] in, For voxel reconstruction loss, y i Let be the predicted occupancy probability of the i-th voxel. is the actual occupancy label of the i-th voxel, where 0 indicates that the voxel is empty, 1 indicates that the voxel is occupied, and N is the total number of voxels in the voxel grid;

[0043] S233. Construct the joint loss function:

[0044]

[0045] in, For joint losses.

[0046] In at least one embodiment of this application, S24, constructing a dataset, includes:

[0047] S241. Obtain actual collected data, including:

[0048] Acquire RGB images of scenes with different occlusions;

[0049] A dense point cloud model is generated using a multi-view stereo reconstruction method to obtain the three-dimensional spatial information of the target structure.

[0050] The dense point cloud model is projected from a top-down perspective and rendered to obtain a depth map of a set size;

[0051] Based on the depth map and camera intrinsic parameters, sparse three-dimensional voxels are generated and assigned occupancy labels, which are then used as training input samples.

[0052] S242. Obtain simulation synthesis data, including:

[0053] Obtain the occlusion model;

[0054] Geometric transformations are performed on the occlusion model to obtain various spatial occlusion forms;

[0055] The occlusion models are randomly arranged and combined to generate different occlusion scenes;

[0056] Render RGB images and depth maps from a virtual overhead view;

[0057] Sparse 3D voxels are generated based on the depth map and the geometric contours of the occluded objects and assigned occupancy labels as training supervision samples.

[0058] S243. Construct a dataset based on the actual collected data and the simulated synthetic data. A second aspect of this application provides a three-dimensional voxel inference system for occluded regions, comprising:

[0059] The initial voxel generation module is used to acquire an RGB image containing semantic segmentation results of occluded regions and its corresponding depth map, and convert the depth map into initial sparse three-dimensional voxels.

[0060] A three-dimensional voxel inference network construction module is used to construct a three-dimensional voxel inference network and infer a three-dimensional voxel occupancy probability map from the initial sparse three-dimensional voxels through the three-dimensional voxel inference network.

[0061] A third aspect of this application provides a computer-readable medium storing computer-executable instructions for performing the occlusion region three-dimensional voxel inference method as described above.

[0062] A fourth aspect of this application provides a computing device, comprising:

[0063] At least one processor, and a memory communicatively connected to said at least one processor; wherein,

[0064] The memory stores instructions that can be executed by the at least one processor, the instructions being used to perform the occlusion region three-dimensional voxel inference method as described above.

[0065] The invention has at least the following beneficial technical effects:

[0066] This application's 3D voxel inference method for occluded regions takes a single image and its depth map as input, integrates structural contour fitting of the occluded region with explicit constraints on the passable space, and utilizes a cavity structure modeling approach to perform refined modeling and occupancy probability prediction of voxels below the occluded region, ultimately generating a reliable 3D voxel occupancy probability map. This application can perform 3D structure reconstruction and passability assessment using monocular images in occluded environments. Attached Figure Description

[0067] Figure 1 This is a flowchart of a three-dimensional voxel inference method for an occluded region according to one embodiment of this application;

[0068] Figure 2 This is a schematic diagram of converting a 2D depth map into sparse three-dimensional voxels according to one embodiment of this application;

[0069] Figure 3 This is a schematic diagram of the cavity contour curve construction according to one embodiment of this application;

[0070] Figure 4This is a schematic diagram of a three-dimensional voxel inference system for an occluded area according to one embodiment of this application;

[0071] Figure 5 This is a schematic diagram of the hardware structure of a computing device for implementing a three-dimensional voxel inference method for occluded regions according to one embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0073] The following is in conjunction with the appendix Figures 1 to 5 This application will be described in further detail.

[0074] The first aspect of this application provides a three-dimensional voxel reasoning method for occluded areas, used for the reconstruction and accessibility reasoning of passable spaces under occluded areas (such as bridges and tree canopies). It can be widely applied to three-dimensional perception systems in complex environments such as urban autonomous driving, low-altitude UAV navigation, and emergency disaster reconnaissance.

[0075] The occlusion region three-dimensional voxel inference method of this application includes the following steps:

[0076] S10. Obtain the RGB image containing the semantic segmentation results of the occluded region and its corresponding depth map, and convert the depth map into an initial sparse three-dimensional voxel.

[0077] S20. Construct a three-dimensional voxel inference network and infer the three-dimensional voxel occupancy probability map from the initial sparse three-dimensional voxels through the three-dimensional voxel inference network.

[0078] Specifically, such as Figure 1 As shown, in S10, the initial sparse three-dimensional voxel generation process includes:

[0079] S11. Obtain the RGB image I containing the semantic segmentation results of the occluded region. t and its corresponding depth map D t ;

[0080] S12. Extract the occluded regions from the depth map based on the semantic segmentation results of the occluded regions;

[0081] S13, such as Figure 2 As shown, the depth map is converted into an initial sparse 3D voxel:

[0082]

[0083] Where (x,y,z) are the initial sparse 3D voxel coordinates, (u,v) are the depth map coordinates, and f u f is the horizontal focal length, an intrinsic parameter of the camera. v The vertical focal length is the camera's intrinsic parameter, (c u ,c v () represents the principal point coordinates of the camera's intrinsic parameters;

[0084] S14. Convert the initial sparse 3D voxels into a binary voxel mesh. The voxel mesh of the binary voxel mesh is V. t ∈{c0,c1} H×W×Z Where c0 represents an empty voxel, c1 represents an occupied voxel, H represents the length of the voxel grid, W represents the width of the voxel grid, and Z represents the height of the voxel grid.

[0085] In a binary voxel grid, each voxel is marked as 1 if it is occupied by at least one point. Each voxel is either empty or occupied.

[0086] The occlusion region three-dimensional voxel inference method of this application, in S20, involves constructing a three-dimensional voxel inference network and inferring a three-dimensional voxel occupancy probability map from an initial sparse three-dimensional voxels through the three-dimensional voxel inference network, including the following process:

[0087] S21. Extract the cavity contour within the occluded area and generate a geometric prior.

[0088] In this embodiment, in step S21, the cavity contour curve is fitted and a geometric prior is generated to construct the spatial prior geometry under the occlusion structure, guiding the voxel network optimization. Specifically, this includes the following steps:

[0089] Obtain unit-voxel slices along the X-axis from the initial sparse 3D voxels. Each slice includes voxel S in the YZ plane. x (y,z), where, y∈[0,Y-1], z∈[0,Z-1];

[0090] For each slice, scan along the y-column from bottom to top to find the voxel with the smallest z-value. min (x, y), where each column y corresponds to a voxel z. min The curve bounded by (x, y) and the lines X = x, Z = 0 is taken as the contour curve of the slice, where the set of points on the contour curve of the slice is P(x) = {(y, z)}. min(x,y))};

[0091] Based on the point set on the contour curve of each slice, the cavity contour curve in the parameterized polar coordinate system is obtained by Fourier series as a geometric prior.

[0092] The cavity profile curve is mapped onto voxel space to obtain the geometric prior voxel.

[0093] In this embodiment, as Figure 3 As shown, a closed curve of the cavity cross-section is constructed and moved longitudinally along the cavity (uniformly along the X-axis). Unit-by-unit voxel slices are then performed along the X-axis, with each slice consisting of voxels on the corresponding YZ plane, and x is fixed. The contour curves of each slice are obtained according to preset rules, and the point set on the contour curves of each slice is used as input for subsequent cavity contour curve fitting.

[0094] The closed-circuit function in the parameterized polar coordinate system is obtained by using Fourier series, and a continuous closed-circuit function is generated as a structural prior. Its mathematical form is as follows:

[0095]

[0096] Where θ is the polar angle, a0 is the average radius, and {a n ,b n} represents the harmonic coefficients at each level, and the K model complexity control coefficient.

[0097] To start from discrete contour points To find the optimal Fourier coefficients, the Cartesian coordinates need to be converted to polar coordinates (r). i ,θ i And construct the objective function by least squares fitting:

[0098]

[0099] The predicted value is:

[0100]

[0101] Constructing a design matrix (including basis function term cos(nθ) i ) and sin(nθ i By solving the linear system Ax = r, the coefficient vector can be obtained:

[0102] x = [a0, a1, b1, ..., a K ,b K ] T

[0103] The above method provides a high-precision curve modeling function for the complex cavity contours within the target occlusion area. It also provides structured output in JSON format, supporting real-time curve visualization. This function can be applied to directly generate the cross-sectional geometric contours of occlusion structures for navigation region segmentation; or the cavity contour curve sequence can be used as the traversable tunnel boundary in a path planner.

[0104] S22. Construct a three-dimensional voxel inference network and integrate geometric priors.

[0105] In a preferred embodiment of this application, a four-layer U-Net-style lightweight voxel encoder-decoder network is constructed to infer a complete three-dimensional voxel occupancy probability map from an initial sparse three-dimensional voxel array, and outputs the passage probability value of each voxel unit. The network input is a three-dimensional voxel grid. Where H, W, and D are the length, width, and depth dimensions of the voxel mesh, respectively, and C... in The number of input channels is specified. Structure recovery is performed by inputting the generated sparse 3D voxel map along with the geometric prior into the 3D voxel inference network.

[0106] Specifically, in this embodiment, the encoder, decoder, and secondary voxel occupancy probability prediction module of the three-dimensional voxel inference network, wherein,

[0107] The encoder is used to convert the height dimension into the feature dimension, outputting three-dimensional features;

[0108] The encoder uses a ResNet-50 network, consisting of L = 4 layers. In the l-th layer (l = 0, 1, 2, 3), the network performs a series of convolutional and pooling operations along the X and Y dimensions, converting the height dimension Z into the feature dimension and outputting 3D features. Each layer of the encoder includes a series of convolutional operations followed by a pooling layer with a stride of 2, reducing the resolution to half the original size.

[0109] The decoder is used to restore spatial resolution and to fit the cavity profile curve using Fourier series.

[0110] The decoder also employs a ResNet-50 network, using deconvolutional layers (upsampling) to progressively restore spatial resolution. Each decoding layer also fuses multi-scale features from the corresponding encoder layer via skip connections. A special module, the voxel occupancy constraint module under the occluder, is introduced, incorporating the aforementioned cavity contour modeling mechanism and fitting the cavity cross-sectional curve using a Fourier series. This curve is then used for secondary prediction of voxel occupancy probabilities.

[0111] The secondary voxel occupancy probability prediction module is used to predict the secondary voxel occupancy probability of voxels around the cavity contour curve. The secondary occupancy probability of each voxel is:

[0112]

[0113] Among them, P occ (x,y,z) represents the second-order occupancy probability of voxel (x,y,z), Bernoulli(p) is the prediction of random perturbation in the p-Beta(α,β) distribution, where p-Beta(α,β) is the random perturbation probability. The voxel position (y, z) is the 8-neighborhood around the cavity profile curve.

[0114] After obtaining a high-precision curve modeling function for the complex cavity contour inside the occluded target area, a secondary occupancy probability prediction is performed for each voxel.

[0115] The optimized voxel grid V can be obtained through the above three-dimensional voxel inference network. t ∈{c0,c1} H×W×Z The occupancy probability P of each voxel occ (x,y,z)∈[0,1], enhancing the accuracy of spatial structure modeling within occluded areas (such as under bridges or under street trees). Combining cavity contour functions and perturbation strategies, multi-layer occupancy probability modulation is applied to the initial voxel map to improve the geometric consistency of the cavity region.

[0116] S23. Construct the joint loss function.

[0117] S231. Construct the cavity edge voxel constraint loss function:

[0118]

[0119] in, The loss is the voxel constraint loss at the cavity edge, γ is the robustness threshold, and L is the number of slices. For each slice x, the Fourier coefficients of the current contour curve are obtained through a contour fitting process. Here, δ represents the true coefficients of the corresponding contours in the training data, and δ represents the contour fitting residuals.

[0120] The cavity edge voxel constraint loss is used to constrain the numerical consistency between the cavities inside the generated voxel mesh and the cavities inside the real voxel mesh, ensuring the accuracy of the internal structure of the cavities. In this embodiment, the Huber loss function is used to improve the robustness of point cloud reconstruction in occluded areas and suppress the interference of extreme errors on voxel mesh fitting.

[0121] S232. Constructing the voxel reconstruction loss function:

[0122]

[0123] in, For voxel reconstruction loss, yi Let be the predicted occupancy probability of the i-th voxel. is the actual occupancy label of the i-th voxel, where 0 indicates that the voxel is empty, 1 indicates that the voxel is occupied, and N is the total number of voxels in the voxel grid;

[0124] Voxel reconstruction loss is used to constrain the generated voxel mesh P l (i,c)∈[0,1] and the true voxel grid Numerical consistency is ensured to guarantee the accuracy of the geometric structure and maintain geometric consistency. In this embodiment, the voxel reconstruction loss is represented by the cross-entropy loss.

[0125] S233. Construct the joint loss function:

[0126]

[0127] in, For joint losses.

[0128] To ensure that the network output matches the actual cavity shape, two types of losses are introduced. and The total loss of the 3D inference network for the entire occluded scene is obtained. The total loss is the objective function for training the 3D voxel inference network for the occluded scene. This ensures that the network can not only generate the correct 3D structure, but also construct cavity regions that conform to the real contours, ultimately achieving accurate accessibility analysis of the occluded region.

[0129] S24. Construct the dataset.

[0130] To train and evaluate the 3D voxel inference network for occluded regions, this embodiment constructs a hybrid voxel occupancy dataset containing real-world collected data and simulated synthetic data, covering the structural modeling needs of typical occluded scenarios (such as street trees, bridges, etc.).

[0131] For typical occlusion areas in real-world environments, the data acquisition and processing workflow for real-world scenarios includes:

[0132] S241. Obtain actual collected data, including:

[0133] Acquire RGB images of scenes with different occlusions;

[0134] A dense point cloud model is generated using a multi-view stereo reconstruction method to obtain the three-dimensional spatial information of the target structure.

[0135] The dense point cloud model is projected from a top-down perspective and rendered to obtain a depth map of a set size.

[0136] Sparse 3D voxels are generated based on the depth map and camera intrinsic parameters and assigned occupancy labels, which are then used as training input samples.

[0137] In this embodiment, during image acquisition, a representative single tree or bridge scene is selected, and no less than 80 RGB images are acquired around the target for subsequent modeling. During point cloud reconstruction, a dense point cloud model is generated using a multi-view stereo reconstruction method to obtain the three-dimensional spatial information of the target structure. When generating a depth map, the three-dimensional point cloud is projected from a top-down perspective and rendered to obtain a standard-sized (256×256) depth map. During voxel construction, based on the depth map and camera intrinsic parameters, pixels are back-projected into three-dimensional space to generate a sparse three-dimensional voxel mesh and assign occupancy labels as training input samples.

[0138] To expand data diversity and improve the model's generalization ability to different occlusion structures, this embodiment also designs a highly controllable occlusion scene simulation data generation process:

[0139] S242. Obtain simulation synthesis data, including:

[0140] Obtain the occlusion model;

[0141] By performing geometric transformations on the occlusion model, various spatial occlusion forms can be obtained;

[0142] By randomly arranging and combining occlusion models, different occlusion scenes can be generated;

[0143] Render RGB images and depth maps from a virtual overhead view;

[0144] Sparse 3D voxels are generated based on the depth map and the geometric contours of occluded objects and assigned occupancy labels, serving as training supervision samples.

[0145] In this embodiment, representative tree and bridge models are selected from an open-source 3D model library, covering various typical occlusion structure forms. During model processing, the selected tree or bridge models undergo random scaling, cropping, rotation, and other geometric transformations to simulate occlusion structures with different heights, widths, occlusion densities, and pose distributions, forming various typical spatial occlusion forms such as tall and sparse tree canopies, low and dense tree canopies, and low-hanging or tilted bridge surfaces. During scene construction, multiple processed models are randomly arranged and combined to generate occlusion scenes of roadside trees on both sides of urban roads or scenes of passage under bridges. During image generation, RGB images and depth maps are rendered under a set virtual overhead view and output in a standardized format. During label generation, corresponding voxel meshes are generated based on the depth map and the geometric contours of the occlusion structure, and each voxel is assigned a real occupancy state as a training supervision sample.

[0146] S243. Construct a dataset based on real collected data and simulated synthetic data.

[0147] Each sample in the dataset uniformly contains the following components:

[0148] RGB images: used as input for semantic and structural recognition;

[0149] Depth maps: used for sparse voxel construction;

[0150] Initial voxel map: Represents the depth projection result;

[0151] Occupancy probability graph: used as a supervision target for network output.

[0152] S25. Train and optimize the 3D voxel inference network based on the joint loss function and the dataset.

[0153] In this embodiment, during network training, a voxelized pseudo-point cloud of size 256×256×16 is used as input, and an occupancy probability map of size 128×128×16 is used as output, which is then upsampled to the scene resolution. Upsampling can employ current mainstream voxel upsampling methods, such as trilinear interpolation, deconvolution, or nearest neighbor interpolation. The Adam optimizer (1 = 0.9, 2 = 0.999) is used, with a learning rate of 0.001 multiplied by 0.98 epochs. Training is performed on a single 11GB GPU with a batch size of 4, and convergence takes approximately 48 hours (80 epochs).

[0154] S26. Infer the three-dimensional voxel occupancy probability map from the initial sparse three-dimensional voxels using a three-dimensional voxel inference network.

[0155] In this embodiment, the 3D voxel inference network ultimately outputs a 3D voxel occupancy probability map. Each voxel stores a probability of 0–1, indicating whether the location is passable at a given height layer. The output data structure supports exporting as NumPy arrays, OctoMap format, or ROS PointCloud2 message format.

[0156] The occlusion region 3D voxel inference method of this application can reconstruct the occlusion region based on monocular images; it supports multi-format output and navigation interface interface, and can be integrated into SLAM, 3D mapping or path search system as spatial feasibility constraint; it can be used to query whether any spatial location is passable (occupancy probability > threshold); it can be used to extract the passability subgraph of a specified area (such as under a bridge or under a tree canopy) for path planning.

[0157] Based on the above-described method for 3D voxel inference in occluded regions, a second aspect of this application provides a 3D voxel inference system for occluded regions, such as... Figure 4 As shown, it includes:

[0158] The initial voxel generation module 10 is used to acquire an RGB image containing the semantic segmentation results of the occluded region and its corresponding depth map, and convert the depth map into initial sparse three-dimensional voxels.

[0159] The 3D voxel inference network construction module 20 is used to construct a 3D voxel inference network and infer the 3D voxel occupancy probability map from the initial sparse 3D voxels through the 3D voxel inference network.

[0160] A third aspect of this application provides a computer-readable medium storing computer-executable instructions for performing the aforementioned three-dimensional voxel inference method for occluded regions.

[0161] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this invention, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0162] Having introduced the occlusion region three-dimensional voxel inference method, system and readable medium of exemplary embodiments of the present invention, the following describes a computing device according to another exemplary embodiment of the present invention.

[0163] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0164] In some possible implementations, the computing device according to the present invention may include at least one processing unit and at least one storage unit. The storage unit stores program code that, when executed by the processing unit, causes the processing unit to perform the steps in the occlusion region three-dimensional voxel inference method according to various exemplary embodiments of the present invention described above.

[0165] The following reference Figure 5 To describe the computing device 30 according to this embodiment of the present invention. Figure 5 The computing device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0166] like Figure 5 As shown, the computing device 30 is presented in the form of a general-purpose computing device. The components of the computing device 30 may include, but are not limited to: at least one processing unit 31, at least one storage unit 32, and a bus 33 connecting different system components (including the storage unit 32 and the processing unit 31).

[0167] Bus 33 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or a local bus using any of the multiple bus structures.

[0168] Storage unit 32 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0169] Storage unit 32 may also include a program / utility 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0170] The computing device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.), one or more devices that enable a user to interact with the computing device 30, and / or any device that enables the computing device 30 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 35. Furthermore, the computing device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 36. As shown, network adapter 36 communicates with other modules used in the computing device 30 via bus 33. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the computing device 30, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0171] In some possible implementations, various aspects of the occlusion region three-dimensional voxel inference method provided by the present invention can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps in the occlusion region three-dimensional voxel inference method according to various exemplary embodiments of the present invention as described above.

[0172] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0173] The program product of the occlusion region three-dimensional voxel inference method according to embodiments of the present invention can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a computing device. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0174] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0175] The program code contained on the readable medium can be transmitted using any suitable medium, including—but not limited to—wireless, wired, fiber optic, RF, etc., or any suitable combination thereof. The program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc.—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0176] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0177] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0178] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0182] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for three-dimensional voxel inference in an occluded region, characterized in that, include: S10. Obtain an RGB image containing the semantic segmentation results of the occluded region and its corresponding depth map, and convert the depth map into an initial sparse three-dimensional voxel; S20. Construct a three-dimensional voxel inference network, and infer a three-dimensional voxel occupancy probability map from the initial sparse three-dimensional voxels through the three-dimensional voxel inference network.

2. The three-dimensional voxel inference method for occluded regions according to claim 1, characterized in that, In S10, an RGB image containing the semantic segmentation results of the occluded region and its corresponding depth map are obtained, and the depth map is converted into an initial sparse three-dimensional voxel, including: S11. Obtain the RGB image containing the semantic segmentation results of the occluded region and its corresponding depth map; S12. Extract the occluded region from the depth map based on the semantic segmentation result of the occluded region; S13. Convert the depth map into an initial sparse three-dimensional voxel: Where (x,y,z) are the initial sparse 3D voxel coordinates, (u,v) are the depth map coordinates, and f u f is the horizontal focal length, an intrinsic parameter of the camera. v The vertical focal length is the camera's intrinsic parameter, (c u ,c v () represents the principal point coordinates of the camera's intrinsic parameters; S14. Convert the initial sparse three-dimensional voxels into a binary voxel mesh diagram, wherein the voxel mesh V of the binary voxel mesh diagram is... t ∈{c0,c1} H×W×Z Where c0 represents an empty voxel, c1 represents an occupied voxel, H represents the length of the voxel grid, W represents the width of the voxel grid, and Z represents the height of the voxel grid.

3. The three-dimensional voxel inference method for occluded regions according to claim 2, characterized in that, In S20, a three-dimensional voxel inference network is constructed, and a three-dimensional voxel occupancy probability map is inferred from the initial sparse three-dimensional voxels through the three-dimensional voxel inference network, including: S21. Extract the cavity contour within the occluded area and generate a geometric prior; S22. Construct a three-dimensional voxel reasoning network and integrate geometric priors; S23. Construct the joint loss function; S24. Construct the dataset; S25. Train and optimize the three-dimensional voxel inference network based on the joint loss function and the dataset; S26. The three-dimensional voxel occupancy probability map is inferred from the initial sparse three-dimensional voxels through the three-dimensional voxel inference network.

4. The three-dimensional voxel inference method for occluded regions according to claim 3, characterized in that, In S21, the cavity contour within the occluded region is extracted to generate a geometric prior, including: From the initial sparse three-dimensional voxels, obtain unit-by-unit voxel slices along the X-axis. Each slice includes voxel S on the YZ plane. x (y,z), where, y∈[0,Y-1], z∈[0,Z-1]; For each slice, scan along the y-column from bottom to top to find the voxel with the smallest z-value. min (x, y), where each column y corresponds to a voxel z. min The curve bounded by (x, y) and the lines X = x, Z = 0 is taken as the contour curve of the slice, where the set of points on the contour curve of the slice is P(x) = {(y, z)}. min (x,y))}; Based on the point set on the contour curve of each slice, the cavity contour curve in the parameterized polar coordinate system is obtained by Fourier series as a geometric prior. The cavity profile curve is mapped onto voxel space to obtain the geometric prior voxel.

5. The three-dimensional voxel inference method for occluded regions according to claim 4, characterized in that, In S22, a three-dimensional voxel inference network is constructed and geometric priors are integrated, including: An encoder, decoder, and a secondary voxel occupancy probability prediction module are constructed for a 3D voxel inference network. The encoder is used to convert the height dimension into the feature dimension and output three-dimensional features; The decoder is used to restore spatial resolution and to fit the cavity profile curve using a Fourier series. The secondary voxel occupancy probability prediction module is used to predict the secondary voxel occupancy probability of voxels around the cavity contour curve, wherein the secondary occupancy probability of each voxel is: Among them, P occ (x,y,z) represents the second-order occupancy probability of voxel (x,y,z), Bernoulli(p) is the prediction of random perturbation in the p-Beta(α,β) distribution, where p-Beta(α,β) is the random perturbation probability. The voxel position (y, z) is the 8-neighborhood around the cavity profile curve.

6. The three-dimensional voxel inference method for occluded regions according to claim 5, characterized in that, In S23, the joint loss function is constructed, including: S231. Construct the cavity edge voxel constraint loss function: in, The loss is the voxel constraint loss at the cavity edge, γ is the robustness threshold, and L is the number of slices. For each slice x, the Fourier coefficients of the current contour curve are obtained through a contour fitting process. Here, δ represents the true coefficients of the corresponding contours in the training data, and δ represents the contour fitting residuals. S232. Constructing the voxel reconstruction loss function: y i ∈[0,1] in, For voxel reconstruction loss, y i Let be the predicted occupancy probability of the i-th voxel. is the actual occupancy label of the i-th voxel, where 0 indicates that the voxel is empty, 1 indicates that the voxel is occupied, and N is the total number of voxels in the voxel grid; S233. Construct the joint loss function: in, This is a joint loss.

7. The three-dimensional voxel inference method for occluded regions according to claim 6, characterized in that, S24. Construct the dataset, including: S241. Obtain actual collected data, including: Acquire RGB images of scenes with different occlusions; A dense point cloud model is generated using a multi-view stereo reconstruction method to obtain the three-dimensional spatial information of the target structure. The dense point cloud model is projected from a top-down perspective and rendered to obtain a depth map of a set size; Based on the depth map and camera intrinsic parameters, sparse three-dimensional voxels are generated and assigned occupancy labels, which are then used as training input samples. S242. Obtain simulation synthesis data, including: Obtain the occlusion model; Geometric transformations are performed on the occlusion model to obtain various spatial occlusion forms; The occlusion models are randomly arranged and combined to generate different occlusion scenes; Render RGB images and depth maps from a virtual overhead view; Sparse 3D voxels are generated based on the depth map and the geometric contours of the occluded objects and assigned occupancy labels as training supervision samples. S243. Construct a dataset based on the actual collected data and the simulated synthetic data.

8. A three-dimensional voxel inference system for occluded regions, characterized in that, include: The initial voxel generation module is used to acquire an RGB image containing semantic segmentation results of occluded regions and its corresponding depth map, and convert the depth map into initial sparse three-dimensional voxels. A three-dimensional voxel inference network construction module is used to construct a three-dimensional voxel inference network and infer a three-dimensional voxel occupancy probability map from the initial sparse three-dimensional voxels through the three-dimensional voxel inference network.

9. A computer-readable medium storing computer-executable instructions, characterized in that, The computer-executable instructions are used to execute the occlusion region three-dimensional voxel inference method as described in any one of claims 1 to 7.

10. A computing device, characterized in that, include: At least one processor, and a memory communicatively connected to said at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being used to perform the occlusion region three-dimensional voxel inference method as described in any one of claims 1 to 7.

Images