A compensation method, apparatus, electronic device, and storage medium for a sparse voxel model.

Abstract

This application provides a compensation method, apparatus, electronic device, and storage medium for sparse voxel models, applicable to vehicle-mounted perception systems equipped with multi-view vehicle cameras. The method includes simultaneously acquiring images from the vehicle's front, left, right, and rear multi-view vehicle cameras; rendering and extracting triangular mesh models representing the physical environment surrounding the vehicle; identifying rendering blank areas and generating corresponding region masks; and re-initializing the identified rendering blank areas for each vehicle camera to reconstruct the voxel models of the rendering blank areas. The sparse region detection and re-initialization steps are optimized without altering the overall framework of existing sparse voxel training, allowing direct integration into existing sparse voxel training systems without large-scale modifications to the original process. It exhibits high adaptability, effectively controls computational resources and memory usage, and is suitable for sparse voxel training scenarios of varying scales, demonstrating strong versatility.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a compensation method, apparatus, electronic device, and storage medium for a sparse voxel model. Background Technology

[0002] With the rapid development of 3D perception technology in fields such as autonomous driving and intelligent cockpits, voxel-based 3D reconstruction has become one of the core methods for understanding the vehicle environment. Voxel representation uses regular meshes to discretize and model 3D space, effectively supporting tasks such as point cloud data processing, obstacle detection, and high-precision map construction. In this process, the mesh model extracted from voxel data directly affects the accuracy and reliability of downstream applications such as path planning and collision detection.

[0003] Existing methods typically employ algorithms such as Poisson reconstruction and Marching Cubes to convert voxels to meshes, and use hole-filling algorithms to handle data gaps caused by occlusion or sensor limitations. Poisson reconstruction relies on global implicit function fitting, which, while generating continuous surfaces, is prone to oversmoothing. Marching Cubes, while preserving local geometric features, is sensitive to noise and easily introduces non-manifold structures when filling holes. In recent years, deep learning-based completion methods (such as 3D-UNet and generative adversarial networks) predict missing regions through data-driven approaches. However, in complex scenarios such as vehicle bumps and multi-object dynamic interactions, insufficient network generalization leads to geometric distortion (e.g., wheels are mistakenly filled as planes), and the model parameter count and computational latency are insufficient to meet the real-time requirements of automotive embedded platforms. Furthermore, traditional methods often use high-resolution voxels to improve detail preservation, but the number of vertices in a 3D mesh increases cubically with resolution, leading to a surge in memory consumption and a sharp drop in computational efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a compensation method, apparatus, electronic device and storage medium for sparse voxel models, so as to solve the technical problem that the existing voxel compensation method is inaccurate in detecting sparse regions and causes a computational burden on in-vehicle systems.

[0005] In a first aspect, the present invention provides a compensation method for sparse voxel models, applicable to vehicle-mounted perception systems equipped with multi-view vehicle-mounted cameras. The method includes simultaneously acquiring images from the vehicle's forward, left, right, and rearward multi-view vehicle-mounted cameras, rendering and extracting triangular mesh models representing the physical environment surrounding the vehicle, identifying rendering blank areas, and generating corresponding region masks; for each vehicle-mounted camera, re-initializing the identified rendering blank areas to reconstruct the voxel models of the rendering blank areas.

[0006] In an optional implementation, the rendering blank area is identified for each vehicle-mounted camera in the following way: Obtain the multi-view rendering depth map and rendering transparency map corresponding to the current sparse voxel model of the vehicle camera; Triangular meshes are extracted based on a sparse voxel model, and the mesh depth map is calculated. Obtain the monocular depth map corresponding to each viewpoint. The monocular depth map is obtained by linearly scaling the original monocular depth map based on the median and dispersion of the effective pixels of the Mesh and the effective pixels of the monocular. Based on the pixel-level differences between the Mesh depth map and the monocular depth map, and combined with the effective rendering area mask, a sparse pixel mask for the vehicle-mounted camera is generated.

[0007] In optional implementations, pixel-level difference determination includes at least one of the following: Mesh depth values ​​are invalid, but monocular depth values ​​are within the preset valid range; Both mesh depth values ​​and monocular depth values ​​are valid, and the absolute value of the difference between the two exceeds the depth threshold. At the same time, after alignment, the monocular depth value is less than the mesh depth value.

[0008] In an optional implementation, it further includes: Connectivity region marking is performed on the sparse pixel masks of the vehicle-mounted cameras from various perspectives, and each connected region is mapped to a unique 3D region identifier. Based on multi-view consistency filtering, 3D region identifiers detected by N different views are retained to generate an effective sparse region dictionary.

[0009] In an optional implementation, for each vehicle-mounted camera, the identified rendering blank areas are reinitialized in the following manner: Based on the sparse region mask of each vehicle camera, 3D point sampling and world coordinate back projection are performed independently for each vehicle camera view to obtain the spatial point cloud corresponding to the sparse region. Based on the distribution scale of the spatial point cloud and the preset spatial range of the vehicle perception scene, the voxel resolution level adapted to the blank area is dynamically determined. For the voxel space covered by the spatial point cloud, voxel parameter reset, parent voxel iterative subdivision, or new voxel creation operations are performed, and all newly participating voxels are given high optimization priority and training protection period.

[0010] In an optional implementation, it further includes: Based on the geometric surface of the real obstacles around the vehicle represented by the Mesh, calculate the distance from the center of each voxel to the surface. A voxel is deleted when it meets all of the following conditions: The distance is greater than the preset safe distance threshold; the training protection period has expired; or it is not a newly created or parameter reset voxel.

[0011] In an optional implementation, the total number of vertices in the Mesh is limited to no more than a preset upper limit; Limit the number of sparse pixels sampled per single viewpoint to no more than the preset maximum number of sampled pixels; The total number of sub-voxels created in each round of voxel iteration subdivision is limited to a preset upper limit; The number of retries for redundant voxel deletion operations is limited to no more than 3.

[0012] Secondly, the present invention provides a compensation device for a sparse voxel model, applicable to an onboard perception system of a vehicle equipped with a multi-view onboard camera, the device comprising: The recognition module is used to simultaneously acquire images from the vehicle's front, left, right and rear multi-view onboard cameras, render and extract triangular mesh models representing the physical environment around the vehicle, identify blank rendering areas, and generate corresponding area masks. The reconstruction module is used to reinitialize the identified rendering blank areas for each vehicle camera in order to reconstruct the voxel model of the rendering blank areas.

[0013] Thirdly, the present invention provides an electronic device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the compensation method for any of the sparse voxel models described in the foregoing embodiments.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of a compensation method for any of the sparse voxel models described in the foregoing embodiments.

[0015] This application provides a compensation method, apparatus, electronic device, and storage medium for sparse voxel models, applicable to vehicle-mounted perception systems equipped with multi-view vehicle cameras. The method includes simultaneously acquiring images from the vehicle's front, left, right, and rear multi-view vehicle cameras; rendering and extracting triangular mesh models representing the physical environment surrounding the vehicle; identifying rendering blank areas and generating corresponding region masks; and re-initializing the identified rendering blank areas for each vehicle camera to reconstruct the voxel models of the rendering blank areas. This application automatically compensates for sparse areas to fill rendering blank areas, improving the geometric integrity and rendering effect of sparse voxel models. It optimizes the sparse area detection and re-initialization steps without changing the overall framework of existing sparse voxel training, allowing direct integration into existing sparse voxel training systems without large-scale modifications to the original process. It exhibits high adaptability, reasonably controls computational resources and memory usage, adapts to different scales of sparse voxel training scenarios, and has strong versatility. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a compensation method for a sparse voxel model provided in this application embodiment; Figure 2 A schematic diagram of the structure of a compensation device for a sparse voxel model provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0019] Example 1 Figure 1 A flowchart illustrating a compensation method for a sparse voxel model provided in an embodiment of this application. Figure 1 As shown, this application provides a compensation method for a sparse voxel model, applicable to vehicle perception systems equipped with multi-view vehicle cameras, including: S1. Simultaneously acquire images from the vehicle's front, left, right, and rear multi-view onboard cameras, render and extract a triangular mesh model representing the physical environment surrounding the vehicle, identify blank rendering areas, and generate corresponding area masks.

[0020] In step S1, the rendering results of the voxel model from each camera viewpoint can be obtained, providing a basis for subsequent depth comparison and ensuring that the rendering data format is consistent for use in subsequent steps.

[0021] Specifically, for each vehicle-mounted camera, the rendering blank areas are identified using the following methods: Obtain the multi-view rendering depth map and rendering transparency map corresponding to the current sparse voxel model of the vehicle camera.

[0022] Triangular meshes are extracted based on a sparse voxel model, and the mesh depth map is calculated.

[0023] Obtain the monocular depth map corresponding to each viewpoint. The monocular depth map is obtained by linearly scaling the original monocular depth map based on the median and dispersion of the effective pixels of the Mesh and the effective pixels of the monocular.

[0024] Based on the pixel-level differences between the Mesh depth map and the monocular depth map, and combined with the effective rendering area mask, a sparse pixel mask for the vehicle-mounted camera is generated.

[0025] Pixel-level difference determination may include at least one of the following: The mesh depth value is invalid, but the monocular depth value is within the preset valid range. Both the mesh depth value and the monocular depth value are valid, and the absolute value of the difference between the two exceeds the depth threshold. At the same time, the monocular depth value is less than the mesh depth value after alignment.

[0026] Furthermore, the sparse pixel masks of the vehicle-mounted cameras from various viewpoints are labeled with connected regions, and each connected region is mapped to a unique 3D region identifier. Based on multi-view consistency filtering, the 3D region identifiers detected jointly by N different viewpoints are retained to generate an effective sparse region dictionary.

[0027] S2. For each vehicle-mounted camera, the identified rendering blank areas are reinitialized in order to reconstruct the voxel model of the rendering blank areas.

[0028] For each vehicle-mounted camera, the identified rendering blank areas are reinitialized in the following way: Based on the sparse region masks of each vehicle-mounted camera, 3D point sampling and world coordinate back-projection are independently performed for each vehicle-mounted camera's viewpoint to obtain the spatial point cloud corresponding to the sparse region. According to the distribution scale of the spatial point cloud and the preset spatial range of the vehicle-mounted perception scene, the voxel resolution level adapted to the blank area is dynamically determined. For the voxel space covered by the spatial point cloud, voxel parameter reset, parent voxel iterative subdivision, or new voxel creation operations are performed, and all newly participating voxels are given high optimization priority and a training protection period.

[0029] It is important to note that during training, the total number of vertices in the Mesh should not exceed a preset limit, the number of sparse pixel samples per view should not exceed a preset maximum number of sampled pixels, the total number of sub-voxels created in each round of voxel iteration subdivision should not exceed a preset limit, and the number of retries for redundant voxel deletion operations should not exceed 3.

[0030] This application provides a compensation method for a sparse voxel model. Through multi-view rendering and depth fusion technology, it accurately identifies blank areas (sparse regions) during sparse voxel training, and achieves precise localization of these sparse regions based on monocular depth alignment and 3D region labeling. Then, through voxel-level calculation, iterative subdivision, and parameter resetting techniques, it fills and optimizes the sparse regions with voxels. Simultaneously, it sets voxel priorities and protection periods to ensure that filled voxels are prioritized for training and not mistakenly deleted. Ultimately, this improves the integrity of the sparse voxel model and enhances its resource adaptability.

[0031] Example 2 In one embodiment of this application, a sparse region detection method is provided.

[0032] In one feasible implementation, parameter values ​​for the core switch and detection switch, as well as iteration information, can be configured in the system. The core switch is set via system configuration parameters and is a Boolean value; True indicates that the entire detection and compensation process is enabled, while False skips all core steps.

[0033] The detection switch is a Boolean parameter independent of the core switch. True indicates that sparse region detection is enabled, while False skips the detection process and directly proceeds to voxel subdivision.

[0034] Iteration information can include the number of iterations and the interval between iterations.

[0035] The iteration count is defined as [A, B]. A and B are integer parameters configured by the system, representing the start and end iteration counts of the detection, respectively. Detection is triggered only when the current training iteration count falls within this range. The iteration interval C represents the interval between detection iterations; that is, a detection is performed after every C iterations to avoid frequent detection consuming resources. If the remaining sparsity ratio after the last detection exceeds a threshold (the threshold is a floating-point parameter configured by the system, ranging from 0 to 1), the iteration interval restriction can be ignored, and detection can continue to ensure full coverage of the sparse region and avoid missed detections.

[0036] The specific testing process may include the following steps: (1) Multi-camera rendering Iterate through all cameras in the system (no sampling required, process each one individually) and obtain the intrinsic parameters of the current camera. ) and extrinsic parameters (world-camera transformation matrix) ).

[0037] Call the voxel model's rendering interface, input the current camera parameters, enable depth output and transparency output functions, and perform the rendering operation.

[0038] Rendering depth ( ), rendering transparency ( Calculated from the rendered transmittance T. ).

[0039] Using the camera name as the key, the rendering depth and transparency are organized into a dictionary format and stored to ensure that they can be quickly retrieved by camera later.

[0040] In this step, the rendering results of the voxel model from each camera's perspective are obtained, providing a basis for subsequent depth comparison and ensuring that the rendering data format is uniform and can be directly used in subsequent steps.

[0041] (2) TSDF Mesh Extraction Construct a TSDF volume (truncated symbolic distance function volume), configure the volume parameters (the parameters are system-configurable and do not require additional calculation), and limit the spatial range and voxel size of the volume.

[0042] Iterate through all cameras, backproject the rendering depth of each camera into 3D space, and combine the camera's intrinsic and extrinsic parameters to generate an RGBD image. Then, sequentially fuse the RGBD images from all cameras into a TSDF volume to achieve multi-view depth fusion.

[0043] Triangle meshes are extracted from the TSDF volume. An upper limit is set on the number of mesh vertices to limit mesh complexity and avoid excessive computational resource consumption. The extracted mesh is then smoothed to remove noisy vertices, ensuring the accuracy of the mesh geometry.

[0044] In this step, a 3D geometric structure (TriangleMesh) is generated as a benchmark for depth alignment and sparse region determination to ensure the integrity and controllable complexity of the Mesh structure.

[0045] (3) Mesh depth map calculation Here, mesh depth maps from each camera's perspective are generated as a reference benchmark for depth comparison, ensuring that the mesh depth value of each pixel is accurate and usable for subsequent depth difference detection. Specifically, this involves traversing all cameras and performing the following operations for each camera: Transform all vertices of the Mesh from world coordinates to the current camera coordinates using camera extrinsic parameters (world-camera transformation matrix M). The calculation formula is as follows: ; in, Let (x, y, z) be the world coordinates of the Mesh vertices. Here are the transformed camera coordinates (x, y, z). It is a 3×4 world-camera transformation matrix, which can be obtained through tools such as calibration or colmap and called directly.

[0046] Projecting the mesh vertices in camera coordinates onto the current camera's image plane yields the corresponding pixel coordinates. The calculation formula is as follows: ; ; in, This is the current camera's focal length (internal parameter). z is the coordinates of the principal point of the current camera (intrinsic parameter), and z is the depth value of the Mesh vertex in the camera coordinate system. ).

[0047] For each triangular mesh, calculate its bounding box on the image plane and iterate through all pixels within the bounding box. Using the barycentric coordinate method, determine whether a pixel is inside the triangle. If it is inside, interpolate the depth values ​​of the three vertices of the triangle to obtain the mesh depth value of that pixel.

[0048] The interpolated depth value of each pixel is filtered to retain valid depths (outliers that are less than the minimum threshold range or greater than the maximum threshold range are removed). If a pixel is not inside any triangle or the interpolated depth is abnormal, the mesh depth of that pixel is marked as invalid (value is 0). (4) Monocular depth acquisition and alignment In the fourth step, the original monocular depth map corresponding to each camera can be read from the path specified by the system, ensuring that the depth map size matches the rendered depth map and mesh depth. Figure 1 To.

[0049] Specifically, the original monocular depth map can be converted into a floating-point format, normalized, and the depth values ​​can be mapped to the [0,1] range (eliminating differences in pixel value ranges).

[0050] If the original monocular depth value is negative (as shown in some monocular depth model output formats), the absolute value of all depth values ​​is taken to ensure that the depth values ​​are positive. Alternatively, the reciprocal of the depth value can be taken (only for pixels with depth values ​​greater than 0) to optimize the depth distribution and facilitate subsequent alignment.

[0051] If the depth map size is inconsistent with the rendered depth map, adjust the size using bilinear interpolation to ensure that the three sizes are completely consistent.

[0052] In one feasible embodiment, the foreground mask can be obtained in the following manner: Obtain the foreground mask (binary image) of the current camera, and perform erosion processing on the foreground mask using a circular structuring element (the radius of the structuring element is a system-configurable parameter to adapt to different image sizes). After erosion, filter out the foreground region (mask value of 1) as the filtering condition for the effective depth region; Furthermore, in one feasible implementation, the depth map can be optimized in the following way: Pixels in the Mesh depth map whose depth values ​​are within a set range and are located in the foreground region after erosion are selected and marked as valid Mesh pixels.

[0053] Pixels in the pre-processed monocular depth map with a depth value greater than 0 and less than 99% of the maximum depth value (to avoid overloaded pixels) and located in the foreground region after erosion are selected and marked as valid monocular pixels.

[0054] If the number of effective pixels in the Mesh or the number of effective pixels in a single camera is less than 50 (minimum effective pixel threshold to avoid alignment deviation due to too few effective points), the alignment is deemed invalid, an alignment depth map with null values ​​is output, and the current camera's single-view depth alignment is abandoned.

[0055] Furthermore, the step of performing linear scale alignment based on the median and dispersion of the effective pixels of the mesh and the effective pixels of a single eye may include: Calculate the median depth of the effective pixels in the Mesh ( and dispersion ( The median is the median of the effective depth values, and the dispersion is the average of the differences between the effective depth values ​​and the median.

[0056] Calculate the median depth of the effective pixels in a single eye ( and dispersion ( The calculation method is consistent with the effective depth of the Mesh.

[0057] The preprocessed monocular depth is aligned to the Mesh depth scale via a linear transformation. The calculation formula is as follows: ; in, This is the preprocessed monocular depth value. This is the aligned monocular depth value.

[0058] By acquiring the monocular depth and aligning it to the Mesh depth scale, we can ensure the consistency of depth comparison and avoid misjudgment of sparse regions due to differences in depth scale.

[0059] (5) Sparse region mask calculation Iterate through all cameras and perform the following operations for each camera: Based on the current camera's rendering depth, rendering opacity, and foreground mask, the effective rendering area is selected, and the judgment criteria include at least the following: (a) The rendering depth value is within the set range (effective rendering depth); (b) The rendering transparency value is greater than the threshold (configurable by the system, ranging from 0 to 1, to ensure effective pixel rendering). (c) Located within the foreground mask area (mask value is 1); Pixels that satisfy all three conditions are marked as valid pixels for rendering.

[0060] The specific criteria for depth difference detection may include: Case 1: Mesh has no depth but effective monocular depth - The mesh depth value is 0 (invalid), and the monocular depth value is within the set range (valid) after alignment. Such pixels are marked as potential sparse pixels. Case 2: Mesh has depth but the deviation is too large - The mesh depth value is valid (within the set range), and the monocular depth value is valid after alignment. The absolute value of the difference between the two exceeds the depth threshold (configurable by the system). At the same time, the monocular depth value is less than the mesh depth value (indicating that the monocular detects an area closer to the camera, which is not covered by the mesh). Such pixels are marked as potential sparse pixels.

[0061] Furthermore, a fixed-size rectangular kernel (system-configurable) can be used to calculate the local density of the mesh depth map. The density value is the proportion of effective mesh depth pixels within the kernel. Pixels with local density values ​​below the density threshold (system-configurable, range 0~1), which are valid rendering pixels and have effective mesh depth values, can be marked as potential sparse pixels.

[0062] Merge all potential sparse pixels to generate an initial sparse mask (potential sparse pixels are marked as 1, and the rest as 0).

[0063] Calculate the local density of the initial sparse mask (using a smaller rectangular kernel). Regions with local density values ​​higher than the dense threshold (configurable by the system) are identified as dense regions and removed from the initial sparse mask (marked as 0).

[0064] The proportion of effective pixels rendered in the entire image is statistically analyzed. If the proportion is less than 5%, it indicates that the effective rendering area of ​​the current camera is too small and the data is unreliable. An empty sparse mask (all 0s) is output, and the sparse mask calculation of the current camera is abandoned.

[0065] The generated sparse mask is subjected to small-region culling, which removes connected small regions with a pixel count less than the minimum region pixel count (configurable by the system) to avoid noise interference.

[0066] (6) Generation of connected regions and 3D identifiers In the sixth step, we can iterate through all cameras and perform the following operations for each camera's sparse mask: First, perform connected component extraction using the following steps: A connected component labeling algorithm is used to label the sparse mask with connected components, assigning a unique label value to each connected component. Connected components with a pixel count not less than the minimum region pixel count (system configurable) are selected, and small region noise is removed.

[0067] Record the core information of each connected region—the coordinates (v, u) of all pixels within the region, and the bounding box coordinates (minimum / maximum row and column).

[0068] Calculate the center pixel coordinates of each connected region: Take the center row (v) and center column (u) of the bounding box of the region, that is... , (Round down to ensure integer pixel coordinates).

[0069] Next, depth values ​​are selected according to priority. If the mesh depth value of the center pixel is valid ([0.1, 20.0]), the mesh depth value is selected as the reference depth for that region. If the mesh depth value is invalid, the aligned monocular depth value of the center pixel (which must be valid) is selected as the reference depth. If both are invalid, the connected region is determined to be invalid, and subsequent 3D label generation is abandoned.

[0070] Finally, a 3D coordinate transformation from pixel to world coordinates is performed. This includes first transforming the pixel coordinates to camera coordinates: Using the center pixel coordinates (v, u) and the reference depth value (d), combined with camera intrinsic parameters, the coordinates are converted to camera coordinates. The calculation formula is as follows: ; ; .

[0071] Then transform the camera coordinates to world coordinates. Use the camera extrinsic parameters (camera-world transformation matrix). The formula for converting camera coordinates to world coordinates is as follows: .

[0072] Calculate the voxel-quantized coordinates in world coordinates, combine them with the scene extent, and generate a unique 3D region identifier (q). The calculation formula is as follows: ; Where s is the scene quantization step size, which is configurable by the system.

[0073] The camera names corresponding to the same 3D region are aggregated to generate a 3D identifier statistical dictionary, which is used for subsequent multi-view consistency filtering.

[0074] (7) Multi-view consistency filtering Here, you can obtain a 3D identifier statistics dictionary (key is 3D identifier, value is a list of camera names).

[0075] Set a maximum number of viewpoints threshold (system configurable, integer), which represents the maximum number of camera viewpoints that can be detected in a single sparse region (if the number exceeds this threshold, it is judged as noise, which may be a misjudgment of a dense region).

[0076] Traverse the 3D identifier statistics dictionary and retain 3D identifiers whose camera name list length does not exceed the maximum number of viewpoints threshold as effective sparse regions.

[0077] Finally, the effective sparse regions are organized into a dictionary format (key is a 3D identifier, value is a list of corresponding camera names), which serves as the core output of sparse region detection, namely the effective sparse region dictionary, and is used in the subsequent re-initialization process.

[0078] By filtering out false detection regions caused by single-view noise, the effective sparse region is retained, ensuring that the output effective sparse region can be directly used for reinitialization and avoiding invalid compensation.

[0079] Example 3 In one embodiment of this application, a method for reinitializing sparse regions is provided.

[0080] In one feasible implementation, sparse region reinitialization can be performed camera-by-camera, rather than aggregated by 3D identifier: each camera individually reads its own sparse mask and aligned monocular depth, completes sparse pixel 3D point sampling and voxel initialization, and then proceeds to the next camera after processing is complete. This avoids memory overload caused by simultaneous processing of data from multiple regions and multiple cameras, ensures stable workflow operation, and facilitates subsequent troubleshooting of processing anomalies in individual cameras.

[0081] In a specific embodiment, the identified rendering blank areas are reinitialized in the following manner: (1) Collect sparse pixel 3D points Similarly, process each camera individually, reading the core data of the current camera: sparse mask, aligned monocular depth map, and camera intrinsic parameters. Camera extrinsic parameters (camera-world transformation matrix) ).

[0082] Check if the aligned monocular depth map of the current camera is valid (not empty, size and sparse mask are consistent). If invalid, skip the processing of the current camera, record the exception log, and proceed to the processing of the next camera.

[0083] For each camera, obtain the coordinates (v, u) of all pixels marked as 1 in the current camera's sparse mask, i.e., the sparse region pixels. Sample the sparse region pixels, with the number of samples not exceeding the maximum number of samples (configurable by the system, an integer). If the number of sparse region pixels is less than the maximum number of samples, sample all of them. Random sampling is used to ensure that the sampling points are evenly distributed within the sparse region.

[0084] Similarly, for each sampled pixel (v, u), its aligned monocular depth value (d) is obtained, and valid depth values ​​(in the range [0.1, 20.0]) are filtered out, removing pixels with abnormal depths. A conversion from pixel coordinates to camera coordinates is also performed. Using the sampled pixel coordinates (v, u) and the effective depth value (d), combined with the camera intrinsic parameters, the values ​​are converted to camera coordinates. The calculation formula is the same as in the previous embodiment: ; ; .

[0085] And to transform camera coordinates to world coordinates. This can be done using a camera-to-world transformation matrix. The formula for converting camera coordinates to world coordinates is as follows: .

[0086] All transformed valid world coordinate 3D points are compiled into a 3D point list for subsequent single-region initialization. If the 3D point list is empty, subsequent processing for the current camera is skipped, and an exception log is recorded.

[0087] In this step, world coordinate 3D point data is provided for voxel initialization to ensure that the 3D points accurately correspond to sparse regions, which can be directly used for voxel-level calculation and voxel creation.

[0088] (2) Single-region initialization The target voxel level can be calculated in the following ways: Calculate the size of the region Calculate the difference between the maximum and minimum values ​​of the 3D point list on the x, y, and z coordinate axes, and take the maximum value of the three differences as the region size.

[0089] Get scene size The scene size is a pre-configured parameter of the system, which is the maximum spatial range of the entire training scene.

[0090] The target voxel level L is calculated using the following formula: ; Where A, B, C, and K are configurable simplification coefficients of the system, A is the minimum voxel level, B is the maximum voxel level, C is the level offset coefficient, and K is the scaling coefficient; L is ultimately limited to the range [A, B] to ensure that the voxel level is adapted to the area size and scene size, while taking into account both filling accuracy and resource consumption.

[0091] The steps for encoding and generating voxel paths, uniquely identifying voxel locations, may include: Calculate voxel size : ; The size of the voxel is related to the target voxel level L. The scope of the scenario is pre-configured by the system.

[0092] For each 3D point, subtract the minimum scene coordinates (x, y, z). (System pre-configuration), then divided by the voxel size Round down to obtain the voxel coordinates (i, j, k) corresponding to the 3D point. The calculation formula is: .

[0093] Here, by using 3D point data to complete voxel-level calculations, existing voxel matching, and voxel creation / parameter reset, it is possible to ensure that voxels adapt to sparse regions and achieve accurate filling.

[0094] (3) Voxel iterative subdivision The step of performing parent voxel iterative subdivision for the voxel space covered by the spatial point cloud may include: Initialize the parent voxel index list (parent voxel indices in input branch 2) and the current iteration count (initial value is 0).

[0095] Perform the following operations in each iteration: (S1) Determine whether the current iteration count has reached the maximum iteration count, or whether the number of voxels subdivided in each round has reached the upper limit. If so, terminate the iteration.

[0096] (S2) For each parent voxel in the parent voxel index list, call the voxel subdivision interface to perform the subdivision operation, and subdivide 1 parent voxel into 8 child voxels.

[0097] (S3) Update the voxel path encoding and voxel level of the child voxel (child voxel level = parent voxel level + 1).

[0098] (S4) Query whether the child voxel matches the required voxel path code. If it matches, add the child voxel index to the matching voxel list and no longer participate in subsequent subdivisions. If it does not match and the child voxel level is less than the target level, add the child voxel index to the new parent voxel index list for the next iteration.

[0099] (S5) Record the sub-voxel index created in this round of subdivision and update the voxel count for each round of subdivision.

[0100] (S6) Increment the current iteration count by 1 and update the parent voxel index list to the new parent voxel index list.

[0101] After the iteration terminates, all sub-voxel indices created by the subdivision are organized into a subdivision-created voxel list for subsequent parameter reset.

[0102] By iteratively subdividing the parent voxel until the voxel level meets the target, sparse regions can be accurately filled, ensuring that the voxel level is consistent with the target level.

[0103] (4) Voxel creation and parameter reset The step of resetting voxel parameters for the voxel space covered by the spatial point cloud may include: Call the voxel creation interface of the voxel model, input the voxel path code, target voxel level, and voxel center world coordinates, and the interface will automatically complete the voxel creation and initialization.

[0104] After creation, call the voxel query interface to verify whether the voxel was created successfully (whether the voxel index is valid). If creation fails, record the exception log and skip the subsequent operations for that voxel.

[0105] When resetting parameters, you can call the voxel model's parameter reset interface and input a list of voxel indices whose parameters need to be reset. The interface automatically resets the voxel's SH parameters and density parameters to the system's pre-configured initial values, ensuring that the voxels' initial states are consistent and can participate in training normally. After the reset is complete, check whether the voxel parameters were reset successfully. If it fails, record the exception log and re-execute the reset operation (maximum of 3 retries, system configurable).

[0106] By generating new voxels or resetting existing voxel parameters in this step, sparse regions can be filled, ensuring that voxels can participate normally in subsequent training without additional configuration.

[0107] (5) Removal of redundant voxels based on Mesh distance In this step, the indexes of all voxels in the current voxel model, the world coordinates of the voxel centers, the Mesh model extracted during the sparse region detection stage, and the current training iteration number can be obtained.

[0108] Next, construct a KD-Tree for the Mesh (for fast distance calculation and improved efficiency), or perform uniform sampling on the Mesh surface to obtain a list of Mesh surface sampling points.

[0109] Iterate through all voxels in the current voxel model and calculate the minimum distance between the world coordinates of each voxel center and the Mesh (i.e., the shortest distance from the voxel center to the Mesh surface).

[0110] The filtering criteria include the minimum distance between the voxel center and the Mesh being greater than the distance threshold (configurable by the system). Such voxels are considered redundant voxels (they deviate too much from the Mesh geometry and do not need to be retained).

[0111] The screening criteria also include voxels that are not within the protection period, i.e., the current training iteration count - voxel creation / reset iteration count > protection period (configurable by the system). Such voxels can be deleted.

[0112] The filtering criteria also include voxels that are not newly created voxels or voxels with reset parameters (i.e., not in the reset voxel index list or the newly created voxel list).

[0113] If a voxel meets all three of the above conditions, it is marked as a removable redundant voxel and compiled into a list of removable voxel indices.

[0114] It also includes a step for deleting redundant voxels. The voxel deletion interface of the voxel model can be called, a list of deletable voxel indices can be input, and the deletion operation can be performed. After deletion, it checks whether the voxels were deleted successfully. If some voxels failed to be deleted, an exception log is recorded, and the deletion operation is retried (maximum of 3 retries).

[0115] By deleting redundant voxels that are too far from the Mesh, the voxel distribution is optimized, preventing invalid voxels from occupying resources, while also protecting newly created voxels with reset parameters from being accidentally deleted.

[0116] Example 4 Figure 2 This is a schematic diagram of the structure of a compensation device for a sparse voxel model provided in an embodiment of this application. Figure 2 As shown, based on the same inventive concept, this application also provides a compensation device for a sparse voxel model, suitable for an in-vehicle perception system of a vehicle equipped with a multi-view in-vehicle camera. The device 20 includes: The recognition module 210 is used to simultaneously acquire images from the vehicle's front, left, right and rear multi-view onboard cameras, render and extract a triangular mesh model representing the physical environment around the vehicle, identify the rendering blank areas, and generate the corresponding area mask. The reconstruction module 220 is used to reinitialize the identified rendering blank areas for each vehicle camera in order to reconstruct the voxel model of the rendering blank areas.

[0117] In a preferred embodiment, the reconstruction module 220 identifies the rendering blank areas for each vehicle-mounted camera in the following manner: Obtain the multi-view rendering depth map and rendering transparency map corresponding to the current sparse voxel model of the vehicle camera; Triangular meshes are extracted based on a sparse voxel model, and the mesh depth map is calculated. Obtain the monocular depth map corresponding to each viewpoint. The monocular depth map is obtained by linearly scaling the original monocular depth map based on the median and dispersion of the effective pixels of the Mesh and the effective pixels of the monocular. Based on the pixel-level differences between the Mesh depth map and the monocular depth map, and combined with the effective rendering area mask, a sparse pixel mask for the vehicle-mounted camera is generated.

[0118] In a preferred embodiment, pixel-level difference determination includes at least one of the following: Mesh depth values ​​are invalid, but monocular depth values ​​are within the preset valid range; Both mesh depth values ​​and monocular depth values ​​are valid, and the absolute value of the difference between the two exceeds the depth threshold. At the same time, after alignment, the monocular depth value is less than the mesh depth value.

[0119] In a preferred embodiment, it further includes: Connectivity region marking is performed on the sparse pixel masks of the vehicle-mounted cameras from various perspectives, and each connected region is mapped to a unique 3D region identifier. Based on multi-view consistency filtering, 3D region identifiers detected by N different views are retained to generate an effective sparse region dictionary.

[0120] In a preferred embodiment, the reconstruction module 220 reinitializes the identified rendering blank areas for each vehicle camera in the following manner: Based on the sparse region mask of each vehicle camera, 3D point sampling and world coordinate back projection are performed independently for each vehicle camera view to obtain the spatial point cloud corresponding to the sparse region. Based on the distribution scale of the spatial point cloud and the preset spatial range of the vehicle perception scene, the voxel resolution level adapted to the blank area is dynamically determined. For the voxel space covered by the spatial point cloud, voxel parameter reset, parent voxel iterative subdivision, or new voxel creation operations are performed, and all newly participating voxels are given high optimization priority and training protection period.

[0121] In a preferred embodiment, the reconstruction module is further configured to: Based on the geometric surface of the real obstacles around the vehicle represented by the Mesh, calculate the distance from the center of each voxel to the surface. A voxel is deleted when it meets all of the following conditions: The distance is greater than the preset safe distance threshold; the training protection period has expired; or it is not a newly created or parameter reset voxel.

[0122] In a preferred embodiment, the total number of vertices in the Mesh is limited to no more than a preset upper limit; Limit the number of sparse pixels sampled per single viewpoint to no more than the preset maximum number of sampled pixels; The total number of sub-voxels created in each round of voxel iteration subdivision is limited to a preset upper limit; The number of retries for redundant voxel deletion operations is limited to no more than 3.

[0123] Example 5 Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 300 includes a processor 310, a memory 320, and a bus 330.

[0124] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 and the memory 320 communicate via the bus 330. When the machine-readable instructions are executed by the processor 310, they can perform the operations described above. Figure 1 The steps of a compensation method for a sparse voxel model shown in the method embodiment are described in detail in the method embodiment, and will not be repeated here.

[0125] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of a compensation method for a sparse voxel model shown in the method embodiment are described in detail in the method embodiment, and will not be repeated here.

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0127] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0128] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0130] It should be noted that if the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0132] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A compensation method for a sparse voxel model, characterized in that, An in-vehicle perception system suitable for vehicles equipped with multi-view in-vehicle cameras, the method comprising: Simultaneously acquire images from multi-view vehicle cameras in the front, left, right and rear directions, render and extract triangular mesh models representing the physical environment around the vehicle, identify blank rendering areas, and generate corresponding area masks. For each vehicle-mounted camera, the identified rendering blank areas are reinitialized in order to reconstruct the voxel model of the rendering blank areas.

2. The method according to claim 1, characterized in that, For each vehicle-mounted camera, the rendering blank areas are identified using the following method: Obtain the multi-view rendering depth map and rendering transparency map corresponding to the current sparse voxel model of the vehicle camera; Based on the sparse voxel model, a triangular mesh is extracted, and the mesh depth map is calculated. Obtain monocular depth maps corresponding to each viewpoint, wherein the monocular depth maps are obtained by linearly scaling the original monocular depth maps based on the median and dispersion of the effective pixels of the mesh and the effective pixels of the monocular. Based on the pixel-level difference between the Mesh depth map and the monocular depth map, and combined with the effective rendering region mask, a sparse pixel mask for the vehicle-mounted camera is generated.

3. The method according to claim 2, characterized in that, The pixel-level difference determination includes at least one of the following: Mesh depth values ​​are invalid, but monocular depth values ​​are within the preset valid range; Both mesh depth values ​​and monocular depth values ​​are valid, and the absolute value of the difference between the two exceeds the depth threshold. At the same time, after alignment, the monocular depth value is less than the mesh depth value.

4. The method according to claim 2 or 3, characterized in that, Also includes: Connectivity region marking is performed on the sparse pixel masks of the vehicle-mounted cameras from various perspectives, and each connected region is mapped to a unique 3D region identifier. Based on multi-view consistency filtering, 3D region identifiers detected by N different views are retained to generate an effective sparse region dictionary.

5. The method according to claim 4, characterized in that, For each vehicle-mounted camera, the identified rendering blank areas are reinitialized in the following way: Based on the sparse region mask of each vehicle camera, 3D point sampling and world coordinate back projection are performed independently for each vehicle camera view to obtain the spatial point cloud corresponding to the sparse region. Based on the distribution scale of the spatial point cloud and the preset spatial range of the vehicle perception scene, the voxel resolution level adapted to the blank area is dynamically determined. For the voxel space covered by the spatial point cloud, voxel parameter reset, parent voxel iterative subdivision, or new voxel creation operations are performed, and all newly participating voxels are given high optimization priority and training protection period.

6. The method according to claim 5, characterized in that, Also includes: Based on the geometric surface of the real obstacles around the vehicle represented by the Mesh, calculate the distance from the center of each voxel to the surface. A voxel is deleted when it meets all of the following conditions: The distance is greater than the preset safe distance threshold; The training protection period has expired; it does not fall under the category of newly created or parameter reset voxels.

7. The method according to claim 6, characterized in that, in, The total number of vertices in the mesh is limited to a preset upper limit; Limit the number of sparse pixels sampled per single viewpoint to no more than the preset maximum number of sampled pixels; The total number of sub-voxels created in each round of voxel iteration subdivision is limited to a preset upper limit; The number of retries for redundant voxel deletion operations is limited to no more than 3.

8. A compensation device for a sparse voxel model, characterized in that, An in-vehicle perception system suitable for vehicles equipped with multi-view in-vehicle cameras, the device comprising: The recognition module is used to simultaneously acquire images from the vehicle's front, left, right and rear multi-view onboard cameras, render and extract triangular mesh models representing the physical environment around the vehicle, identify blank rendering areas, and generate corresponding area masks. The reconstruction module is used to reinitialize the identified rendering blank areas for each vehicle camera in order to reconstruct the voxel model of the rendering blank areas.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the compensation method for the sparse voxel model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the compensation method for the sparse voxel model as described in any one of claims 1 to 7.

Images