Lightweight three-dimensional display method based on multi-source data fusion

Abstract

This application relates to the technical field of 3D display and discloses a lightweight 3D display method based on multi-source data fusion, comprising: acquiring spatial observation data and target display parameters from different data sources; dividing a 3D spatial region into different spatial blocks based on the target display parameters, and identifying potential units of interest in the spatial blocks; calculating the spatial structure contour provided by each type of spatial observation data and the stability contribution of each type of spatial observation data in each potential unit of interest; selecting a fusion data source for each potential unit of interest based on the spatial structure contour and stability contribution; and performing lightweight 3D display on each potential unit of interest based on the corresponding fusion data source. This application can reduce the computational load and improve the real-time performance of 3D display while ensuring the stability of 3D structure representation.

Description

Technical Field

[0001] This application relates to the technical field of 3D display, specifically to a lightweight 3D display method based on multi-source data fusion. Background Technology

[0002] In recent years, 3D visualization technology has been widely used in fields such as autonomous driving, robot perception, digital twins, smart cities, and industrial inspection. In practical applications, to obtain more complete and accurate spatial information, multiple types of sensing devices are typically used simultaneously for data acquisition, and 3D scenes are constructed by fusing observation data from different data sources. Different data sources have their own advantages in terms of spatial information acquisition methods, observation range, and accuracy characteristics. Multi-source data fusion can, to some extent, compensate for the limitations of a single sensor, thereby improving the completeness and reliability of the 3D scene representation.

[0003] In existing technologies, multi-source 3D display systems typically integrate point cloud data or depth map data from different data sources, further constructing 3D point cloud models or mesh models, and then using graphics rendering technology to visualize the 3D scene. With the continuous improvement of sensor resolution and sampling density, the scale of 3D spatial data continues to grow, placing increasing computational pressure on 3D display systems in terms of data processing and real-time rendering. Especially in large-scale spatial scenes or when multiple data sources are running simultaneously, the amount of data the system needs to process is often large, thus placing higher demands on computing resources and system responsiveness. Furthermore, during dynamic perspective browsing or interactive 3D display, user observation behavior often exhibits significant locality, and existing systems often lack close coordination between data processing and display processes and actual observation needs. This leads to the system needing to process large amounts of data with low relevance to the current display requirements in some cases, thus affecting overall operational efficiency.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The technical problem to be solved by this application is to overcome the defects of the prior art and provide a lightweight 3D display method based on multi-source data fusion, which reduces the computational load and improves the real-time performance of 3D display while ensuring the stability of 3D structure expression.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution: A lightweight 3D display method based on multi-source data fusion includes the following steps: Acquire spatial observation data and target display parameters from different data sources; Based on the target display parameters, the three-dimensional spatial region is divided into different spatial blocks, and potential interest units in the spatial blocks are identified; Calculate the spatial structure profile provided by each type of space observation data and the stability contribution of each type of space observation data in each potential unit of interest. Based on the spatial structure profile and stability contribution, a fusion data source is selected for each potential unit of interest. Each potential unit of interest is displayed in a lightweight 3D format based on the corresponding fused data source.

[0007] As a preferred embodiment of the lightweight 3D display method based on multi-source data fusion described in this application, the method for dividing the 3D spatial region into different spatial blocks is as follows: S101: Take the minimum bounding box of the three-dimensional spatial region as the initial spatial block; divide the initial spatial block evenly into m spatial blocks; m is a positive integer; S102: Record the coordinates of the geometric center of each spatial block as the center coordinates of each spatial block; calculate the size threshold of each spatial block based on the center coordinates; S103: Calculate the maximum side length and number of 3D points for each spatial block; determine whether each spatial block meets the stopping segmentation condition based on the maximum side length, number of 3D points, and size threshold of each spatial block. S104: Divide each spatial block that does not meet the stopping segmentation condition into m spatial blocks; S105: Repeat S102 to S104 until all space blocks meet the stop partitioning condition.

[0008] As a preferred embodiment of the lightweight 3D display method based on multi-source data fusion described in this application, the target display parameters include viewpoint coordinates; the method for calculating the size threshold of any spatial block is as follows: calculate the distance between the geometric center of the spatial block and the viewpoint coordinates, which is taken as the observation distance of the spatial block; calculate the size threshold of the spatial block based on the observation distance, and the observation distance is positively correlated with the size threshold; The method for determining whether any spatial block satisfies the stopping partitioning condition is as follows: If the maximum side length of any spatial block is less than the corresponding size threshold, then the spatial block satisfies the stopping segmentation condition. Set a point count threshold; if the number of three-dimensional points in any spatial block is less than the point count threshold, then the spatial block meets the stop segmentation condition.

[0009] As a preferred embodiment of the lightweight 3D display method based on multi-source data fusion described in this application, the data sources include at least a LiDAR data source, a depth camera data source, and a multi-view camera data source; wherein, the spatial observation data from the LiDAR data source is 3D point cloud data; the spatial observation data from the depth camera data source is a first depth map; and the spatial observation data from the multi-view camera data source is a second depth map. In any potential unit of interest, the method for calculating the spatial structure contour provided by the first depth map is as follows: Project each pixel in the first depth map of the potential unit of interest onto the screen to obtain the first pixel region, and record the correspondence between each pixel in the first depth map and each pixel in the first pixel region. A first neighborhood is set for each pixel in the first depth map; based on the corresponding first neighborhood, it is determined whether there is a depth abrupt change for each pixel in the first depth map; each pixel in the first depth map with a depth abrupt change is marked as a abrupt pixel; the pixel corresponding to each abrupt pixel in the first pixel region is marked as a first edge point; the spatial structure contour provided by the first depth map includes all the first edge points in the first pixel region; The method for determining whether any pixel in the first depth map has a depth abrupt change is as follows: Mark any pixel in the first depth map as the target pixel; in the first depth map, calculate the difference between the first depth value of the target pixel and each pixel in its first neighborhood; if the absolute value of the difference between the first depth value of the target pixel and any pixel in its first neighborhood is greater than a preset depth difference threshold, then the target pixel has a depth abrupt change.

[0010] As a preferred embodiment of the lightweight 3D display method based on multi-source data fusion described in this application, the method for calculating the spatial structure contour provided by the second depth map in any potential unit of interest is as follows: Project each pixel in the second depth map of the potential unit of interest onto the screen to obtain the second pixel region, and record the correspondence between each pixel in the second depth map and each pixel in the second pixel region. Determine whether there is a depth abrupt change for each pixel in the second depth map; mark each pixel in the second depth map with a depth abrupt change as a potential edge point; Potential edge points are filtered based on the reference RGB image corresponding to the second depth map to obtain abrupt edge points; Each mutation edge point is marked as a second edge point in the second pixel region corresponding to the pixel point; the spatial structure contour provided by the second depth map includes all the second edge points in the second pixel region.

[0011] As a preferred embodiment of the lightweight 3D display method based on multi-source data fusion described in this application, the method for filtering potential edge points is as follows: Calculate the gradient magnitude of each pixel in the reference RGB image; mark each pixel in the reference RGB image whose gradient magnitude is greater than a preset gradient threshold as an edge pixel in the reference RGB image; mark the pixel corresponding to the position of any edge pixel in the reference RGB image in the second depth image as an edge pixel in the second depth image; For any potential edge point, if there is at least one edge pixel in the reference neighborhood of the second depth map, the potential edge point is marked as a mutation edge point.

[0012] As a preferred embodiment of the lightweight 3D display method based on multi-source data fusion described in this application, the method for calculating the spatial structure contour provided by the 3D point cloud data in any potential interest unit is as follows: project each 3D point in the potential interest unit onto the screen to obtain the third pixel region, and record the correspondence between each 3D point and each pixel in the third pixel region. A third neighborhood is set for each 3D point; a normal vector for each 3D point is constructed based on the third neighborhood of each 3D point; if the normal vector of any 3D point changes abruptly, the corresponding 3D point is marked as a mutated 3D point; the pixel point corresponding to each mutated 3D point in the third pixel region is marked as a third edge point; the spatial structure contour provided by the 3D point cloud data includes all third edge points in the third pixel region; The method for determining whether a normal vector mutation occurs at any 3D point is as follows: Mark any 3D point as the target 3D point; calculate the angle between the normal vector of the target 3D point and the normal vector of each 3D point in its third neighborhood, and record them as the direction mutation amount between the target 3D point and each 3D point in its third neighborhood; if the direction mutation amount between the target 3D point and any 3D point in its third neighborhood is greater than a preset mutation amount threshold, then the target 3D point has a normal vector mutation.

[0013] As a preferred embodiment of the lightweight 3D display method based on multi-source data fusion described in this application, the method for calculating the stability contribution of the first depth map in any potential interest unit is as follows: within the first pixel region, determine whether each first edge point is a stable edge point; calculate the ratio of the number of stable edge points in the first pixel region to the total number of first edge points as the stability contribution of the first depth map. The method for determining whether any first edge point is a stable edge point is as follows: Calculate the gradient direction for each first edge point; mark any first edge point as a target edge point; set a reference pixel window for the target edge point, and mark each first edge point in the reference pixel window other than the target edge point as a reference pixel point of the target edge point; Calculate the angle between the gradient direction of the target edge point and each reference pixel point respectively; if the angle between the gradient direction of any reference pixel point and the target edge point is less than a preset angle threshold, then mark the corresponding reference pixel point as a supporting pixel point of the target edge point. If the ratio of the number of supporting pixels of the target edge point to the total number of reference pixels is less than a preset ratio threshold, then the target edge point is a stable edge point.

[0014] As a preferred embodiment of the lightweight 3D display method based on multi-source data fusion described in this application, the method for selecting the fusion data source for any potential unit of interest is as follows: S201: Construct a data source queue and add each type of data source to the data source queue; S202: Mark the data source corresponding to the spatial observation data with the highest stability contribution as a fusion data source and remove it from the data source queue; S203: Determine whether the fused data source meets the contour coverage requirements; if not, return to execute S202. S204: Repeat S202-S203 until the merged data source meets the profile coverage requirements or the data source queue is empty.

[0015] As a preferred embodiment of the lightweight 3D display method based on multi-source data fusion described in this application, the method for determining whether the fused data source meets the contour coverage requirements is as follows: Calculate the union of the spatial structure contours corresponding to all data sources, and use it as the reference contour set; calculate the union of the spatial structure contours corresponding to all fused data sources, and use it as the fused contour set. The ratio of the number of pixels in the fused contour set to the number of pixels in the reference contour set is calculated as the contour coverage rate; if the contour coverage rate is greater than a preset coverage threshold, then the fused data source meets the contour coverage requirements. The method for lightweight 3D display of any potential unit of interest is as follows: the spatial observation data corresponding to each fusion data source of the potential unit of interest is fused to obtain the fused observation data of the potential unit of interest; the fused observation data is rendered and projected onto the screen to achieve lightweight 3D display.

[0016] Compared with the prior art, the beneficial effects achieved by this application are as follows: This application divides the three-dimensional spatial region into blocks and dynamically controls the granularity of the spatial blocks based on the viewpoint position. This enables the spatial data processing accuracy to match the observation requirements, thereby reducing unnecessary data processing overhead while ensuring display quality.

[0017] By analyzing the spatial structure contours provided by different data sources and calculating the stable contribution of each data source to the current spatial region, the effectiveness of different data sources in spatial structure representation can be objectively reflected. Based on this evaluation result, the data sources for fusion are selected so that the datasets participating in the fusion can more centrally represent scene structure information, thereby reducing the computational overhead caused by redundant data and improving the stability and consistency of 3D structure representation. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 A flowchart of a lightweight 3D display method based on multi-source data fusion provided for this application. Detailed Implementation

[0019] The technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of this application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0020] This embodiment introduces a lightweight 3D display method based on multi-source data fusion, referring to... Figure 1 The method includes the following steps: Acquire spatial observation data and target display parameters from different data sources; The data sources include at least a lidar data source, a depth camera data source, and a multi-view camera data source; wherein, the spatial observation data from the lidar data source is 3D point cloud data; the spatial observation data from the depth camera data source is a first depth map; and the spatial observation data from the multi-view camera data source is a second depth map. The three-dimensional point cloud data consists of the three-dimensional coordinates of three-dimensional points at different locations within a three-dimensional spatial region; the pixel value of each pixel in the first depth map represents the first depth value of the corresponding location within the three-dimensional spatial region; the pixel value of each pixel in the second depth map represents the second depth value of the corresponding location within the three-dimensional spatial region.

[0021] In one embodiment, the lidar data source is configured with a mechanical lidar or a solid-state lidar for scanning a three-dimensional spatial region to acquire three-dimensional point cloud data. The depth camera data source is configured with a structured light depth camera or a time-of-flight (ToF) depth camera; the first depth value represents the distance from the depth camera to the corresponding position in three-dimensional space. The multi-view camera data source is configured with multiple RGB cameras; the second depth value is obtained by: acquiring RGB images from different viewpoints, performing feature matching or pixel-level matching between the images to calculate the disparity value, and then converting the disparity value of each pixel segment into the corresponding second depth value according to the camera's baseline distance and focal length parameters. Optionally, a stereo matching algorithm such as block matching (BM) or semi-global matching (SGM) is used for pixel matching, and the pixel position offset of the same spatial point in different RGB images is calculated, i.e., the disparity value. There is a definite geometric relationship between the disparity value and the depth value. For a multi-view camera system with known camera baseline distance and focal length, the depth value of the target point is inversely proportional to the disparity value. Based on this correspondence, the disparity value can be converted into a second depth value, thereby generating a second depth map. The depth information obtained by multi-view RGB cameras through stereo matching can supplement the first depth value acquired by the depth camera, providing additional depth observations in areas where the depth camera has depth holes or where the depth accuracy is reduced at long distances, thereby improving the integrity of spatial observation data.

[0022] In existing technologies, 3D display methods based on multi-source data fusion often involve fusing multi-source point clouds or depth maps into a dense point cloud, then reconstructing a local mesh, and applying a weighted average to areas of conflict. During the display phase, techniques such as view frustum clipping and thinning are performed to reduce rendering load. The fusion computation boundary of this approach is determined by data validity, often fusing multi-source data from large areas, resulting in a significant amount of redundant information. Furthermore, users typically only view a small portion or frequently rotate their viewpoint rapidly, causing the computational load during the fusion phase to not converge with display requirements.

[0023] The target display parameters include the current viewpoint pose and view frustum parameters; The viewpoint pose includes viewpoint coordinates and viewpoint orientation; the view frustum parameters include field of view angle and clipping distance. The viewpoint coordinates are the current coordinates of the observation viewpoint within the three-dimensional spatial region; the viewpoint orientation is the current observation direction of the observation viewpoint, recorded via a direction vector; the three-dimensional spatial region refers to the scene space range requiring spatial observation and three-dimensional display, which can be described by the observation space range covered by various data sources when observing the scene; the observation viewpoint refers to the virtual observation position used in the three-dimensional display system to generate the current scene image, i.e., the user's viewing position within the three-dimensional spatial region. The system performs visibility judgment and projection rendering of structures within the three-dimensional spatial region based on the observation viewpoint.

[0024] The field of view is the maximum angle formed by the visible range on both sides of the viewpoint; the clipping distance is the visible distance set along the viewpoint direction, including near clipping distance and far clipping distance; for example, a field of view of 60 degrees means that the current visible range is a 60-degree fan-shaped area with the observation viewpoint as the vertex and the viewpoint direction as the center line. A near clipping distance of 0.5 meters and a far clipping distance of 80 meters mean that spatial structures within 0.5 meters or more than 80 meters from the observation viewpoint do not participate in the real-time projection display.

[0025] Based on the spatial observation data and target display parameters, the three-dimensional spatial region is divided into different spatial blocks, and potential units of interest in the spatial blocks are identified. The method for dividing a three-dimensional spatial region into different spatial blocks is as follows: S101: Take the minimum bounding box of the three-dimensional spatial region as the initial spatial block; divide the initial spatial block evenly into m spatial blocks; m is a positive integer; Optionally, m is set to 8, and each time the space block is divided, the cuboid-shaped space block is divided into 8 smaller cuboids on an equal basis.

[0026] S102: Record the coordinates of the geometric center of each spatial block as the center coordinates of each spatial block; calculate the size threshold of each spatial block based on the center coordinates; The method for calculating the size threshold of any spatial block is as follows: calculate the distance between the geometric center of the spatial block and the viewpoint coordinates, which is taken as the observation distance of the spatial block; calculate the size threshold of the spatial block based on the observation distance, and the observation distance is positively correlated with the size threshold.

[0027] Alternatively, the method for calculating the size threshold of the spatial block based on the observation distance is as follows: Set the maximum size threshold and the minimum size threshold; the formula for calculating the size threshold is as follows: ; Where L is the size threshold of the spatial block; d is the observation distance of the spatial block; The minimum size threshold to be set, for example, 0.5m or 1m; The maximum size threshold is set, for example, from 8m to 16m; The distance to the far clipping point is given. Based on the above calculations, the size threshold of any spatial block is between the maximum size threshold and the minimum size threshold, and the farther away from the viewpoint coordinates, the larger the size threshold.

[0028] S103: Calculate the maximum side length and number of 3D points for each spatial block; determine whether each spatial block meets the stopping segmentation condition based on the maximum side length, number of 3D points, and size threshold of each spatial block. If the maximum side length of any space block is less than the corresponding size threshold, the space block satisfies the stopping segmentation condition. In this embodiment, each space block is a cuboid, and the length of the longest side of the cuboid is taken as its maximum side length.

[0029] Set a point count threshold; if the number of three-dimensional points in any spatial block is less than the point count threshold, then the spatial block meets the stop segmentation condition.

[0030] In one embodiment, the number of 3D points in any spatial block is determined based on 3D point cloud data provided by a LiDAR data source; the coordinate range of each spatial block is recorded, and the number of 3D points falling within each spatial block is counted based on the 3D coordinates of each 3D point in the 3D point cloud data. The point count threshold can be set to 20 to 30.

[0031] S104: Divide each spatial block that does not meet the stopping segmentation condition into m spatial blocks; S105: Repeat S102 to S104 until all space blocks meet the stop partitioning condition.

[0032] Based on the above method of dividing spatial blocks, it is possible to achieve the following: relative to the observation viewpoint, the spatial region near the viewpoint is subdivided, while the spatial region far away is coarsely divided. That is, the closer to the observation viewpoint, the smaller the spatial block, and the higher the accuracy of its fusion calculation and projection display. At the same time, if the number of three-dimensional points of a spatial block is small, it will not be further subdivided to prevent continuous subdivision in areas with little geometric information, which would increase the amount of ineffective calculation.

[0033] In this embodiment, the spatial blocks are divided based on a size threshold determined by the observation viewpoint. As the observation viewpoint moves, the size distribution of the spatial blocks also changes. One feasible method for spatial block reconstruction is as follows: a viewpoint displacement threshold is set. When the movement distance of the observation viewpoint reaches the threshold, the spatial block division is re-executed. The viewpoint displacement threshold can be set to 1 to 2 times the size threshold of the smallest spatial block currently in use. This ensures that re-division is triggered only when the movement of the observation viewpoint is sufficient to change the distribution of nearby spatial blocks, while reconstruction is not performed during small-scale movements or minor viewpoint adjustments, thus avoiding the computational overhead caused by frequent re-division.

[0034] The method for identifying potential units of interest in a spatial block is as follows: Calculate the observation direction vector for each spatial block; the starting coordinates of the observation direction vector for any spatial block are the viewpoint coordinates, and the ending coordinates are the center coordinates of the spatial block. Calculate the angle between the observation direction vector of each spatial block and the viewpoint orientation, and use it as the observation deviation angle of each spatial block; Set an observation distance threshold and an observation angle threshold; if the observation distance of any spatial block is less than the observation distance threshold and the observation deviation angle is less than the observation angle threshold, then the corresponding spatial block is a potential unit of interest.

[0035] In one implementation, the observation distance threshold can be set to 0.7 to 1.0 times the far clipping distance; the observation angle threshold can be set to 0.2 to 0.4 times the field of view. Potential units of interest are spatial blocks expected to enter the observation field of view, i.e., areas expected to be projected onto the screen for display. These potential units of interest will subsequently undergo refined data fusion and visualization. For other spatial blocks that do not currently require projection, the existing fusion results can be maintained. In the early stages of the 3D display system's operation, fusion results have been established for the entire 3D spatial region, i.e., stable geometric representations generated after several fusion calculations, such as constructed point cloud fragments, generated local meshes, and calculated voxel models. When spatial blocks do not require projection, the existing fusion results can be maintained without real-time updates.

[0036] Calculate the spatial structure profile provided by each type of space observation data and the stability contribution of each type of space observation data in each potential unit of interest. In any potential cell of interest, the method for calculating the spatial structure profile provided by the first depth map is as follows: Project each pixel in the first depth map of the potential unit of interest onto the screen to obtain the first pixel region, and record the correspondence between each pixel in the first depth map and each pixel in the first pixel region. A first neighborhood is set for each pixel in the first depth map; based on the corresponding first neighborhood, it is determined whether there is a depth abrupt change for each pixel in the first depth map; each pixel in the first depth map with a depth abrupt change is marked as a abrupt pixel; the pixel corresponding to each abrupt pixel in the first pixel region is marked as a first edge point; the spatial structure contour provided by the first depth map includes all the first edge points in the first pixel region.

[0037] The method for determining whether any pixel in the first depth map has a sudden depth change is as follows: Mark any pixel in the first depth map as the target pixel; in the first depth map, calculate the difference between the first depth value of the target pixel and each pixel in its first neighborhood; if the absolute value of the difference between the first depth value of the target pixel and any pixel in its first neighborhood is greater than a preset depth difference threshold, then the target pixel has a sudden change in depth.

[0038] Those skilled in the art can set the specific value of the depth difference threshold based on actual needs. In one embodiment, using meters as the unit of pixel depth values ​​in the depth map, the depth difference threshold can be set to 0.02 to 0.1 meters; when any pixel experiences a sudden change in depth, it indicates the existence of the edge contour of a spatial structure at the corresponding location. Optionally, the first neighborhood of any pixel in the first depth map can be set to its 8-neighborhood in the first depth map.

[0039] In any potential cell of interest, the method for calculating the spatial structure profile provided by the second depth map is as follows: Project each pixel in the second depth map of the potential unit of interest onto the screen to obtain the second pixel region, and record the correspondence between each pixel in the second depth map and each pixel in the second pixel region. Determine whether there is a depth abrupt change for each pixel in the second depth map; mark each pixel in the second depth map with a depth abrupt change as a potential edge point; Potential edge points are filtered based on the reference RGB image corresponding to the second depth map to obtain abrupt edge points; Each mutation edge point is marked as a second edge point in the second pixel region corresponding to the pixel point; the spatial structure contour provided by the second depth map includes all the second edge points in the second pixel region.

[0040] The method for filtering potential edge points is as follows: Calculate the gradient magnitude of each pixel in the reference RGB image; mark each pixel in the reference RGB image whose gradient magnitude is greater than a preset gradient threshold as an edge pixel in the reference RGB image; mark the pixel corresponding to the position of any edge pixel in the reference RGB image in the second depth image as an edge pixel in the second depth image; For any potential edge point, if there is at least one edge pixel in the reference neighborhood of the second depth map, the potential edge point is marked as a mutation edge point.

[0041] Those skilled in the art can set the specific value of the gradient threshold based on actual needs. In one implementation, the reference RGB image is first converted to a grayscale image, and then the gradient of the grayscale value of each pixel is calculated using the Sobel operator to obtain the gradient magnitude. The gradient threshold is set to 30. When the gradient magnitude of any pixel is greater than the gradient threshold, a color abrupt change occurs at that location, corresponding to the existence of an edge contour structure.

[0042] Optionally, the reference neighborhood of any potential edge point in the second depth map is either its 8-neighborhood or its 24-neighborhood. When an edge pixel exists in the reference neighborhood, it indicates that the potential edge point has image structure support and can be retained as a valid contour, i.e., marked as abrupt edge point. This can effectively eliminate isolated depth abrupt changes caused by stereo matching errors, i.e., potential edge points that do not correspond to real structural edges caused by stereo matching errors or noise.

[0043] In this scheme, the second depth map is generated from RGB images from different perspectives provided by the multi-view camera data source; its reference RGB image is the RGB image from a certain perspective; for example, in a stereo system, the RGB image captured by the left camera is used as the reference RGB image; the second depth map and the reference RGB image have the same size and the pixels correspond one-to-one.

[0044] Similar to the first depth map, the method for determining whether any pixel in the second depth map has a depth abrupt change is as follows: A second neighborhood is set for each pixel in the second depth map; in the second depth map, if the absolute value of the difference between the second depth value of any pixel and any pixel in its second neighborhood is greater than the depth difference threshold, then the pixel has a depth abrupt change. Similarly, the second neighborhood of any pixel in the second depth map can be set to its 8-neighborhood in the second depth map.

[0045] In any potential unit of interest, the method for calculating the spatial structure contour provided by the 3D point cloud data is as follows: Project each 3D point in the potential unit of interest onto the screen to obtain the third pixel region, and record the correspondence between each 3D point and each pixel in the third pixel region. A third neighborhood is set for each 3D point; based on the third neighborhood of each 3D point, a normal vector for each 3D point is constructed; if any 3D point experiences a sudden change in its normal vector, the corresponding 3D point is marked as a mutated 3D point; the pixel point corresponding to each mutated 3D point in the third pixel region is marked as a third edge point; the spatial structure contour provided by the 3D point cloud data includes all third edge points in the third pixel region.

[0046] The method for determining whether a normal vector of any 3D point has a sudden change is as follows: Mark any 3D point as the target 3D point; calculate the angle between the normal vector of the target 3D point and the normal vector of each 3D point in the third neighborhood of the target 3D point, and record them as the direction mutation amount between the target 3D point and each 3D point in the third neighborhood; if the direction mutation amount between the target 3D point and any 3D point in the third neighborhood is greater than the preset mutation amount threshold, then the target 3D point has a normal vector mutation.

[0047] Those skilled in the art can set specific values ​​for the third neighborhood and the mutation threshold based on actual needs. In one embodiment, the mutation threshold is set to 30 degrees, and the third neighborhood of any 3D point includes the k other 3D points closest to its spatial location, where k is between 6 and 12. When a 3D point experiences a sudden change in its normal vector, it indicates a significant change in the local surface orientation at that location, typically corresponding to the geometric boundary or contour position of an object. Optionally, the normal vector of the target 3D point can be constructed as follows: based on the 3D coordinates of the target 3D point and each 3D point in its third neighborhood, a plane is fitted using the least squares method or principal component analysis; the normal vector of this plane is the normal vector of the target 3D point.

[0048] It should be noted that the projection of any spatial observation data from a potential unit of interest onto the screen is performed as follows: a view transformation matrix and a projection transformation matrix are constructed using the viewpoint pose and view frustum parameters. The view transformation matrix is ​​a coordinate transformation matrix determined by the viewpoint coordinates and viewpoint orientation, used to transform points in three-dimensional space to a viewpoint coordinate system with the observation viewpoint as the origin. The projection transformation matrix is ​​a matrix determined by the field of view angle and clipping distance, used to map the three-dimensional coordinates in the viewpoint coordinate system to the screen space, thereby obtaining the corresponding screen projection position. For 3D point cloud data, the 3D coordinates of each 3D point are first transformed to the view coordinate system using a view transformation matrix, and then the 3D coordinates are mapped to the screen space using a projection transformation matrix to obtain the third pixel region. For the first depth map or the second depth map, the pixel coordinates and corresponding depth values ​​are first transformed into 3D spatial coordinates according to the camera intrinsic parameters (including the horizontal focal length, the vertical focal length, and the principal point coordinates) to obtain the set of 3D points corresponding to the depth map. Then, in the same way as for the 3D point cloud data, these 3D points are projected to the screen space using a view transformation matrix and a projection transformation matrix.

[0049] In any potential unit of interest, the method for calculating the stability contribution of the first depth map is as follows: within the first pixel region, determine whether each first edge point is a stable edge point; calculate the ratio of the number of stable edge points in the first pixel region to the total number of first edge points as the stability contribution of the first depth map; The method for determining whether any first edge point is a stable edge point is as follows: Calculate the gradient direction for each first edge point; Mark any first edge point as a target edge point; set a reference pixel window for the target edge point, and mark each first edge point in the reference pixel window other than the target edge point as a reference pixel point of the target edge point; Calculate the angle between the gradient direction of the target edge point and each reference pixel point respectively; if the angle between the gradient direction of any reference pixel point and the target edge point is less than a preset angle threshold, then mark the corresponding reference pixel point as a supporting pixel point of the target edge point. If the ratio of the number of supporting pixels of the target edge point to the total number of reference pixels is less than a preset ratio threshold, then the target edge point is a stable edge point.

[0050] Those skilled in the art can set the reference pixel window, angle threshold, and scale threshold based on actual needs. In one embodiment, the reference pixel window for the target edge point is its 8-neighborhood or 24-neighborhood; the angle threshold is 20 degrees, and the scale threshold is 0.7. Optionally, the gradient direction of each first edge point is calculated using the Sobel operator; when more than 70% of the reference pixels are close to the gradient direction between the target edge points, it indicates that the contour direction near the target edge point is consistent; otherwise, the contour distribution in the vicinity is messy, and the risk of the target edge point being caused by noise or false detection is high.

[0051] It should be noted that in this embodiment, the calculation of the stability contribution of the second depth map or 3D point cloud data is the same as that of the first depth map. That is, within the second pixel region or the third pixel region, it is determined whether each second edge point or third edge point is a stable edge point. The ratio of the number of stable edge points to the total number of second edge points or third edge points is calculated as the corresponding stability contribution. The method for determining whether a second edge point or third edge point is a stable edge point is also the same as that for the first edge point, and will not be repeated here. The larger the stability contribution, the more stable the contour representation of the corresponding spatial observation data on the screen; the smaller the contribution, the more noisy the contour of the spatial observation data, which has an adverse effect on the stability and accuracy of the projection display.

[0052] Based on the spatial structure profile and stability contribution, a fusion data source is selected for each potential unit of interest. The method for selecting fusion data sources for any potential unit of interest is as follows: S201: Construct a data source queue and add each type of data source to the data source queue; S202: Mark the data source corresponding to the spatial observation data with the highest stability contribution as a fusion data source and remove it from the data source queue; S203: Determine whether the fused data source meets the contour coverage requirements; if not, return to execute S202. S204: Repeat S202-S203 until the merged data source meets the profile coverage requirements or the data source queue is empty.

[0053] The method for determining whether the fused data source meets the contour coverage requirements is as follows: Calculate the union of the spatial structure contours corresponding to all data sources, and use it as the reference contour set; calculate the union of the spatial structure contours corresponding to all fused data sources, and use it as the fused contour set. The ratio of the number of pixels in the fused contour set to the number of pixels in the reference contour set is calculated as the contour coverage rate; if the contour coverage rate is greater than a preset coverage threshold, then the fused data source meets the contour coverage requirements.

[0054] Those skilled in the art can set the specific value of the coverage threshold based on actual needs. In one embodiment, the coverage threshold is 0.8.

[0055] This embodiment introduces three data sources; in actual operation, those skilled in the art can set more data sources based on actual needs; by executing S201 to S204, the minimum set of data sources that can fully express the three-dimensional spatial contour can be selected from three or more data sources. When the contour coverage is greater than the coverage threshold, it indicates that the currently selected fusion data source can express most of the structural information of the potential unit of interest. At this time, the contours provided by the other data sources are mainly redundant information. Otherwise, other data sources need to be introduced as fusion data sources to supplement the information of the projected structure of the potential unit of interest.

[0056] Each potential unit of interest is displayed in a lightweight 3D format based on the corresponding fused data source.

[0057] The method for lightweight 3D visualization of any potential unit of interest is as follows: The spatial observation data corresponding to each fusion data source of the potential unit of interest is fused to obtain the fused observation data of the potential unit of interest; the fused observation data is then rendered and projected onto the screen to achieve lightweight 3D display.

[0058] Data fusion of spatial observation data from different data sources refers to point cloud fusion. Taking a first depth map and 3D point cloud data as an example, as mentioned earlier, firstly, the pixel coordinates and corresponding depth values ​​of the first depth map are converted into 3D spatial coordinates based on camera intrinsic parameters, resulting in a set of 3D points corresponding to the depth map. This set of 3D points is then superimposed with the point cloud corresponding to the 3D point cloud data. The superimposed point cloud undergoes deduplication and downsampling of 3D points to reduce the number of 3D points, resulting in fused observation data. It should be noted that if the camera coordinate system referenced by the first depth map is different from the radar coordinate system referenced by the 3D point cloud data, coordinate unification is required, for example, transforming the coordinates of the 3D point set corresponding to the depth map from the camera coordinate system to the radar coordinate system.

[0059] In practice, the 3D display system automatically triggers the rendering and projection of each potential unit of interest (TUI) based on the viewpoint pose and frustum parameters. Specifically, a frustum is constructed using the viewpoint pose and frustum parameters. If a TUI intersects with the frustum, rendering of it is triggered. For example, the fused observation data is rendered as a point cloud, mesh, or voxel and projected onto the screen to achieve real-time 3D visualization. Taking mesh rendering as an example, after obtaining the fused observation data, the point cloud is subjected to neighborhood connectivity or triangulation. For instance, triangular patches are created based on the spatial adjacency relationships between points, thus forming a local triangular mesh. The resulting mesh can represent the surface structure more continuously than a discrete point cloud, making it suitable for 3D display. Through the implementation of this solution, the computational load in the 3D display process can dynamically change with the user's display behavior and the quality of multi-source data, achieving lightweight 3D display while ensuring display stability.

[0060] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of protection of this application, and these forms are all within the protection scope of this application.

Claims

1. A lightweight 3D display method based on multi-source data fusion, characterized in that: include Acquire spatial observation data and target display parameters from different data sources; Based on the target display parameters, the three-dimensional spatial region is divided into different spatial blocks, and potential interest units in the spatial blocks are identified; Calculate the spatial structure profile provided by each type of space observation data and the stability contribution of each type of space observation data in each potential unit of interest. Based on the spatial structure profile and stability contribution, a fusion data source is selected for each potential unit of interest. Each potential unit of interest is displayed in a lightweight 3D format based on the corresponding fused data source.

2. The lightweight 3D display method based on multi-source data fusion as described in claim 1, characterized in that: The method for dividing a three-dimensional spatial region into different spatial blocks is as follows: S101: Take the minimum bounding box of the three-dimensional spatial region as the initial spatial block; divide the initial spatial block evenly into m spatial blocks; m is a positive integer; S102: Record the coordinates of the geometric center of each spatial block as the center coordinates of each spatial block; calculate the size threshold of each spatial block based on the center coordinates; S103: Calculate the maximum side length and the number of 3D points for each spatial block; Determine whether each spatial block meets the stopping segmentation condition based on the maximum side length, number of 3D points, and size threshold of each spatial block. S104: Divide each spatial block that does not meet the stopping segmentation condition into m spatial blocks; S105: Repeat S102 to S104 until all space blocks meet the stop partitioning condition.

3. The lightweight 3D display method based on multi-source data fusion as described in claim 2, characterized in that: The target display parameters include viewpoint coordinates; the method for calculating the size threshold of any spatial block is as follows: calculate the distance between the geometric center of the spatial block and the viewpoint coordinates, which is taken as the observation distance of the spatial block; calculate the size threshold of the spatial block based on the observation distance, and the observation distance is positively correlated with the size threshold; The method for determining whether any spatial block satisfies the stopping partitioning condition is as follows: If the maximum side length of any spatial block is less than the corresponding size threshold, then the spatial block satisfies the stopping segmentation condition. Set a point count threshold; if the number of three-dimensional points in any spatial block is less than the point count threshold, then the spatial block meets the stop segmentation condition.

4. The lightweight 3D display method based on multi-source data fusion as described in claim 3, characterized in that: The data source includes at least a lidar data source, a depth camera data source, and a multi-view camera data source; wherein, the spatial observation data from the lidar data source is 3D point cloud data; the spatial observation data from the depth camera data source is a first depth map; and the spatial observation data from the multi-view camera data source is a second depth map. In any potential unit of interest, the method for calculating the spatial structure contour provided by the first depth map is as follows: Project each pixel in the first depth map of the potential unit of interest onto the screen to obtain the first pixel region, and record the correspondence between each pixel in the first depth map and each pixel in the first pixel region. A first neighborhood is set for each pixel in the first depth map; based on the corresponding first neighborhood, it is determined whether there is a depth abrupt change for each pixel in the first depth map; each pixel in the first depth map with a depth abrupt change is marked as a abrupt pixel; the pixel corresponding to each abrupt pixel in the first pixel region is marked as a first edge point; the spatial structure contour provided by the first depth map includes all the first edge points in the first pixel region; The method for determining whether any pixel in the first depth map has a depth abrupt change is as follows: Mark any pixel in the first depth map as the target pixel; in the first depth map, calculate the difference between the first depth value of the target pixel and each pixel in its first neighborhood; if the absolute value of the difference between the first depth value of the target pixel and any pixel in its first neighborhood is greater than a preset depth difference threshold, then the target pixel has a depth abrupt change.

5. The lightweight 3D display method based on multi-source data fusion as described in claim 4, characterized in that: In any potential cell of interest, the method for calculating the spatial structure profile provided by the second depth map is as follows: Project each pixel in the second depth map of the potential unit of interest onto the screen to obtain the second pixel region, and record the correspondence between each pixel in the second depth map and each pixel in the second pixel region. Determine whether there is a sudden depth change for each pixel in the second depth map; In the second depth map, each pixel with a depth abrupt change is marked as a potential edge point; Potential edge points are filtered based on the reference RGB image corresponding to the second depth map to obtain abrupt edge points; Each mutation edge point is marked as a second edge point in the second pixel region corresponding to the pixel point; the spatial structure contour provided by the second depth map includes all the second edge points in the second pixel region.

6. The lightweight 3D display method based on multi-source data fusion as described in claim 5, characterized in that: The method for filtering potential edge points is as follows: Calculate the gradient magnitude of each pixel in the reference RGB image; mark each pixel in the reference RGB image whose gradient magnitude is greater than a preset gradient threshold as an edge pixel in the reference RGB image; mark the pixel corresponding to the position of any edge pixel in the reference RGB image in the second depth image as an edge pixel in the second depth image; For any potential edge point, if at least one edge pixel exists in the reference neighborhood of the second depth map, the potential edge point is marked as a mutation edge point.

7. The lightweight 3D display method based on multi-source data fusion as described in claim 6, characterized in that: In any potential unit of interest, the method for calculating the spatial structure contour provided by the 3D point cloud data is as follows: Project each 3D point in the potential unit of interest onto the screen to obtain the third pixel region, and record the correspondence between each 3D point and each pixel in the third pixel region. A third neighborhood is set for each 3D point; a normal vector for each 3D point is constructed based on the third neighborhood of each 3D point; if the normal vector of any 3D point changes abruptly, the corresponding 3D point is marked as a mutated 3D point; the pixel point corresponding to each mutated 3D point in the third pixel region is marked as a third edge point; the spatial structure contour provided by the 3D point cloud data includes all third edge points in the third pixel region; The method for determining whether a normal vector mutation occurs at any 3D point is as follows: Mark any 3D point as the target 3D point; calculate the angle between the normal vector of the target 3D point and the normal vector of each 3D point in its third neighborhood, and record them as the direction mutation amount between the target 3D point and each 3D point in its third neighborhood; if the direction mutation amount between the target 3D point and any 3D point in its third neighborhood is greater than a preset mutation amount threshold, then the target 3D point has a normal vector mutation.

8. The lightweight 3D display method based on multi-source data fusion as described in claim 7, characterized in that: In any potential unit of interest, the method for calculating the stability contribution of the first depth map is as follows: within the first pixel region, determine whether each first edge point is a stable edge point; calculate the ratio of the number of stable edge points in the first pixel region to the total number of first edge points as the stability contribution of the first depth map; The method for determining whether any first edge point is a stable edge point is as follows: Calculate the gradient direction for each first edge point; mark any first edge point as a target edge point; set a reference pixel window for the target edge point, and mark each first edge point in the reference pixel window other than the target edge point as a reference pixel point of the target edge point; Calculate the angle between the gradient direction of the target edge point and each reference pixel point respectively; if the angle between the gradient direction of any reference pixel point and the target edge point is less than a preset angle threshold, then mark the corresponding reference pixel point as a supporting pixel point of the target edge point. If the ratio of the number of supporting pixels of the target edge point to the total number of reference pixels is less than a preset ratio threshold, then the target edge point is a stable edge point.

9. The lightweight 3D display method based on multi-source data fusion as described in claim 8, characterized in that: The method for selecting fusion data sources for any potential unit of interest is as follows: S201: Construct a data source queue and add each type of data source to the data source queue; S202: Mark the data source corresponding to the spatial observation data with the highest stability contribution as a fusion data source and remove it from the data source queue; S203: Determine whether the fused data source meets the contour coverage requirements; if not, return to execute S202. S204: Repeat S202-S203 until the merged data source meets the profile coverage requirements or the data source queue is empty.

10. The lightweight 3D display method based on multi-source data fusion as described in claim 9, characterized in that: The method for determining whether the merged data source meets the contour coverage requirements is as follows: Calculate the union of the spatial structure contours corresponding to all data sources, and use it as the reference contour set; calculate the union of the spatial structure contours corresponding to all fused data sources, and use it as the fused contour set. The ratio of the number of pixels in the fused contour set to the number of pixels in the reference contour set is calculated as the contour coverage rate; if the contour coverage rate is greater than a preset coverage threshold, then the fused data source meets the contour coverage requirements. The method for lightweight 3D display of any potential unit of interest is as follows: the spatial observation data corresponding to each fusion data source of the potential unit of interest is fused to obtain the fused observation data of the potential unit of interest; the fused observation data is rendered and projected onto the screen to achieve lightweight 3D display.

Images