Point cloud coloring processing method, device and electronic equipment

Abstract

Embodiments of the present disclosure disclose a point cloud coloring processing method and device and electronic equipment, wherein the method comprises: rendering a point cloud of a visible voxel under a first view angle to obtain a sparse depth map under the first view angle; searching for a depth value of each pixel without an effective depth in the sparse depth map by using a preset depth value searching method, filling pixels in the sparse depth map based on a depth value searching result to obtain a dense depth map under the first view angle; determining a risk coloring area in the dense depth map based on a data relationship between a depth value of each pixel in the dense depth map and a minimum depth value of a pixel within a preset range near each pixel in the dense depth map; and coloring pixels in the dense depth map except pixels in the risk coloring area. Embodiments of the present disclosure can effectively reduce the probability of foreground and background coloring misalignment.

Description

Technical Field

[0001] This disclosure relates to three-dimensional reconstruction technology and point cloud processing technology, and in particular to a point cloud coloring processing method, apparatus and electronic device. Background Technology

[0002] In 3D scene reconstruction, it is usually necessary to perform point cloud shading on objects in the 3D scene. During point cloud shading, there may be a misalignment of foreground and background shading near occluded areas. For example, when a utility pole occludes the ground, the color belonging to the utility pole may be shading onto the nearby ground, or when a tree trunk occludes the ground, the color belonging to the tree trunk may be shading onto the nearby ground, and so on.

[0003] How to reduce the probability of foreground and background color misalignment during 3D reconstruction and accurately perform point cloud coloring is an urgent problem to be solved. Summary of the Invention

[0004] This disclosure provides a point cloud coloring processing method, apparatus, and electronic device to solve the above-mentioned problems.

[0005] A first aspect of this disclosure provides a point cloud coloring processing method, including:

[0006] Render the point cloud of visible voxels in the first view to obtain a sparse depth map in the first view.

[0007] Using a preset depth value search method, depth values ​​are searched for each pixel in the sparse depth map that has no effective depth. Based on the depth value search results, the sparse depth map is filled with pixels to obtain a dense depth map under the first view.

[0008] Based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range near each pixel in the dense depth map, the risk-colored region in the dense depth map is determined.

[0009] The pixels in the dense depth map, excluding the risk-colored region, are colored.

[0010] In some embodiments of this disclosure, rendering the point cloud of visible voxels from a first viewpoint to obtain a sparse depth map from the first viewpoint includes:

[0011] Voxel visibility detection is performed on the point cloud data of the voxels to be shaded from a first-view perspective to obtain the point cloud set of visible voxels in the first-view perspective.

[0012] The point cloud set of visible voxels from the first viewpoint is rendered to obtain the sparse depth map.

[0013] In some embodiments of this disclosure, the step of performing voxel visibility detection on the point cloud data of the voxels to be shaded from a first viewpoint to obtain a point cloud set of visible voxels from the first viewpoint includes:

[0014] Based on the point cloud data, a voxel structure is constructed to obtain the hierarchical bounding volume structure of the voxel to be shaded;

[0015] Rays are emitted from the camera center in the first viewpoint toward the centers of each voxel in the hierarchical enclosing structure. The first depth of each voxel is determined based on the distance from the camera center to the center of each voxel, and the second depth of each voxel is determined based on the distance from the camera center to the intersection of the hierarchical enclosing structure.

[0016] Based on the data relationship between the first depth and the second depth of each voxel, the visible voxels under the first viewpoint are determined.

[0017] Obtain the point cloud set corresponding to the visible voxels in the first view.

[0018] In some embodiments of this disclosure, the step of using a preset depth value search method to perform depth value search on each pixel in the sparse depth map that has no effective depth, and then filling the sparse depth map with pixels based on the depth value search results to obtain a dense depth map under the first viewpoint includes:

[0019] Using the preset depth value search method, a depth value search is performed on each pixel in the sparse depth map that has no effective depth, and the depth value of each pixel is determined based on the depth value search results.

[0020] The sparse depth map is filled with pixels based on the depth values ​​of each pixel to obtain a dense depth map from the first viewpoint.

[0021] In some embodiments of this disclosure, the preset depth value search method is a breadth-first search method.

[0022] In some embodiments of this disclosure, determining the risk-colored region in the dense depth map based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range near each pixel in the dense depth map includes:

[0023] Obtain the depth value of a pixel in the dense depth map, and obtain the minimum depth value of pixels within a preset range near the pixel;

[0024] In response to a situation where the ratio between the depth value of the pixel and the minimum depth value of pixels within a preset range near the pixel is less than a preset threshold, the pixel is identified as a pixel within the risk-colored area.

[0025] In some embodiments of this disclosure, after coloring the pixels in the dense depth map excluding the risk-colored region, the method further includes:

[0026] In response to the presence of at least one pixel within the risky coloring region that belongs to the safe coloring region from a second viewpoint, the at least one pixel is colored from the second viewpoint.

[0027] A second aspect of this disclosure provides a point cloud coloring processing apparatus, comprising:

[0028] The sparse depth map acquisition module is used to render the point cloud of visible voxels in the first view to obtain the sparse depth map in the first view.

[0029] The dense depth map acquisition module is used to perform depth value search on each pixel in the sparse depth map that has no effective depth using a preset depth value search method, and fill the sparse depth map with pixels based on the depth value search results to obtain a dense depth map under the first view.

[0030] The risk-colored region determination module is used to determine the risk-colored region in the dense depth map based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range near each pixel in the dense depth map.

[0031] A coloring module is used to color the pixels in the dense depth map except for the risk coloring region.

[0032] In some embodiments of this disclosure, the sparse depth map acquisition module is used to perform voxel visibility detection on the point cloud data of the voxels to be shaded from a first viewpoint to obtain a point cloud set of visible voxels from the first viewpoint; the sparse depth map acquisition module is also used to render the point cloud set of visible voxels from the first viewpoint to obtain the sparse depth map.

[0033] In some embodiments of this disclosure, the sparse depth map acquisition module is used to construct a voxel structure based on the point cloud data to obtain a hierarchical bounding volume structure of the voxel to be colored; the sparse depth map acquisition module is further used to emit rays from the camera center in the first viewpoint to the centers of each voxel in the hierarchical bounding volume structure, determine a first depth of each voxel based on the distance from the camera center to the centers of each voxel, and determine a second depth of each voxel based on the distance from the camera center to the intersection of the hierarchical bounding volume structure; the sparse depth map acquisition module is further used to determine the visible voxels in the first viewpoint based on the data relationship between the first depth and the second depth of each voxel; the sparse depth map acquisition module is further used to obtain the point cloud set corresponding to the visible voxels in the first viewpoint.

[0034] In some embodiments of this disclosure, the dense depth map acquisition module is used to perform depth value search on each pixel in the sparse depth map that has no effective depth using the preset depth value search method, and determine the depth value of each pixel based on the depth value search results; the dense depth map acquisition module is also used to fill the sparse depth map with pixels based on the depth values ​​of each pixel to obtain a dense depth map under the first viewpoint.

[0035] In some embodiments of this disclosure, the preset depth value search method is a breadth-first search method.

[0036] In some embodiments of this disclosure, the risk-colored region determination module is used to obtain the depth value of a pixel in the dense depth map and obtain the minimum depth value of pixels within a preset range near the pixel; the risk-colored region determination module is further used to determine the pixel as a pixel within the risk-colored region in response to the ratio between the depth value of the pixel and the minimum depth value of pixels within the preset range near the pixel being less than a preset threshold.

[0037] In some embodiments of this disclosure, the coloring module is further configured to color the at least one pixel in the second view in response to the presence of at least one pixel in the risky coloring region belonging to the safe coloring region in the second view.

[0038] A third aspect of this disclosure provides an electronic device, comprising:

[0039] Memory, used to store computer program products;

[0040] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in the first aspect above.

[0041] A fourth aspect of this disclosure provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the method described in the first aspect above.

[0042] The point cloud shading processing method, apparatus, and electronic device of this disclosure, since the occlusion relationships between objects differ from different viewpoints, renders the point cloud of visible voxels from a certain viewpoint to obtain a sparse depth map for that viewpoint. Then, pixels without effective depth in the sparse depth map are filled to obtain a dense depth map. Based on the data relationship between the depth values ​​of each pixel in the dense depth map and the minimum depth value of pixels within a preset range nearby, risky shading regions in the dense depth map are determined. Shading is applied only to pixels outside these risky shading regions, effectively reducing the probability of foreground-background shading misalignment. Furthermore, since a risky shading region in one viewpoint may not be a risky shading region in other viewpoints, multi-view rendering can significantly reduce the probability of foreground-background shading misalignment.

[0043] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0044] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0045] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0046] Figure 1 This is a flowchart illustrating a point cloud coloring processing method in one embodiment of the present disclosure;

[0047] Figure 2 This is a camera image taken of a first-person view scene to be rendered, which is one example disclosed in this publication;

[0048] Figure 3 yes Figure 2 The corresponding sparse depth map;

[0049] Figure 4 This is a sparse depth map in one example of the present disclosure;

[0050] Figure 5 yes Figure 4 The corresponding dense depth map;

[0051] Figure 6 This is another example of a camera image taken of a first-person view scene to be rendered, as disclosed in this disclosure.

[0052] Figure 7 yes Figure 6 A schematic diagram of the corresponding risk-colored areas;

[0053] Figure 8 This is an example of the effect of point cloud coloring using point cloud coloring methods from related technologies;

[0054] Figure 9 This is a diagram showing the effect of point cloud coloring using the point cloud coloring method of this embodiment.

[0055] Figure 10 This is a flowchart illustrating step S1 in some embodiments of this disclosure;

[0056] Figure 11 This is a flowchart illustrating step S1-1 in some embodiments of this disclosure;

[0057] Figure 12 This is a flowchart illustrating step S2 in some embodiments of this disclosure;

[0058] Figure 13 This is a flowchart illustrating step S3 in some embodiments of this disclosure;

[0059] Figure 14 This is a structural block diagram of a point cloud coloring processing apparatus in one embodiment of the present disclosure;

[0060] Figure 15 This is a structural block diagram of an electronic device in one embodiment of the present disclosure. Detailed Implementation

[0061] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0062] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0063] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0064] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0065] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0066] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0067] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0068] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0069] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0070] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0071] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0072] Figure 1 This is a flowchart illustrating a point cloud coloring processing method in one embodiment of this disclosure. Figure 1 As shown, point cloud coloring methods include:

[0073] S1: Render the point cloud of visible voxels in the first-view perspective to obtain a sparse depth map in the first-view perspective.

[0074] A camera can be used to photograph the scene to be reconstructed from multiple perspectives, obtaining images of the scene from multiple viewpoints. Point cloud data of each voxel in the scene can then be extracted from these images. Alternatively, radar can be used to scan the scene to be rendered from multiple different locations to obtain point cloud data of each voxel in the scene. Here, a voxel is short for volume element; a solid containing voxels can be represented through stereo rendering or by extracting polygonal isosurfaces from a given threshold contour.

[0075] It should be noted that the point cloud data of each voxel in the image obtained by processing the image captured by the camera, and the point cloud data of each voxel of the scanned object obtained by processing the point cloud data obtained by radar scanning, are all technical means well known to those skilled in the art, and will not be described in detail.

[0076] Given a first-person viewpoint, the point cloud data of each voxel in the scene to be rendered is read from the point cloud data of each voxel in the first-person viewpoint. The first-person viewpoint can be the current viewpoint, meaning that the point cloud shading method of this embodiment can be applied to real-time point cloud shading.

[0077] Rendering all point clouds directly to the first-person viewpoint to obtain a sparse depth map is inefficient. Therefore, a rendering model based on 3D reconstruction technology can be used to render the point cloud of visible voxels in the first-person viewpoint to obtain a sparse depth map. The time complexity of directly rendering all point clouds to the current viewpoint to obtain a sparse depth map is much greater than the time complexity of rendering the point cloud of visible voxels in the first-person viewpoint.

[0078] Figure 2 This is a camera image taken of a first-person view scene to be rendered, which is one example disclosed in this publication. Figure 3 yes Figure 2 The corresponding sparse depth map. For example... Figure 2 and Figure 3 As shown, for a scene where leaves are in the foreground and buildings or the sky are in the background from a first-person perspective, a corresponding sparse depth map can be rendered. The sparse depth map contains pixel regions with no depth information (i.e., black image regions).

[0079] S2: Using a preset depth value search method, perform depth value search on each pixel in the sparse depth map that has no effective depth, and fill the sparse depth map with pixels based on the depth value search results to obtain a dense depth map in the first view.

[0080] In this context, pixels without effective depth in the sparse depth map are pixels located within the image region (e.g., a rectangular region) of the sparse depth map that lack depth information.

[0081] Using a preset depth value search method, for each pixel in the sparse depth map that has no effective depth, the method searches for pixels with effective depth within a preset range near that pixel. The depth value of each pixel in the sparse depth map that has no effective depth is then determined based on the depth values ​​of the searched pixels. The preset depth value search method can include Depth-First Search (DFS), Breadth-First Search (BFS), Best-First Search (BFS), and so on.

[0082] Based on the depth value determined for each pixel in the sparse depth map that originally had no effective depth, the dense depth map in the first-view perspective is obtained by filling in all pixels in the sparse depth map that originally had no effective depth.

[0083] Figure 4 This is a sparse depth map in one example disclosed in this publication. Figure 5 yes Figure 4 The corresponding dense depth map. For example... Figure 4 and Figure 5 As shown, a dense depth map can be obtained by filling the pixel regions (i.e., black image regions) in the sparse depth map with no effective depth.

[0084] S3: Based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range near each pixel in the dense depth map, determine the risk-colored area in the dense depth map.

[0085] For each pixel in the dense depth map, obtain the depth value of that pixel, and then obtain the minimum depth value of pixels within a preset range near that pixel. The preset range can be an N*M pixel region, where N and M are both integers greater than 1.

[0086] After obtaining the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range near that pixel, the data relationship between the two depth values ​​can be calculated using a preset data relationship calculation formula.

[0087] Based on the preset data relationship-risk coloring region correspondence, determine whether there are foreground pixels in the preset pixel region near each pixel in the dense depth map. The region composed of all pixels in the preset pixel region near which there are foreground pixels is determined as the risk coloring region in the dense depth map, and the pixel region in the dense depth map other than the risk coloring region is determined as the safe coloring region.

[0088] Figure 6 This is another example of a camera image taken of a first-person view scene to be rendered, as disclosed in this disclosure. Figure 7 yes Figure 6 A schematic diagram of the corresponding risk-colored areas. (Example) Figure 6 and Figure 7 As shown, the risk-colored region (i.e. gray image region) in the dense depth map is determined by the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of the pixels within a preset range near each pixel in the dense depth map.

[0089] S4: Color the pixels in the dense depth map except for the risky coloring region.

[0090] By using a rendering model to color pixels in dense depth maps except for those in risky coloring regions (i.e., pixels in safe coloring regions), the chance of foreground and background color misalignment can be effectively reduced.

[0091] Figure 8 This is an example of the effect of point cloud coloring using a point cloud coloring method from a related technology. Figure 9 This is a diagram showing the effect of point cloud coloring using the point cloud coloring method of this embodiment. For example... Figure 8 and Figure 9 As shown, the point cloud coloring method of this embodiment can significantly reduce the probability of foreground and background color misalignment.

[0092] In this embodiment, since the occlusion relationship between objects is different from different viewpoints, the point cloud of visible voxels at a certain viewpoint is rendered to obtain a sparse depth map at that viewpoint. Pixels with no effective depth in the sparse depth map are filled to obtain a dense depth map. Then, based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range nearby, the risky coloring region in the dense depth map is determined. Only pixels outside the risky coloring region are colored, which can effectively reduce the probability of foreground and background color misalignment.

[0093] Figure 10 This is a flowchart illustrating step S1 in some embodiments of this disclosure. For example... Figure 10 As shown, step S1 may include the following steps:

[0094] S1-1: Perform voxel visibility detection on the point cloud data of the voxels to be shaded from a first-view perspective to obtain the point cloud set of visible voxels from the first-view perspective.

[0095] Using a preset location as the observation point of the first viewpoint, voxel visibility detection is performed on the point cloud data of all voxels to be shaded. Based on the voxel visibility detection results, all point clouds of visible voxels under the first viewpoint are determined, and a point cloud set of visible voxels under the first viewpoint is generated.

[0096] In some optional embodiments of this disclosure, the shooting position from the first perspective can be used as the preset position for the point cloud data obtained by capturing images with a camera and processing them.

[0097] S1-2: Render the point cloud set of visible voxels from the first-person perspective to obtain a sparse depth map.

[0098] A rendering model based on 3D reconstruction technology can be used to render the point cloud of visible voxels in the first-person view, and obtain a sparse depth map in the first-person view.

[0099] In this embodiment, by performing voxel visibility detection on the point cloud data of each voxel from the first viewpoint, a set of point clouds of visible voxels from the first viewpoint can be obtained. Then, only the set of point clouds of visible voxels from the first viewpoint is rendered to obtain a sparse depth map. Compared with the method of directly rendering all point clouds to the first viewpoint to generate a sparse depth map, the generation efficiency of the sparse depth map can be greatly improved.

[0100] Figure 11 This is a flowchart illustrating step S1-1 in some embodiments of this disclosure. For example... Figure 11 As shown, step S1-1 may include the following steps:

[0101] S1-1-1: Construct a voxel structure based on point cloud data to obtain the hierarchical bounding volume structure of the voxel to be shaded.

[0102] By utilizing the Bounding Volume Hierarchy (BVH) construction technique, a voxel structure is constructed from the point cloud data of the voxels to be shaded, thus obtaining the BVH of the voxels to be shaded.

[0103] S1-1-2: Rays are emitted from the camera center in the first-person view to the centers of each voxel in the hierarchical enclosing structure. The first depth of each voxel is determined based on the distance from the camera center to the center of each voxel, and the second depth of each voxel is determined based on the distance from the camera center to the intersection of the hierarchical enclosing structure.

[0104] Using the camera center from a first-person perspective as the ray origin, rays are emitted towards the centers of each voxel in the hierarchical enclosing structure, yielding a first depth *d* between the ray origin and each voxel center. A second depth *t* is then obtained based on the distance from the ray origin to the intersection point of the enclosing structure. The first depth can also be referred to as the theoretical depth, and the second depth as the actual measured depth.

[0105] S1-1-3: Based on the data relationship between the first depth and the second depth of each voxel, determine the visible voxels in the first view.

[0106] For each voxel of the BVH, if the voxel is not occluded by other voxels, then t≥d, indicating that the voxel is visible in the first view; if the voxel is occluded by other voxels, then t<d, indicating that the voxel is not visible in the first view.

[0107] S1-1-4: Obtain the point cloud set corresponding to the visible voxels in the first view.

[0108] Obtain all the visible voxels in the first view, and then obtain the point cloud set corresponding to all the visible voxels.

[0109] In this embodiment, since the BVH is a structure for spatially sparse geometric bodies, after constructing the scene BVH structure, scene voxelization can be achieved. Therefore, the BVH of the voxels to be colored can be constructed according to the point cloud data, and the theoretical depth and actual depth of each voxel in the BVH can be obtained by means of ray detection. Furthermore, according to the data relationship between the theoretical depth and the actual depth of each voxel, the point cloud set corresponding to the visible voxels in the first view can be obtained quickly and accurately.

[0110] Figure 12 is a schematic flowchart of step S2 in some embodiments of the present disclosure. As Figure 12 shown, step S2 may include the following steps:

[0111] S2-1: Use a preset depth value search method to search for the depth value of each pixel without valid depth in the sparse depth map, and determine the depth value of each pixel based on the depth value search result.

[0112] For each pixel without valid depth in the sparse depth map, a preset depth value search method can be used to search for the target pixel closest to the pixel and having depth information, and the depth value of the target pixel is used as the depth value of the pixel.

[0113] S2-2: Fill the sparse depth map based on the depth value of each pixel to obtain a dense depth map in the first view.

[0114] Fill the sparse depth map according to the depth value determined for each pixel without valid depth in the sparse depth map to obtain a dense depth map in the first view.

[0115] In this embodiment, the preset depth value search method can quickly search for the pixels with depth information near each pixel without valid depth in the sparse depth map, and then filling the sparse depth map can obtain a dense depth map in the first view.

[0116] In some embodiments of the present disclosure, the preset depth value search method is a breadth-first search method. For each pixel without valid depth in the sparse depth map, the breadth-first search method can quickly find the nearest pixel with depth, which helps to improve the generation efficiency of the dense depth map from the first perspective.

[0117] Figure 13 It is a schematic flowchart of step S3 in some embodiments of the present disclosure. As Figure 13 shown, step S3 may include the following steps:

[0118] S3-1: Obtain the depth value of a pixel in the dense depth map, and obtain the minimum depth value of the pixels within a preset range near the pixel.

[0119] Obtain the depth value depth of a pixel from the dense depth map, and obtain the minimum depth value depth of the pixels within a preset range near the pixel min .

[0120] S3-2: In response to the ratio between the depth value of the pixel and the minimum depth value of the pixels within a preset range near the pixel being less than a preset threshold, determine the pixel as a pixel within the risk coloring area.

[0121] If depth min / depth < a, then determine the pixel as a pixel within the risk coloring area (that is, determine that the pixel belongs to the risk coloring area), otherwise determine the pixel as a pixel within the safe coloring area (that is, determine that the pixel belongs to the safe coloring area). Here, a represents the preset threshold.

[0122] Repeat steps S3-1 to S3-2 until it is determined whether all pixels of the dense depth map are pixels within the risk coloring area, and obtain the risk coloring area of the dense depth map.

[0123] In this embodiment, since the depth values of the pixels in the area between the foreground and the background are quite different from the depth values of other nearby pixels, it is possible to quickly determine whether each pixel in the dense depth map is located in the area where color confusion is likely to occur between the foreground and the background according to the ratio between the depth value of the pixel in the dense depth map and the minimum depth value of its nearby pixels, and thus quickly obtain the risk coloring area of the dense depth map.

[0124] In some embodiments of the present disclosure, after step S4, the following steps may further be included:

[0125] S5: In response to at least one pixel within the risk coloring area belonging to the safe coloring area from the second perspective, color at least one pixel from the second perspective.

[0126] After the pixels in the non-risk coloring area are colored in the first view, the view is switched to the second view. The view is then checked to see if all pixels in the risk coloring area in the first view belong to the safe coloring area (i.e., non-risk coloring area) in the second view. If at least one pixel in the risk coloring area in the first view is found to belong to the safe coloring area in the second view, then that at least one pixel is colored in the second view.

[0127] In this embodiment, since the risk-colored area in the first view may not be the risk-colored area in other views, the probability of foreground and background color misalignment can be greatly reduced by using a multi-view rendering method that switches perspectives.

[0128] Figure 14 This is a structural block diagram of a point cloud coloring processing apparatus in one embodiment of this disclosure. Figure 14 As shown, the point cloud coloring processing apparatus includes:

[0129] The sparse depth map acquisition module 100 is used to render the point cloud of visible voxels in the first view to obtain a sparse depth map in the first view.

[0130] The dense depth map acquisition module 200 is used to perform depth value search on each pixel in the sparse depth map that has no effective depth using a preset depth value search method, and fill the sparse depth map with pixels based on the depth value search results to obtain a dense depth map in the first view.

[0131] The risk coloring region determination module 300 is used to determine the risk coloring region in the dense depth map based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of the pixels within a preset range near each pixel in the dense depth map.

[0132] The coloring module 400 is used to color pixels in the dense depth map except for the risky coloring region.

[0133] In some embodiments of this disclosure, the sparse depth map acquisition module 100 is used to perform voxel visibility detection on the point cloud data of the voxels to be shaded based on a first viewpoint, and obtain a point cloud set of visible voxels under the first viewpoint; the sparse depth map acquisition module 100 is also used to render the point cloud set of visible voxels under the first viewpoint to obtain a sparse depth map.

[0134] In some embodiments of this disclosure, the sparse depth map acquisition module 100 is used to construct a voxel structure based on point cloud data to obtain a hierarchical bounding volume structure of the voxels to be colored; the sparse depth map acquisition module 100 is also used to emit rays from the camera center in the first viewpoint to the center of each voxel in the hierarchical bounding volume structure, determine the first depth of each voxel based on the distance from the camera center to the center of each voxel, and determine the second depth of each voxel based on the distance from the camera center to the intersection of the hierarchical bounding volume structure; the sparse depth map acquisition module 100 is also used to determine the visible voxels in the first viewpoint based on the data relationship between the first depth and the second depth of each voxel; the sparse depth map acquisition module is also used to acquire the point cloud set corresponding to the visible voxels in the first viewpoint.

[0135] In some embodiments of this disclosure, the dense depth map acquisition module 200 is used to perform depth value search on each pixel in the sparse depth map that has no effective depth using a preset depth value search method, and determine the depth value of each pixel based on the depth value search results; the dense depth map acquisition module 200 is also used to fill the sparse depth map with pixels based on the depth value of each pixel to obtain a dense depth map under the first view.

[0136] In some embodiments of this disclosure, the preset depth value search method is a breadth-first search method.

[0137] In some embodiments of this disclosure, the risk coloring region determination module 300 is used to obtain the depth value of a pixel in a dense depth map and obtain the minimum depth value of pixels within a preset range near the pixel; the risk coloring region determination module 300 is also used to determine the pixel as a pixel within the risk coloring region in response to the ratio between the depth value of the pixel and the minimum depth value of pixels within the preset range near the pixel being less than a preset threshold.

[0138] In some embodiments of this disclosure, the coloring module 400 is further configured to color at least one pixel in the second view in response to the presence of at least one pixel in the risky coloring region belonging to the safe coloring region in the second view.

[0139] It should be noted that the specific implementation of the point cloud coloring device in this disclosure is similar to the specific implementation of the point cloud coloring method in this disclosure, and the technical effects of the point cloud coloring device in this disclosure are similar to the technical effects of the point cloud coloring method in this disclosure. For details, please refer to the description of the point cloud coloring method section. In order to reduce redundancy, it will not be repeated.

[0140] In addition, this disclosure also provides an electronic device, including:

[0141] Memory, used to store computer programs;

[0142] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the point cloud coloring method described in any of the above embodiments of the present disclosure.

[0143] Below, for reference Figure 15 To describe an electronic device according to embodiments of this disclosure. For example... Figure 15 As shown, the electronic device includes one or more processors and memory.

[0144] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0145] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the point cloud coloring methods of the various embodiments of this disclosure described above and / or other desired functions.

[0146] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0147] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0148] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0149] Of course, for the sake of simplicity, Figure 15 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0150] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the point cloud coloring methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0151] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0152] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the point cloud coloring methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0153] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0154] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0155] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0156] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0157] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0158] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0159] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0160] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A point cloud coloring processing method, characterized in that, include: Render the point cloud of visible voxels in the first view to obtain a sparse depth map in the first view. Using a preset depth value search method, depth values ​​are searched for each pixel in the sparse depth map that has no effective depth. Based on the depth value search results, the sparse depth map is filled with pixels to obtain a dense depth map under the first view. Based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of the pixels within a preset range near each pixel in the dense depth map, a risk-colored region in the dense depth map is determined. The risk-colored region is a region in the dense depth map where there are pixels with a foreground region within a preset pixel region. Color the pixels in the dense depth map except for the risk-colored region; The method of determining the risk-colored region in the dense depth map based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range near each pixel in the dense depth map includes: Obtain the depth value of a pixel in the dense depth map, and obtain the minimum depth value of pixels within a preset range near the pixel; In response to a situation where the ratio between the depth value of the pixel and the minimum depth value of pixels within a preset range near the pixel is less than a preset threshold, the pixel is identified as a pixel within the risk-colored area.

2. The method according to claim 1, characterized in that, The rendering of the point cloud of visible voxels in the first viewpoint to obtain a sparse depth map in the first viewpoint includes: Voxel visibility detection is performed on the point cloud data of the voxels to be shaded from a first-view perspective to obtain the point cloud set of visible voxels in the first-view perspective. The point cloud set of visible voxels from the first viewpoint is rendered to obtain the sparse depth map.

3. The method according to claim 2, characterized in that, The voxel visibility detection based on the point cloud data of the voxels to be shaded from a first-view perspective, to obtain the point cloud set of visible voxels from the first-view perspective, includes: Based on the point cloud data, a voxel structure is constructed to obtain the hierarchical bounding volume structure of the voxel to be shaded; Rays are emitted from the camera center in the first viewpoint toward the centers of each voxel in the hierarchical enclosing structure. The first depth of each voxel is determined based on the distance from the camera center to the center of each voxel, and the second depth of each voxel is determined based on the distance from the camera center to the intersection of the hierarchical enclosing structure. Based on the data relationship between the first depth and the second depth of each voxel, the visible voxels under the first viewpoint are determined. Obtain the point cloud set corresponding to the visible voxels in the first view.

4. The method according to any one of claims 1-3, characterized in that, The method of using a preset depth value search to perform depth value search on each pixel in the sparse depth map that has no effective depth, and then filling the sparse depth map with pixels based on the depth value search results to obtain a dense depth map under the first viewpoint includes: Using the preset depth value search method, a depth value search is performed on each pixel in the sparse depth map that has no effective depth, and the depth value of each pixel is determined based on the depth value search results. The sparse depth map is filled with pixels based on the depth values ​​of each pixel to obtain a dense depth map from the first viewpoint.

5. The method according to any one of claims 1-3, characterized in that, The preset depth value search method is a breadth-first search method.

6. The method according to any one of claims 1-3, characterized in that, After coloring the pixels in the dense depth map excluding the risk-colored region, the method further includes: In response to the presence of at least one pixel within the risky coloring region that belongs to the safe coloring region from a second viewpoint, the at least one pixel is colored from the second viewpoint.

7. A point cloud coloring processing device, characterized in that, include: The sparse depth map acquisition module is used to render the point cloud of visible voxels in the first view to obtain the sparse depth map in the first view. The dense depth map acquisition module is used to perform depth value search on each pixel in the sparse depth map that has no effective depth using a preset depth value search method, and fill the sparse depth map with pixels based on the depth value search results to obtain a dense depth map under the first view. The risk coloring region determination module is used to determine the risk coloring region in the dense depth map based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of the pixels within a preset range near each pixel in the dense depth map. The risk coloring region is the region in the dense depth map where there are pixels with a foreground region within the preset pixel region. A coloring module is used to color the pixels in the dense depth map except for the risk coloring region; The risk-colored region determination module is used to obtain the depth value of a pixel in the dense depth map and obtain the minimum depth value of pixels within a preset range near the pixel; in response to the ratio between the depth value of the pixel and the minimum depth value of pixels within the preset range near the pixel being less than a preset threshold, the pixel is determined to be a pixel within the risk-colored region.

8. The apparatus according to claim 7, characterized in that, The sparse depth map acquisition module is used to perform voxel visibility detection on the point cloud data of the voxels to be shaded from a first viewpoint, and obtain the point cloud set of visible voxels under the first viewpoint; the sparse depth map acquisition module is also used to render the point cloud set of visible voxels under the first viewpoint to obtain the sparse depth map.

9. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein, when the computer program product is executed, it implements the method described in any one of claims 1-6.

Images