Point cloud coloring processing method and device and electronic equipment

Abstract

The embodiment of the invention discloses a point cloud coloring processing method and device and electronic equipment, and the method comprises the steps: carrying out the rendering of a point cloud of visible voxels at a first visual angle, and obtaining a sparse depth map at the first visual angle; using a preset depth value search method to perform depth value search on each pixel without effective depth in the sparse depth map, and performing pixel filling on the sparse depth map based on a depth value search result to obtain a dense depth map under the first view angle; determining a risk coloring area in the dense depth map based on a data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of pixels in a preset range near each pixel in the dense depth map; and coloring the pixels in the dense depth map except the pixels in the risk coloring area. According to the embodiment of the invention, the probability of foreground and background coloring dislocation can be effectively reduced.

Description

Technical Field

[0001] The present disclosure relates to three-dimensional reconstruction technology and point cloud processing technology, and in particular to a point cloud coloring processing method, device and electronic device. Background Art

[0002] When reconstructing a scene in 3D, it is usually necessary to colorize the point cloud of the objects in the 3D scene. When coloring the point cloud, the foreground and background coloring may be misaligned near the occluded area. For example, when a telephone pole blocks the ground, the color of the telephone pole may be colored on the nearby ground, or when a tree trunk blocks the ground, the color of the tree trunk may be colored on the nearby ground, and so on.

[0003] How to reduce the probability of misalignment between foreground and background coloring during 3D reconstruction and accurately color the point cloud is an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present disclosure provide a point cloud coloring processing method, device and electronic device to solve the above problems.

[0005] A first aspect of an embodiment of the present disclosure provides a point cloud coloring processing method, comprising:

[0006] Rendering a point cloud of voxels visible at a first viewing angle to obtain a sparse depth map at the first viewing angle;

[0007] Using a preset depth value search method, perform a depth value search on each pixel without a valid depth in the sparse depth map, and fill pixels in the sparse depth map based on the depth value search results to obtain a dense depth map under the first viewing angle;

[0008] Determine a risk colored area in the dense depth map based on a data relationship between a depth value of each pixel in the dense depth map and a minimum depth value of pixels within a preset range near each pixel in the dense depth map;

[0009] Coloring is performed on pixels in the dense depth map except for pixels in the risk coloring area.

[0010] In some embodiments of the present disclosure, rendering the point cloud of visible voxels at the first viewing angle to obtain a sparse depth map at the first viewing angle includes:

[0011] Performing voxel visibility detection on the point cloud data of the voxels to be colored based on the first perspective to obtain a point cloud set of visible voxels under the first perspective;

[0012] The point cloud set of visible voxels under the first viewing angle is rendered to obtain the sparse depth map.

[0013] In some embodiments of the present disclosure, performing voxel visibility detection on point cloud data of voxels to be colored based on the first perspective to obtain a point cloud set of visible voxels under the first perspective includes:

[0014] Constructing a voxel structure based on the point cloud data to obtain a hierarchical bounding volume structure of the voxels to be colored;

[0015] Emitting rays from the center of the camera at the first viewing angle to the center of each voxel of the hierarchical bounding volume structure, determining a first depth of each voxel based on a distance from the center of the camera to the center of each voxel, and determining a second depth of each voxel based on a distance from the center of the camera to an intersection of the hierarchical bounding volume structure;

[0016] Determining visible voxels at the first viewing angle based on a data relationship between the first depth of each voxel and the second depth of each voxel;

[0017] Obtain the point cloud set corresponding to the visible voxels at the first viewing angle.

[0018] In some embodiments of the present disclosure, the method of searching for depth values ​​of pixels without valid depth in the sparse depth map by using a preset depth value search method, and performing pixel filling on the sparse depth map based on the depth value search results to obtain a dense depth map under the first viewing angle includes:

[0019] Using the preset depth value search method, performing a depth value search on each pixel without a valid depth in the sparse depth map, and determining a depth value of each pixel based on the depth value search result;

[0020] The sparse depth map is pixel-filled based on the depth values ​​of the pixels to obtain a dense depth map at the first viewing angle.

[0021] In some embodiments of the present disclosure, the preset depth value search method is a breadth-first search method.

[0022] In some embodiments of the present disclosure, determining the risk colored area in the dense depth map based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range near each pixel in the dense depth map includes:

[0023] Obtaining a depth value of a pixel in the dense depth map, and obtaining a minimum depth value of pixels within a preset range near the pixel;

[0024] In response to a ratio between the depth value of the pixel and a minimum depth value of pixels within a preset range near the pixel being smaller than a preset threshold, the pixel is determined to be a pixel within the risk colored area.

[0025] In some embodiments of the present disclosure, after coloring the pixels in the dense depth map except for the risk coloring area, the method further includes:

[0026] In response to at least one pixel in the risk colored area belonging to the safe colored area at a second viewing angle, the at least one pixel is colored at the second viewing angle.

[0027] According to a second aspect of an embodiment of the present disclosure, a point cloud coloring processing device is provided, comprising:

[0028] A sparse depth map acquisition module, configured to render a point cloud of visible voxels at a first viewing angle to obtain a sparse depth map at the first viewing angle;

[0029] A dense depth map acquisition module is used to perform a depth value search for each pixel without a valid depth in the sparse depth map using a preset depth value search method, and fill the sparse depth map with pixels based on the depth value search results to obtain a dense depth map under the first viewing angle;

[0030] a risk colored area determination module, configured to determine a risk colored area in the dense depth map based on a data relationship between a depth value of each pixel in the dense depth map and a minimum depth value of pixels within a preset range near each pixel in the dense depth map;

[0031] A coloring module is used to color the pixels in the dense depth map except the pixels in the risk coloring area.

[0032] In some embodiments of the present disclosure, the sparse depth map acquisition module is used to perform voxel visibility detection on the point cloud data of the voxels to be colored based on the first perspective to obtain a point cloud set of visible voxels under the first perspective; the sparse depth map acquisition module is also used to render the point cloud set of visible voxels under the first perspective to obtain the sparse depth map.

[0033] In some embodiments of the present disclosure, the sparse depth map acquisition module is used to construct a voxel structure based on the point cloud data to obtain a hierarchical bounding volume structure of the voxels to be colored; the sparse depth map acquisition module is also used to emit rays from the camera center at the first perspective to the centers of each voxel in the hierarchical bounding volume structure, determine the first depth of each voxel based on the distance from the camera center to the center of each voxel, and determine the second depth of each voxel based on the distance from the camera center to the intersection of the hierarchical bounding volume structure; the sparse depth map acquisition module is also used to determine the visible voxels at the first perspective based on the data relationship between the first depth of each voxel and the second depth of each voxel; the sparse depth map acquisition module is also used to obtain the point cloud set corresponding to the visible voxels at the first perspective.

[0034] In some embodiments of the present disclosure, the dense depth map acquisition module is used to use the preset depth value search method to perform a depth value search for each pixel without a valid depth in the sparse depth map, and determine the depth value of each pixel based on the depth value search result; the dense depth map acquisition module is also used to fill pixels of the sparse depth map based on the depth value of each pixel to obtain a dense depth map under the first viewing angle.

[0035] In some embodiments of the present disclosure, the preset depth value search method is a breadth-first search method.

[0036] In some embodiments of the present disclosure, the risk coloring area determination module is used to obtain the depth value of a pixel in the dense depth map, and obtain the minimum depth value of pixels within a preset range near the pixel; the risk coloring area determination module is also used to determine the pixel as a pixel within the risk coloring area in response to the ratio between the depth value of the pixel and the minimum depth value of pixels within a preset range near the pixel being less than a preset threshold.

[0037] In some embodiments of the present disclosure, the coloring module is further used to color the at least one pixel in the risk coloring area in response to the presence of the at least one pixel belonging to the safe coloring area at a second viewing angle, at the second viewing angle.

[0038] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0039] a memory for storing a computer program product;

[0040] The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in the first aspect above is implemented.

[0041] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0042] The point cloud coloring processing method, device, and electronic device of the disclosed embodiments render the point cloud of visible voxels at a certain viewing angle to obtain a sparse depth map at that viewing angle, and fill the pixels without effective depth in the sparse depth map to obtain a dense depth map, and then determine the risk coloring area in the dense depth map based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of the pixels in a preset range nearby, and only color the pixels outside the risk coloring area, which can effectively reduce the probability of foreground and background coloring misalignment. In addition, since the risk coloring area at one viewing angle may not be a risk coloring area at other viewing angles, the probability of foreground and background coloring misalignment can be greatly reduced through multi-view rendering.

[0043] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0045] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0046] Figure 1 A schematic diagram of a process of a point cloud coloring processing method in one embodiment of the present disclosure;

[0047] Figure 2 is a camera-photographed image of a scene to be rendered from a first-person perspective in an example of the present disclosure;

[0048] Figure 3 yes Figure 2 The corresponding sparse depth map;

[0049] Figure 4 is a sparse depth map in an example of the present disclosure;

[0050] Figure 5 yes Figure 4 The corresponding dense depth map;

[0051] Figure 6 is a camera-photographed image of a scene to be rendered from a first perspective in another example of the present disclosure;

[0052] Figure 7 yes Figure 6 Schematic diagram of the corresponding risk colored areas;

[0053] Figure 8 This is an example of the effect of using the point cloud coloring method in the related technology to color the point cloud;

[0054] Fig. 9 This is an effect diagram after the point cloud is colored using the point cloud coloring method of this embodiment;

[0055] Fig.10 is a flowchart of step S1 in some embodiments of the present disclosure;

[0056] Fig.11 is a flowchart of step S1-1 in some embodiments of the present disclosure;

[0057] Fig.12 is a flowchart of step S2 in some embodiments of the present disclosure;

[0058] Fig.13 is a flowchart of step S3 in some embodiments of the present disclosure;

[0059] Fig.14 This is a structural block diagram of a point cloud coloring processing device in one embodiment of the present disclosure;

[0060] Fig.15 The structure block diagram of an electronic device in one embodiment of the present disclosure is shown in FIG. DETAILED DESCRIPTION

[0061] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0062] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0063] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0064] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0065] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0066] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0067] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0068] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0069] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0070] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0071] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0072] Figure 1 FIG. 1 is a flow chart of a point cloud coloring processing method in one embodiment of the present disclosure. Figure 1 As shown, the point cloud coloring method includes:

[0073] S1: Render the point cloud of the voxels visible in the first perspective to obtain a sparse depth map in the first perspective.

[0074] A camera can be used to take pictures of the scene to be reconstructed from multiple perspectives to obtain images of the scene to be reconstructed from multiple perspectives, and point cloud data of each voxel in the scene to be reconstructed can be extracted from the images of the scene to be reconstructed from multiple perspectives. A radar can also be used to scan the scene to be rendered at multiple different positions to obtain point cloud data of each voxel in the scene to be rendered. Voxel is the abbreviation of volume element, and a volume containing voxels can be expressed by stereo rendering or extracting polygonal isosurfaces with a given threshold contour.

[0075] It should be noted that obtaining point cloud data of each voxel in the image by processing the image taken by the camera, and obtaining point cloud data of each voxel of the scanned object by processing the point cloud data obtained by radar scanning, are both technical means well known to those skilled in the art and will not be elaborated herein.

[0076] Given a first perspective, point cloud data of each voxel under the first perspective is read from point cloud data of each voxel in the scene to be rendered. The first perspective may be the current perspective, that is, the point cloud coloring method of the embodiment of the present disclosure may be applicable to real-time point cloud coloring.

[0077] Since directly rendering all point clouds to the first perspective to obtain a sparse depth map at the first perspective is inefficient, a rendering model based on 3D reconstruction technology can be used to render the point cloud of the voxels visible at the first perspective to obtain a sparse depth map at the first perspective. The time complexity of directly rendering all point clouds to the current perspective to obtain a sparse depth map is much greater than the time complexity of rendering the point cloud of the voxels visible at the first perspective.

[0078] Figure 2 is a camera-photographed image of a scene to be rendered from a first-person perspective in an example of the present disclosure, Figure 3 yes Figure 2 The corresponding sparse depth map. Figure 2 and Figure 3 As shown, for a scene with leaves as the foreground and buildings or the sky as the background in the first perspective, a corresponding sparse depth map can be rendered. There are pixel areas without depth information (ie, black image areas) in the sparse depth map.

[0079] S2: Using a preset depth value search method, perform a depth value search on each pixel without a valid depth in the sparse depth map, and fill pixels in the sparse depth map based on the depth value search results to obtain a dense depth map under the first perspective.

[0080] The pixels without effective depth in the sparse depth map are pixels at pixel positions without depth information within an image region (eg, a rectangular region) of the sparse depth map.

[0081] Using a preset depth value search method, for each pixel without valid depth in the sparse depth map, search for pixels with valid depth within a preset range near the pixel, and determine the depth value of each pixel without valid depth in the sparse depth map according to the depth value of the searched pixel. The preset depth value search method may include a depth first search (DFS) method, a breadth first search (BFS) method, a best first search (BFS) method, and the like.

[0082] According to the depth value determined for each pixel without valid depth in the sparse depth map, each pixel that originally has no valid depth in the sparse depth map is filled to obtain a dense depth map under the first viewing angle.

[0083] Figure 4 is a sparse depth map in an example of the present disclosure, Figure 5 yes Figure 4 The corresponding dense depth map. Figure 4 and Figure 5 As shown, the corresponding dense depth map can be obtained by filling the pixel area without valid depth (ie, the black image area) in the sparse depth map.

[0084] S3: Determine a risk colored area in the dense depth map based on a data relationship between a depth value of each pixel in the dense depth map and a minimum depth value of pixels within a preset range near each pixel in the dense depth map.

[0085] For each pixel in the dense depth map, the depth value of the pixel is obtained, and the minimum depth value of the pixels within a preset range near the pixel is obtained. The preset range may be an N*M pixel area, and N and M are both integers greater than 1.

[0086] After obtaining the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range near the pixel, the data relationship between the two depth values ​​may be calculated using a preset data relationship calculation formula.

[0087] According to the correspondence between the preset data relationship and the risk coloring area, determine whether there are pixels in the foreground area in the preset pixel area near each pixel in the dense depth map, and determine the area composed of pixels in the foreground area in all nearby preset pixel areas in the dense depth map as the risk coloring area in the dense depth map, and determine the pixel area other than the risk coloring area in the dense depth map as the safe coloring area.

[0088] Figure 6 is a camera-photographed image of a scene to be rendered from a first-person perspective in another example of the present disclosure, Figure 7 yes Figure 6 Schematic diagram of the corresponding risk colored area. Figure 6 and Figure 7 As shown, the risk colored area (ie, gray image area) in the dense depth map is determined by the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range near each pixel in the dense depth map.

[0089] S4: Colorize the pixels in the dense depth map except the risk colored area.

[0090] By using the rendering model to colorize the pixels in the dense depth map except the risky coloring area (i.e., the pixels in the safe coloring area), the probability of foreground and background coloring misalignment can be effectively reduced.

[0091] Figure 8 This is an example of the effect of using the point cloud coloring method in the related technology to color the point cloud. Fig. 9 : is the effect diagram after the point cloud is colored using the point cloud coloring method of this embodiment. Figure 8 and Fig. 9 As shown, the point cloud coloring method of this embodiment can greatly reduce the probability of misalignment of foreground and background coloring.

[0092] In this embodiment, since the occlusion relationship between objects at different viewing angles is different, the point cloud of visible voxels at a certain viewing angle is rendered to obtain a sparse depth map at that viewing angle, and the pixels without valid depth in the sparse depth map are filled to obtain a dense depth map. Then, the risk coloring area in the dense depth map is determined based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of the pixels within a preset range nearby. Only pixels outside the risk coloring area are colored, which can effectively reduce the probability of foreground and background coloring misalignment.

[0093] Fig.10 is a flow chart of step S1 in some embodiments of the present disclosure. Fig.10 As shown, step S1 may include the following steps:

[0094] S1-1: Perform voxel visibility detection on the point cloud data of the voxels to be colored based on the first perspective to obtain a point cloud set of visible voxels in the first perspective.

[0095] Taking the preset position as the observation point of the first perspective, voxel visibility detection is performed on the point cloud data of all voxels to be colored, and all point clouds of visible voxels in the first perspective are determined according to the voxel visibility detection results to generate a point cloud set of visible voxels in the first perspective.

[0096] In some optional embodiments of the present disclosure, for point cloud data obtained by taking images with a camera and processing them, the photographing position at the first viewing angle may be used as a preset position.

[0097] S1-2: Render the point cloud set of visible voxels in the first perspective to obtain a sparse depth map.

[0098] A rendering model based on three-dimensional reconstruction technology can be used to render the point cloud of voxels visible in the first perspective to obtain a sparse depth map in the first perspective.

[0099] In this embodiment, by performing voxel visibility detection on the point cloud data of each voxel at the first perspective, a point cloud set of visible voxels at the first perspective can be obtained, and then only the point cloud set of visible voxels at the first perspective is rendered to obtain a sparse depth map. Compared with the method of directly rendering all point clouds to the first perspective to generate a sparse depth map, the generation efficiency of the sparse depth map can be greatly improved.

[0100] Fig.11 is a flow chart of step S1-1 in some embodiments of the present disclosure. Fig.11 As shown, step S1-1 may include the following steps:

[0101] S1-1-1: Construct a voxel structure based on the point cloud data to obtain a hierarchical bounding volume structure of the voxels to be colored.

[0102] The voxel structure is constructed for the point cloud data of the voxels to be colored by using the bounding volume hierarchy (BVH) construction technology to obtain the BVH of the voxels to be colored.

[0103] S1-1-2: Emit rays from the center of the camera in the first perspective to the center of each voxel of the hierarchical bounding volume structure, determine the first depth of each voxel based on the distance from the center of the camera to the center of each voxel, and determine the second depth of each voxel based on the distance from the center of the camera to the intersection of the hierarchical bounding volume structure.

[0104] The camera center at the first viewing angle is used as the starting point of the ray, and the ray is emitted to the center of each voxel of the hierarchical bounding volume structure to obtain the first depth d between the starting point of the ray and the center of each voxel. The second depth t is obtained according to the distance from the starting point of the ray to the intersection of the bounding volume structure. The first depth can also be called the theoretical depth, and the second depth can be called the actual measured depth.

[0105] S1-1-3: Determine the visible voxels at the first viewing angle based on the data relationship between the first depth of each voxel and the second depth of each voxel.

[0106] For each voxel of the BVH, if the voxel is not occluded by other voxels, then t≥d, indicating that the voxel is visible in the first view; if the voxel is occluded by other voxels, then t<d, indicating that the voxel is not visible in the first view.

[0107] S1-1-4: Obtain the point cloud set corresponding to the visible voxels in the first view.

[0108] Obtain all the visible voxels in the first view, and then obtain the point cloud set corresponding to all the visible voxels.

[0109] In this embodiment, since the BVH is a structure for spatially sparse geometric bodies, after constructing the scene BVH structure, scene voxelization can be achieved. Therefore, the BVH of the voxels to be colored can be constructed according to the point cloud data, and the theoretical depth and actual depth of each voxel in the BVH can be obtained by using ray detection. Furthermore, according to the data relationship between the theoretical depth and actual depth of each voxel, the point cloud set corresponding to the visible voxels in the first view can be obtained quickly and accurately.

[0110] Fig.12 It is a schematic flowchart of step S2 in some embodiments of the present disclosure. As Fig.12 shown, step S2 may include the following steps:

[0111] S2-1: Use a preset depth value search method to search for the depth value of each pixel without valid depth in the sparse depth map, and determine the depth value of each pixel based on the depth value search result.

[0112] For each pixel without valid depth in the sparse depth map, a preset depth value search method can be used to search for the target pixel closest to the pixel and having depth information, and the depth value of the target pixel is used as the depth value of the pixel.

[0113] S2-2: Fill the sparse depth map based on the depth value of each pixel to obtain a dense depth map in the first view.

[0114] Fill the sparse depth map according to the depth value determined for each pixel without valid depth in the sparse depth map to obtain a dense depth map in the first view.

[0115] In this embodiment, by using the preset depth value search method, the pixels with depth information near each pixel without valid depth in the sparse depth map can be quickly searched, and then filling the sparse depth map can obtain a dense depth map in the first view.

[0116] In some embodiments of the present disclosure, the preset depth value search method is a breadth-first search method. For each pixel without valid depth in the sparse depth map, the breadth-first search method can quickly find the nearest pixel with depth, which helps to improve the generation efficiency of the dense depth map from the first perspective.

[0117] Fig.13 It is a schematic flowchart of step S3 in some embodiments of the present disclosure. As Fig.13 shown, step S3 may include the following steps:

[0118] S3-1: Obtain the depth value of a pixel in the dense depth map, and obtain the minimum depth value of the pixels within a preset range near the pixel.

[0119] Obtain the depth value depth of a pixel from the dense depth map, and obtain the minimum depth value depth of the pixels within a preset range near the pixel min .

[0120] S3-2: In response to the ratio between the depth value of the pixel and the minimum depth value of the pixels within a preset range near the pixel being less than a preset threshold, determine the pixel as a pixel within the risk coloring area.

[0121] If depth min / depth < a, determine the pixel as a pixel within the risk coloring area (i.e., determine that the pixel belongs to the risk coloring area), otherwise determine the pixel as a pixel within the safe coloring area (i.e., determine that the pixel belongs to the safe coloring area). Here, a represents the preset threshold.

[0122] Repeat steps S3-1 to S3-2 until it is determined whether all pixels of the dense depth map are pixels within the risk coloring area, and obtain the risk coloring area of the dense depth map.

[0123] In this embodiment, since the depth values of the pixels in the area between the foreground and the background are quite different from the depth values of other nearby pixels, it is possible to quickly determine whether each pixel in the dense depth map is located in the area where color confusion is likely to occur between the foreground and the background according to whether the ratio between the depth value of the pixel in the dense depth map and the minimum depth value of its nearby pixels is less than the preset threshold, and thus the risk coloring area of the dense depth map can be quickly obtained.

[0124] In some embodiments of the present disclosure, after step S4, the following steps may further be included:

[0125] S5: In response to at least one pixel within the risk coloring area belonging to the safe coloring area from the second perspective, color at least one pixel from the second perspective.

[0126] After completing coloring the pixels in the non-risk colored area at the first viewing angle, switch to the second viewing angle to detect whether all pixels in the risk colored area at the first viewing angle belong to the safe colored area (i.e., the non-risk colored area) at the second viewing angle. When it is detected that there is at least one pixel in the risk colored area at the first viewing angle that belongs to the safe colored area at the second viewing angle, the at least one pixel is colored at the second viewing angle.

[0127] In this embodiment, since the risk colored area at the first viewing angle may not be a risk colored area at other viewing angles, the multi-view rendering method by converting the viewing angle can greatly reduce the probability of foreground and background coloring misalignment.

[0128] Fig.14 FIG. 1 is a structural block diagram of a point cloud coloring processing device in one embodiment of the present disclosure. Fig.14 As shown, the point cloud coloring processing device includes:

[0129] A sparse depth map acquisition module 100 is used to render a point cloud of visible voxels at a first viewing angle to obtain a sparse depth map at the first viewing angle;

[0130] A dense depth map acquisition module 200 is used to perform a depth value search for each pixel without a valid depth in the sparse depth map using a preset depth value search method, and fill the sparse depth map with pixels based on the depth value search results to obtain a dense depth map under the first viewing angle;

[0131] A risk colored area determination module 300 is used to determine a risk colored area in a dense depth map based on a data relationship between a depth value of each pixel in the dense depth map and a minimum depth value of pixels within a preset range near each pixel in the dense depth map;

[0132] The coloring module 400 is used to color the pixels in the dense depth map except for the risk coloring area.

[0133] In some embodiments of the present disclosure, the sparse depth map acquisition module 100 is used to perform voxel visibility detection on the point cloud data of the voxels to be colored based on the first perspective to obtain a point cloud set of visible voxels under the first perspective; the sparse depth map acquisition module 100 is also used to render the point cloud set of visible voxels under the first perspective to obtain a sparse depth map.

[0134] In some embodiments of the present disclosure, the sparse depth map acquisition module 100 is used to construct a voxel structure based on point cloud data to obtain a hierarchical bounding volume structure of voxels to be colored; the sparse depth map acquisition module 100 is also used to emit rays from the camera center at the first perspective to the centers of each voxel in the hierarchical bounding volume structure, determine the first depth of each voxel based on the distance from the camera center to the center of each voxel, and determine the second depth of each voxel based on the distance from the camera center to the intersection of the hierarchical bounding volume structure; the sparse depth map acquisition module 100 is also used to determine the visible voxels at the first perspective based on the data relationship between the first depth of each voxel and the second depth of each voxel; the sparse depth map acquisition module is also used to obtain a point cloud set corresponding to the visible voxels at the first perspective.

[0135] In some embodiments of the present disclosure, the dense depth map acquisition module 200 is used to use a preset depth value search method to perform a depth value search for each pixel without a valid depth in the sparse depth map, and determine the depth value of each pixel based on the depth value search result; the dense depth map acquisition module 200 is also used to fill the sparse depth map with pixels based on the depth value of each pixel to obtain a dense depth map under the first perspective.

[0136] In some embodiments of the present disclosure, the preset depth value search method is a breadth-first search method.

[0137] In some embodiments of the present disclosure, the risk coloring area determination module 300 is used to obtain the depth value of a pixel in a dense depth map and obtain the minimum depth value of pixels within a preset range near the pixel; the risk coloring area determination module 300 is also used to determine the pixel as a pixel within the risk coloring area in response to the ratio between the depth value of the pixel and the minimum depth value of pixels within a preset range near the pixel being less than a preset threshold.

[0138] In some embodiments of the present disclosure, the coloring module 400 is further configured to color at least one pixel at the second viewing angle in response to at least one pixel in the risk coloring area belonging to the safe coloring area at the second viewing angle.

[0139] It should be noted that the specific implementation of the point cloud coloring device of the embodiment of the present disclosure is similar to the specific implementation of the point cloud coloring method of the embodiment of the present disclosure, and the technical effect of the point cloud coloring device of the embodiment of the present disclosure is similar to the technical effect of the point cloud coloring method of the embodiment of the present disclosure. Please refer to the description of the point cloud coloring method part for details. In order to reduce redundancy, it will not be repeated.

[0140] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0141] Memory for storing computer programs;

[0142] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the point cloud coloring method described in any of the above embodiments of the present disclosure is implemented.

[0143] Below, reference Fig.15 To describe the electronic device according to the embodiment of the present disclosure. Fig.15 As shown, the electronic device includes one or more processors and memory.

[0144] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0145] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the point cloud coloring method of each embodiment of the present disclosure described above and / or other desired functions.

[0146] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0147] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0148] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0149] Of course, to simplify, Fig.15 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0150] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the point cloud coloring method according to various embodiments of the present disclosure described in the above part of this specification.

[0151] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0152] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the point cloud coloring method according to various embodiments of the present disclosure described in the above part of this specification.

[0153] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0154] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0155] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0156] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0157] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0158] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0159] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0160] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A point cloud coloring processing method, characterized in that: include: Rendering a point cloud of voxels visible at a first viewing angle to obtain a sparse depth map at the first viewing angle; Using a preset depth value search method, perform a depth value search on each pixel without a valid depth in the sparse depth map, and fill pixels in the sparse depth map based on the depth value search results to obtain a dense depth map under the first viewing angle; Determine a risk colored area in the dense depth map based on a data relationship between a depth value of each pixel in the dense depth map and a minimum depth value of pixels within a preset range near each pixel in the dense depth map; Coloring is performed on pixels in the dense depth map except for pixels in the risk coloring area.

2. The method according to claim 1, characterized in that The rendering of the point cloud of the visible voxels at the first viewing angle to obtain the sparse depth map at the first viewing angle includes: Performing voxel visibility detection on the point cloud data of the voxels to be colored based on the first perspective to obtain a point cloud set of visible voxels under the first perspective; The point cloud set of visible voxels under the first viewing angle is rendered to obtain the sparse depth map.

3. The method according to claim 2, characterized in that The step of performing voxel visibility detection on the point cloud data of the voxels to be colored based on the first perspective to obtain a point cloud set of visible voxels under the first perspective includes: Constructing a voxel structure based on the point cloud data to obtain a hierarchical bounding volume structure of the voxels to be colored; Emitting rays from the camera center at the first viewing angle to the center of each voxel of the hierarchical bounding volume structure, determining a first depth of each voxel based on a distance from the camera center to the center of each voxel, and determining a second depth of each voxel based on a distance from the camera center to an intersection of the hierarchical bounding volume structure; Determining visible voxels at the first viewing angle based on a data relationship between the first depth of each voxel and the second depth of each voxel; Obtain the point cloud set corresponding to the visible voxels at the first viewing angle.

4. The method according to any one of claims 1 to 3, characterized in that: The method of using a preset depth value search method to search for depth values ​​of pixels without valid depth in the sparse depth map, and filling pixels of the sparse depth map based on the depth value search results to obtain a dense depth map under the first viewing angle includes: Using the preset depth value search method, performing a depth value search on each pixel without a valid depth in the sparse depth map, and determining a depth value of each pixel based on the depth value search result; The sparse depth map is pixel-filled based on the depth values ​​of the pixels to obtain a dense depth map at the first viewing angle.

5. The method according to any one of claims 1 to 3, characterized in that: The preset depth value search method is a breadth-first search method.

6. The method according to any one of claims 1 to 3, characterized in that: The determining of the risk colored area in the dense depth map based on the data relationship between the depth value of each pixel in the dense depth map and the minimum depth value of pixels within a preset range near each pixel in the dense depth map includes: Obtaining a depth value of a pixel in the dense depth map, and obtaining a minimum depth value of pixels within a preset range near the pixel; In response to a ratio between the depth value of the pixel and a minimum depth value of pixels within a preset range near the pixel being smaller than a preset threshold, the pixel is determined to be a pixel within the risk colored area.

7. The method according to any one of claims 1 to 3, characterized in that: After coloring the pixels in the dense depth map except for the pixels in the risk coloring area, the method further includes: In response to at least one pixel in the risk colored area belonging to the safe colored area at a second viewing angle, the at least one pixel is colored at the second viewing angle.

8. A point cloud coloring processing device, characterized in that: include: A sparse depth map acquisition module, configured to render a point cloud of visible voxels at a first viewing angle to obtain a sparse depth map at the first viewing angle; A dense depth map acquisition module is used to perform a depth value search for each pixel without a valid depth in the sparse depth map using a preset depth value search method, and fill the sparse depth map with pixels based on the depth value search results to obtain a dense depth map under the first viewing angle; a risk colored area determination module, configured to determine a risk colored area in the dense depth map based on a data relationship between a depth value of each pixel in the dense depth map and a minimum depth value of pixels within a preset range near each pixel in the dense depth map; A coloring module is used to color the pixels in the dense depth map except the pixels in the risk coloring area.

9. The device according to claim 8, characterized in that The sparse depth map acquisition module is used to perform voxel visibility detection on the point cloud data of the voxels to be colored based on the first perspective to obtain a point cloud set of visible voxels under the first perspective; the sparse depth map acquisition module is also used to render the point cloud set of visible voxels under the first perspective to obtain the sparse depth map.

10. An electronic device, characterized in that: include: A memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the method described in any one of claims 1 to 7.

Images