Point cloud processing method, apparatus, device, and storage medium

Abstract

The application provides a point cloud processing method, device and equipment and a storage medium. The method comprises the following steps: in response to a selection operation on a point cloud, mapping the point cloud from a three-dimensional space to a two-dimensional space according to the distance between each three-dimensional point in the point cloud and adjacent three-dimensional points, and obtaining a depth map under a current view angle; wherein the depth value in the depth map is the depth value of at least part of the three-dimensional points on the surface of the point cloud under the current view angle; determining the mapping position of each three-dimensional point in the point cloud in the depth map; determining all three-dimensional points on the surface of the point cloud under the current view angle according to the difference between the depth value of each three-dimensional point in the point cloud and the depth value of the three-dimensional point at the mapping position of the three-dimensional point in the depth map; and determining target three-dimensional points in a selection region indicated by the selection operation from all the three-dimensional points on the surface of the point cloud under the current view angle. The embodiment realizes accurate selection of three-dimensional points on the surface of the point cloud.

Description

Technical Field

[0001] This application relates to the field of point cloud processing technology, and in particular to a point cloud processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Point cloud data refers to a set of vectors in a three-dimensional coordinate system, which is rendered as points when displayed (for example, in OpenGL, the primitive type is GL_POINTS).

[0003] In one scenario, a user needs to select a local area of ​​a point cloud, selecting points on the point cloud surface to perform certain processing on the selected 3D points.

[0004] One point cloud selection method in related technologies is based on the normal vector direction of 3D points. Specifically, it uses a penetration method to select 3D points whose normal vector direction intersects the plane of the screen (i.e., the normal vector points outwards from the screen) and are within the selection area, as the final selection result. However, the above selection method has the problem of missing or selecting too many points, and cannot completely select points belonging to the point cloud surface.

[0005] Therefore, it is necessary to provide a point cloud processing method to enable the selection of points on the point cloud surface. Summary of the Invention

[0006] In view of this, this application provides a point cloud processing method, apparatus, electronic device, and computer-readable storage medium.

[0007] Specifically, this application is implemented through the following technical solution:

[0008] According to a first aspect of the embodiments of this application, a point cloud processing method is provided, comprising:

[0009] In response to a selection operation on the point cloud, the point cloud is mapped from three-dimensional space to two-dimensional space based on the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points to obtain a depth map from the current viewpoint; wherein the depth values ​​in the depth map are the depth values ​​of at least some of the three-dimensional points on the surface of the point cloud from the current viewpoint.

[0010] Determine the mapping position of each 3D point in the point cloud in the depth map;

[0011] Based on the difference between the depth value of each 3D point in the point cloud and the depth value of the 3D point at its mapped position in the depth map, all 3D points on the surface of the point cloud at the current viewpoint are determined.

[0012] From all the three-dimensional points on the point cloud surface from the current perspective, determine the target three-dimensional point that is within the selection area indicated by the selection operation.

[0013] Optionally, determining all three-dimensional points on the surface of the point cloud from the current viewpoint based on the difference between the depth value of each three-dimensional point in the point cloud and the depth value of the three-dimensional point at its mapped position in the depth map includes:

[0014] For each 3D point in the point cloud, if the difference between the depth value of the 3D point and the depth value of the 3D point at its mapping position in the depth map is less than a preset threshold, the 3D point is determined to be a 3D point on the surface of the point cloud from the current viewpoint; otherwise, the 3D point is determined to be a 3D point on a non-surface of the point cloud from the current viewpoint.

[0015] Optionally, the step of mapping the point cloud from three-dimensional space to two-dimensional space based on the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points to obtain a depth map from the current viewpoint includes:

[0016] The display size of each three-dimensional point is determined based on the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points. The display size is used to ensure that there are no gaps between the three-dimensional point and its adjacent three-dimensional points when they are displayed.

[0017] The point cloud, whose display size for each 3D point is determined, is mapped from 3D space to 2D space to obtain a depth map from the current viewpoint; wherein, the pixels in the depth map are the mapping results of some 3D points on the surface of the point cloud from the current viewpoint.

[0018] Optionally, the distance between each 3D point in the point cloud and its adjacent 3D points is a preset value;

[0019] Wherein, if the point cloud is a point cloud with uniform gaps, the preset value is determined based on the statistical values ​​of the distances between each three-dimensional point in the point cloud and its adjacent three-dimensional points.

[0020] If the point cloud has uneven spacing, the preset value is determined based on the adjacency conditions indicated by the business scenario of the point cloud.

[0021] Optionally, the display size of each 3D point is determined based on the distance between the 3D point and its neighboring 3D points, as well as the conversion relationship between the distance in the 3D space and the number of pixels in the 2D space.

[0022] Optionally, the step of mapping the point cloud, which determines the display size of each 3D point, from 3D space to 2D space to obtain a depth map from the current viewpoint includes:

[0023] The point cloud, with the display size of each 3D point determined, is mapped from 3D space to 2D space to obtain the first mapping result;

[0024] Perform a depth test on the first mapping result to obtain a depth map from the current viewpoint.

[0025] Optionally, determining the mapping position of each 3D point in the point cloud in the depth map includes:

[0026] The point cloud is then mapped from three-dimensional space back to two-dimensional space to obtain a second mapping result;

[0027] Based on the second mapping result, the mapping position of each 3D point in the point cloud in the depth map is determined.

[0028] Optional, also includes:

[0029] Display the target 3D point on the display interface with a specified color and / or a specified texture; or

[0030] Set the target 3D point to the selected state; or

[0031] Delete the target 3D point.

[0032] According to a second aspect of the embodiments of this application, a point cloud processing apparatus is provided, comprising:

[0033] The depth map acquisition module is used to respond to the selection operation of the point cloud, and map the point cloud from three-dimensional space to two-dimensional space according to the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points to obtain a depth map from the current viewpoint; wherein, the depth value in the depth map is the depth value of at least a portion of the three-dimensional points on the surface of the point cloud from the current viewpoint.

[0034] The mapping position determination module is used to determine the mapping position of each 3D point in the point cloud in the depth map;

[0035] The surface 3D point determination module is used to determine all 3D points on the surface of the point cloud from the current viewpoint based on the difference between the depth value of each 3D point in the point cloud and the depth value of the 3D point at the mapping position in the depth map.

[0036] The target 3D point determination module is used to determine the target 3D point within the selection area indicated by the selection operation from all 3D points on the point cloud surface from the current viewpoint.

[0037] According to a third aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and executable instructions stored in the memory and executable on the processor;

[0038] Wherein, when the processor executes the executable instructions, it implements the steps in the method as described in any one of the first aspects.

[0039] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer instructions are stored, wherein when executed by a processor, the computer instructions implement the steps of the method described in any one of the first aspects.

[0040] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0041] In this embodiment of the disclosure, in response to a selection operation on a point cloud, the point cloud is mapped from three-dimensional space to two-dimensional space based on the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points to obtain a depth map from the current viewpoint. Considering the gaps between different three-dimensional points, to avoid presenting non-point cloud surface three-dimensional points in the depth map through these gaps, the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points is referenced to ensure that the depth values ​​in the depth map represent the depth values ​​of at least a portion of the three-dimensional points on the point cloud surface from the current viewpoint. Then, after determining the mapping position of each three-dimensional point in the point cloud in the depth map, the difference between the depth value of each three-dimensional point in the point cloud and the depth value at its mapping position in the depth map determines whether the three-dimensional point is on the point cloud surface. After traversing all three-dimensional points in the point cloud, all three-dimensional points on the point cloud surface from the current viewpoint can be determined, as well as the target three-dimensional points within the selection area indicated by the selection operation, thus achieving accurate selection of three-dimensional points on the point cloud surface.

[0042] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0044] Figure 1 This is a schematic flowchart illustrating a point cloud processing method according to an exemplary embodiment of this application.

[0045] Figure 2A This is a schematic diagram of a point cloud as observed from a frontal view, illustrating an exemplary embodiment of this application.

[0046] Figure 2B This is a schematic diagram illustrating a point cloud as seen from a side view and a selection area drawn by the user on the point cloud, as shown in an exemplary embodiment of this application.

[0047] Figure 3A This is an exemplary embodiment of the present application illustrating the display of target 3D points within a selected area in a point cloud viewed from a side view.

[0048] Figure 3B This is an exemplary embodiment of the present application illustrating the display of target 3D points within a selected area in a point cloud viewed from a frontal perspective.

[0049] Figure 4 This is a schematic diagram of a depth map shown in an exemplary embodiment of this application.

[0050] Figure 5 This is an exemplary schematic diagram of an electronic device illustrated in an exemplary embodiment of this application.

[0051] Figure 6 This is a schematic diagram of the structure of a point cloud processing device shown in an exemplary embodiment of this application. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0053] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0054] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0055] One point cloud selection method in related technologies is based on the normal vector direction of three-dimensional points. Specifically, a penetration method is used to select points by mapping all three-dimensional points in the point cloud from three-dimensional space to two-dimensional space. Based on the mapping result of each three-dimensional point, it is determined whether the normal vector of the three-dimensional point is pointing out of the screen and within the selection area. Finally, the three-dimensional points whose normal vector direction intersects with the plane where the screen is located (that is, whose normal vector is pointing out of the screen) and are within the selection area are taken as the final selection result.

[0056] However, the above selection method has the following problem: it cannot select all points belonging to the point cloud surface.

[0057] (1) If the point cloud data is viewed from the current perspective and consists of several layers of three-dimensional points with different depths, and the normal vectors of multiple layers of three-dimensional points intersect with the plane where the screen is located, then all of these three-dimensional points will be selected, which leads to the problem of multiple selection, that is, selecting three-dimensional points that do not belong to the point cloud surface.

[0058] (2) If the point cloud is scanned by 3D scanning software, some 3D points are noisy points. The normal vector direction of the noisy 3D points may not be consistent with the normal vector of the corresponding position on the real object. Therefore, although these 3D points are on the surface, the normal vector direction may be facing inwards from the screen and there is no intersection with the plane where the screen is located, which leads to the problem of missed selection.

[0059] Therefore, to address the problems in related technologies, this application provides a point cloud processing method. In response to a selection operation on the point cloud, the method maps the point cloud from three-dimensional space to two-dimensional space based on the distance between each three-dimensional point and its adjacent three-dimensional points, obtaining a depth map from the current viewpoint. Considering the gaps between different three-dimensional points, to avoid presenting non-point cloud surface three-dimensional points in the depth map through these gaps, the method references the distance between each three-dimensional point and its adjacent three-dimensional points to ensure that the depth values ​​in the depth map represent the depth values ​​of at least a portion of the three-dimensional points on the point cloud surface from the current viewpoint. After determining the mapping position of each three-dimensional point in the point cloud in the depth map, the difference between the depth value of each three-dimensional point in the point cloud and the depth value at its mapping position in the depth map determines whether the three-dimensional point is on the point cloud surface. After traversing all three-dimensional points in the point cloud, all three-dimensional points on the point cloud surface from the current viewpoint can be identified. Finally, target three-dimensional points within the selection area indicated by the selection operation can be determined from all three-dimensional points on the point cloud surface from the current viewpoint, achieving accurate selection of three-dimensional points on the point cloud surface.

[0060] Whether it is noise or other types of 3D points, as long as they are on the point cloud surface, they can be filtered out by the point cloud processing method provided in this application, reducing the risk of missed selection. Furthermore, since the depth value in the depth map is the depth value of at least some of the 3D points on the point cloud surface from the current viewpoint, by comparing the depth value of each 3D point in the point cloud with the depth value at the mapped position in the depth map, 3D points that are far from the point cloud surface (not belonging to the point cloud surface) can be filtered out, reducing the risk of over-selection and achieving accurate selection of 3D points on the point cloud surface.

[0061] The point cloud processing method provided in this application embodiment can be executed by an electronic device, including but not limited to smartphones / mobile phones, tablet computers, personal digital assistants (PDAs), laptop computers, desktop computers, media content players, video game consoles / systems, virtual reality systems, augmented reality systems, wearable devices (e.g., watches, glasses, gloves, headwear (e.g., hats, helmets, virtual reality headsets, augmented reality headsets, head-mounted devices (HMDs), headbands), pendants, armbands, leg rings, shoes, vests), remote controls, or any other device.

[0062] For example, the electronic device includes a processor and a memory, the memory storing executable instructions that can run on the processor, and the processor executing the executable instructions to implement the point cloud processing method provided in the embodiments of this application.

[0063] For example, the electronic device integrates a computer program product, and when the electronic device executes the computer program product, it implements the point cloud processing method provided in the embodiments of this application.

[0064] In an exemplary application scenario, during the process of obtaining the point cloud of a three-dimensional target object by scanning it with a measuring instrument, the measuring instrument may also scan some other debris on the surface of the target object that does not belong to the target object. For example, when scanning a vehicle, water droplets splashed onto the vehicle, large amounts of dust, and other debris may also be scanned; or when scanning teeth, fingers that are opening the mouth may be scanned. As a result, the surface of the obtained point cloud may contain three-dimensional points of debris that do not belong to the target object. In this case, the point cloud processing method provided in this application can be applied to select the target three-dimensional points that belong to the point cloud surface and are within the selected area according to the user's selection operation, and delete the target three-dimensional points that belong to the debris, thereby realizing the processing of noise points in the point cloud.

[0065] Please see Figure 1 , Figure 1 This is a schematic diagram of a point cloud processing method provided in an embodiment of this application. The method can be executed by an electronic device, and the method includes:

[0066] In S101, in response to the selection operation of the point cloud, the point cloud is mapped from three-dimensional space to two-dimensional space according to the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points to obtain a depth map under the current view; wherein, the depth value in the depth map is the depth value of at least some three-dimensional points on the surface of the point cloud under the current view.

[0067] In S102, the mapping position of each 3D point in the point cloud in the depth map is determined.

[0068] In S103, based on the difference between the depth value of each three-dimensional point in the point cloud and the depth value of the three-dimensional point at its mapped position in the depth map, all three-dimensional points on the surface of the point cloud at the current viewpoint are determined.

[0069] In S104, from all the three-dimensional points on the point cloud surface from the current viewpoint, the target three-dimensional point within the selection area indicated by the selection operation is determined.

[0070] In this embodiment, considering the gaps between different 3D points, even if the point cloud is relatively dense, gaps can still be seen when magnified to a certain extent, and even distant 3D points can be seen through these gaps. Therefore, to avoid presenting 3D points that are not on the point cloud surface through gaps in the depth map, this application considers the distance between each 3D point in the point cloud and its adjacent 3D points during the mapping process from 3D space to 2D space. This ensures that the depth value of the mapped depth map only includes the depth values ​​of at least some 3D points that are on the point cloud surface at the current viewpoint, and does not include the depth values ​​of 3D points that are not on the point cloud surface. This allows... Using the depth value of the depth map as a reference, after determining the mapping position of each 3D point in the point cloud in the depth map, the difference between the depth value of each 3D point in the point cloud and the depth value at the mapping position of the 3D point in the depth map can be used to determine whether the 3D point is on the surface of the point cloud, thereby determining all 3D points on the surface of the point cloud from the current viewpoint; then, from all 3D points on the surface of the point cloud from the current viewpoint, the target 3D point within the selection area indicated by the selection operation can be determined, thereby achieving accurate selection of 3D points on the surface of the point cloud and meeting the user's processing needs for 3D points on the surface of the point cloud.

[0071] In some embodiments, please refer to Figure 2A and Figure 2B , Figure 2A Point clouds as observed from a frontal view. Figure 2B This is a point cloud as seen from a side view. See also an example. Figure 2B Users can make selections on the point cloud display interface, defining a specific area of ​​the point cloud from a side view (i.e., Figure 2B(The selected area is shown in the image). The electronic device then responds to the user's selection operation by executing the point cloud processing method provided in this application embodiment to determine the target 3D point within the selected area indicated by the selection operation. For example, please refer to... Figure 3A The electronic device can display the selected target 3D point in a specified color on the display interface. This target 3D point is a 3D point that is located on the point cloud surface and within the selected area from the current side viewpoint. (See [link to relevant documentation]). Figure 3B When the point cloud is adjusted from a side view to a front view, the display result of selecting the point cloud from the side view can be seen. The 3D points on the surface of the point cloud that are within the selection area in the side view are selected, while the 3D points in the areas blocked in the side view are not selected.

[0072] In one possible implementation, in response to a selection operation on the point cloud, the electronic device first determines the display size of each 3D point based on the distance between each 3D point in the point cloud and its adjacent 3D points. This display size ensures that the 3D point and its adjacent 3D points are displayed without gaps. Since a display size is determined that fills the gaps when each 3D point is displayed, only the 3D points on the point cloud surface are visible from any viewing angle, thus avoiding the presentation of 3D points not on the point cloud surface through gaps in the acquired depth map. Mapping the point cloud with the determined display size of each 3D point from 3D space to 2D space to obtain a depth map from the current viewing angle ensures that the pixels in the obtained depth map are mapping results of the portion of 3D points on the point cloud surface at the current viewing angle. In other words, the depth values ​​in the depth map represent the depth values ​​of at least a portion of the 3D points on the point cloud surface at the current viewing angle. This embodiment fills the gaps between three-dimensional points in the above manner, so that no matter how much the point cloud is scaled, the final mapping result in the depth map is only one layer of the point cloud surface; that is to say, no matter how much the point cloud is magnified, this application can achieve the selection of three-dimensional points on the point cloud surface by filling the gaps in the above manner.

[0073] In one example, please refer to Figure 4 , Figure 4 The image shows the depth map obtained after determining the display size of each 3D point to fill the gaps between the 3D points.

[0074] For example, the distance between each 3D point in the point cloud and its adjacent 3D points can be calculated based on the 3D coordinates between them.

[0075] For example, considering that accurately calculating the distance between each 3D point in the point cloud and its adjacent 3D points based on 3D coordinates requires a lot of computing resources, in order to save computing resources and improve computing efficiency, the distance between each 3D point in the point cloud and its adjacent 3D points can be a preset value.

[0076] In one example, if the point cloud is a point cloud with uniform gaps, the preset value is determined based on the statistical values ​​of the distances between each of the three-dimensional points in the point cloud and its adjacent three-dimensional points; the statistical values ​​include, but are not limited to, the average, median, or maximum value, etc.; taking the average as an example, in order to ensure that the display size of the determined three-dimensional points can fill the gaps, the preset value can be a value slightly larger than the average.

[0077] In another example, if the point cloud is a point cloud with uneven spacing, the preset value can be predetermined based on the adjacency conditions indicated by the business scenario of the point cloud. This embodiment does not impose any restrictions on this. It is assumed that the preset value is 5cm, that is, any two three-dimensional points within 5cm are adjacent to each other, and any two three-dimensional points beyond 5cm are not adjacent.

[0078] In this context, uniform gaps can be understood as the difference between any two gaps being less than a preset difference. Conversely, if a point cloud does not satisfy the condition that the difference between any two gaps is less than the preset difference, it is considered a point cloud with uneven gaps. This preset difference can be set according to the specific application scenario. Gap refers to the distance between a 3D point and its adjacent 3D points.

[0079] For example, the electronic device pre-stores the conversion relationship between distance in three-dimensional space and the number of pixels in two-dimensional space. For each three-dimensional point in the point cloud, the electronic device can determine the display size of the three-dimensional point based on the distance between the three-dimensional point and its neighboring three-dimensional points, as well as the pre-stored conversion relationship between viewing distance in three-dimensional space and the number of pixels in two-dimensional space.

[0080] For example, after determining the display size of each 3D point in a point cloud, the electronic device can map the point cloud, with the determined display size of each 3D point, from 3D space to 2D space to obtain a first mapping result. The first mapping result includes the 2D information (such as 2D texture and depth value) of each 3D point in the point cloud. Then, a depth test is performed on the first mapping result. The purpose of the depth test is to achieve the occlusion effect of the point cloud. For example, if a 3D point on a non-surface part of the point cloud is occluded by a 3D point on the surface of the point cloud, the 2D information of the non-surface 3D point will not be rendered in the depth map to match the actual observation effect of the human eye. Through the depth test, a depth map is obtained based on the 2D information of the 3D points visible from the current viewpoint. Furthermore, because the display size that each 3D point can fill the gaps when displayed is redefined before the point cloud mapping, only the 3D points on the surface of the point cloud are visible from the current viewpoint. Therefore, the depth values ​​in the mapped depth map are all the depth values ​​of at least some of the 3D points on the surface of the point cloud from the current viewpoint.

[0081] In depth testing, 2D information of 3D points that are not visible from the current viewpoint is discarded. Furthermore, if a pixel in the depth map corresponds to at least two 3D points on the point cloud surface, considering that the depth values ​​of at least two 3D points corresponding to the same pixel on the point cloud surface are similar, only the 2D information of one of the 3D points can be used to render that pixel, discarding the others. Therefore, the depth values ​​in the mapped depth map include the depth values ​​of at least some of the 3D points on the point cloud surface from the current viewpoint, but not necessarily the depth values ​​of all 3D points on the point cloud surface.

[0082] In some embodiments, since some 2D mapping results of 3D points are discarded during depth testing, it is necessary to remap the point cloud from 3D space to 2D space to obtain a second mapping result. Then, based on the second mapping result, the mapping position of each 3D point in the point cloud in the depth map is determined. Since the depth values ​​in the depth map are the depth values ​​of at least some 3D points on the surface of the point cloud from the current viewpoint, the depth values ​​in the depth map can be used as a reference. The difference between the depth value of each 3D point in the point cloud and the depth value at the mapping position of the 3D point in the depth map is used to determine whether the 3D point is on the surface of the point cloud. After traversing all 3D points in the point cloud, all 3D points on the surface of the point cloud from the current viewpoint can be determined.

[0083] It should be noted that, considering that the depth values ​​between adjacent pixels are generally small, the mapping position of each three-dimensional point in the point cloud determined above in the depth map can be a precise position or a rough position that allows for a preset error, that is, a position within a certain range around the precise position. This embodiment does not impose any limitations on this.

[0084] For example, if the first and second mapping processes are exactly the same, then the mapping position of each 3D point in the point cloud determined by the second mapping process in the depth map is an exact position. If the first and second mapping processes are different, for example, the first mapping process determines the mapping position through OpenGL rasterization, while the second mapping process calculates the position in the vertex shader, then the mapping position of each 3D point in the point cloud determined by the second mapping process in the depth map may be a coarse position with errors, that is, the mapping position may be a position within a certain range around the exact position.

[0085] For example, for each 3D point in the point cloud, if the difference between the depth value of the 3D point and the depth value of its mapped position in the depth map is less than a preset threshold, then the 3D point can be determined to be a 3D point on the surface of the point cloud from the current viewpoint; otherwise, the 3D point is determined to be a 3D point on a non-surface part of the point cloud from the current viewpoint. After traversing all the 3D points, all 3D points on the surface of the point cloud from the current viewpoint can be determined. The preset threshold can be specifically set according to the actual application scenario; that is, if the depth value of the 3D point is not significantly different from the depth value of its mapped position in the depth map, then the 3D point can be considered to be on the surface of the point cloud.

[0086] In other embodiments, before the aforementioned depth test value, the 3D points corresponding to each pixel in the depth map can be determined in advance based on the first mapping result, and the determined information can be stored in the depth map. For example, each pixel in the depth map can store a minimum depth value and a list indicating all 3D points mapped to that pixel. For example, the list includes the ID (unique identifier) ​​of each 3D point mapped to that pixel and its depth value. After acquiring the depth map, the electronic device can traverse the list of each pixel in the depth map and determine whether the 3D point is on the surface based on the difference between the depth value of each 3D point indicated by the list and the depth value of the pixel, thereby determining all 3D points on the surface of the point cloud from the current viewpoint. For example, when the electronic device traverses each 3D point indicated by the list of each pixel in the depth map, if the difference between the depth value of the 3D point and the minimum depth value of the pixel is less than a preset threshold, the 3D point is determined to be a 3D point on the surface of the point cloud from the current viewpoint; otherwise, the 3D point is determined to be a 3D point on a non-surface of the point cloud from the current viewpoint. For example, each pixel's list stores the depth values ​​of all 3D points mapped to that pixel. After sorting all the depth values ​​mapped to that pixel in ascending order of depth, the other depth values ​​in the list (excluding the minimum depth value) are compared with the minimum depth value. If the difference between the two is less than a preset threshold, the 3D point corresponding to that other depth value is determined to be a 3D point on the point cloud surface from the current viewpoint. In this embodiment, by storing information in list form, the number of point cloud mapping operations can be reduced, thereby improving efficiency.

[0087] After determining all 3D points on the point cloud surface from the current viewpoint, the electronic device identifies a target 3D point within the selection area indicated by the selection operation from all 3D points on the point cloud surface from the current viewpoint, and then performs a preset operation on the target 3D point. The preset operation includes, but is not limited to, at least one of the following: displaying the target 3D point on the display interface with a specified color and / or a specified texture, setting the target 3D point to a selected state, deleting the target 3D point, dragging the target 3D point to a preset position, performing a deformation operation on the target 3D point, performing a carving operation on the target 3D point, etc.

[0088] In one example, see, for example Figure 3A The electronic device can display the target 3D point on a display interface with a specified color and / or a specified texture. In another example, the electronic device can modify the state of the target 3D point, such as changing the target 3D point from an unselected state to a selected state, or changing the target 3D point from a selected state to an unselected state. In yet another example, the electronic device can delete the target 3D point.

[0089] It should be noted that determining whether a 3D point is on the point cloud surface from the current viewpoint, and determining whether a 3D point on the point cloud surface is within the selection area indicated by the selection operation, can be performed in two separate traversals or in one traversal. This embodiment does not impose any restrictions on this.

[0090] For example, the target 3D point can be determined in two traversals. The first traversal traverses all 3D points in the point cloud to determine whether each 3D point is on the point cloud surface from the current viewpoint. The second traversal traverses all 3D points on the point cloud surface from the current viewpoint to determine whether each 3D point on the point cloud surface is within the selection area indicated by the selection operation, thereby determining the target 3D point.

[0091] For example, the target 3D point can be determined during a single traversal. When traversing each 3D point in the point cloud, if it is determined that the 3D point is on the surface of the point cloud from the current viewpoint, it can be further determined whether the 3D point is within the selection area indicated by the selection operation, thereby determining the target 3D point.

[0092] It is easy to understand that the solutions described in the above embodiments can be combined when there is no conflict, and not all of them will be listed in this disclosure.

[0093] Figure 5 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into memory 508 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0094] Please refer to Figure 6 Point cloud processing devices can be applied to, for example... Figure 5 The device shown is used to implement the technical solution of this specification. The point cloud processing apparatus may include:

[0095] The depth map acquisition module 201 is used to respond to the selection operation of the point cloud, and map the point cloud from three-dimensional space to two-dimensional space according to the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points to obtain a depth map under the current view; wherein, the depth value in the depth map is the depth value of at least some three-dimensional points on the surface of the point cloud under the current view.

[0096] The mapping position determination module 202 is used to determine the mapping position of each 3D point in the point cloud in the depth map;

[0097] The surface 3D point determination module 203 is used to determine all 3D points on the surface of the point cloud from the current viewpoint based on the difference between the depth value of each 3D point in the point cloud and the depth value of the 3D point at the mapping position in the depth map.

[0098] The target 3D point determination module 204 is used to determine the target 3D point within the selection area indicated by the selection operation from all 3D points on the point cloud surface from the current viewpoint.

[0099] In some embodiments, the surface three-dimensional point determination module 203 is specifically used to determine, for each three-dimensional point in the point cloud, if the difference between the depth value of the three-dimensional point and the depth value of the three-dimensional point at the mapping position in the depth map is less than a preset threshold, that the three-dimensional point is a three-dimensional point on the surface of the point cloud from the current viewpoint; otherwise, that the three-dimensional point is a three-dimensional point on the non-surface of the point cloud from the current viewpoint.

[0100] In some embodiments, the depth map acquisition module 201 includes a display size determination unit and a mapping unit. The display size determination unit determines the display size of each 3D point based on the distance between each 3D point in the point cloud and its adjacent 3D points. The display size ensures that the 3D point and its adjacent 3D points are displayed without gaps. The mapping unit maps the point cloud, with the determined display size of each 3D point, from 3D space to 2D space to obtain a depth map from the current viewpoint; wherein the pixels in the depth map are the mapping results of some 3D points on the surface of the point cloud from the current viewpoint.

[0101] In some embodiments, the distance between each 3D point in the point cloud and its adjacent 3D points is a preset value; wherein, if the point cloud is a point cloud with uniform gaps, the preset value is determined based on the statistical values ​​of the distances between each 3D point in the point cloud and its adjacent 3D points; if the point cloud is a point cloud with non-uniform gaps, the preset value is determined based on the adjacency conditions indicated by the business scenario of the point cloud.

[0102] In some embodiments, the display size of each 3D point is determined based on the distance between the 3D point and its neighboring 3D points, and the conversion relationship between the distance in 3D space and the number of pixels in 2D space.

[0103] In some embodiments, the mapping unit is specifically used to map a point cloud, whose display size for each three-dimensional point is determined, from three-dimensional space to two-dimensional space to obtain a first mapping result; and to perform a depth test on the first mapping result to obtain a depth map under the current viewpoint.

[0104] In some embodiments, the mapping position determination module 202 is specifically used to remap the point cloud from three-dimensional space to two-dimensional space to obtain a second mapping result; and to determine the mapping position of each three-dimensional point in the point cloud in the depth map based on the second mapping result.

[0105] In some embodiments, a target 3D point processing module is further included, which is used to display the target 3D point on a display interface with a specified color and / or a specified texture; or modify the state of the target 3D point; or delete the target 3D point.

[0106] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0107] Accordingly, embodiments of this application also provide an electronic device, including a memory, a processor, and executable instructions stored in the memory and executable on the processor;

[0108] The processor executes the executable instructions to implement the steps in the above method.

[0109] For example, the processor includes, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0110] For example, the memory may include at least one type of storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc.

[0111] Accordingly, this application also provides a computer program product, including a computer program that, when executed by a processor, is used to implement the above-described method.

[0112] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions that can be executed by a processor of the device to perform the described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0113] A non-transitory computer-readable storage medium that, when instructions in the storage medium are executed by a terminal's processor, enables the terminal to perform the methods described above.

[0114] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0115] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0116] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0117] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0118] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0119] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0120] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0121] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method of point cloud processing, the method comprising: include: In response to a selection operation on the point cloud, the point cloud is mapped from three-dimensional space to two-dimensional space based on the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points to obtain a depth map from the current viewpoint; wherein the depth values ​​in the depth map are the depth values ​​of at least some of the three-dimensional points on the surface of the point cloud from the current viewpoint. Determine the mapping position of each 3D point in the point cloud in the depth map; Based on the difference between the depth value of each 3D point in the point cloud and the depth value of the 3D point at its mapped position in the depth map, all 3D points on the surface of the point cloud at the current viewpoint are determined. From all the three-dimensional points on the point cloud surface from the current perspective, determine the target three-dimensional point that is within the selection area indicated by the selection operation; The step of mapping the point cloud from three-dimensional space to two-dimensional space based on the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points to obtain a depth map from the current viewpoint includes: The display size of each three-dimensional point is determined based on the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points. The display size is used to ensure that there are no gaps between the three-dimensional point and its adjacent three-dimensional points when they are displayed. The point cloud, whose display size for each 3D point is determined, is mapped from 3D space to 2D space to obtain a depth map from the current viewpoint; wherein, the pixels in the depth map are the mapping results of some 3D points on the surface of the point cloud from the current viewpoint.

2. The method according to claim 1, characterized in that, The step of determining all three-dimensional points on the surface of the point cloud from the current viewpoint based on the difference between the depth value of each three-dimensional point in the point cloud and the depth value of the three-dimensional point at its mapped position in the depth map includes: For each 3D point in the point cloud, if the difference between the depth value of the 3D point and the depth value of the 3D point at its mapping position in the depth map is less than a preset threshold, the 3D point is determined to be a 3D point on the surface of the point cloud from the current viewpoint; otherwise, the 3D point is determined to be a 3D point on a non-surface of the point cloud from the current viewpoint.

3. The method according to claim 1, characterized in that, The distance between each 3D point in the point cloud and its adjacent 3D points is a preset value; Wherein, if the point cloud is a point cloud with uniform gaps, the preset value is determined based on the statistical values ​​of the distances between each three-dimensional point in the point cloud and its adjacent three-dimensional points. If the point cloud has uneven spacing, the preset value is determined based on the adjacency conditions indicated by the business scenario of the point cloud.

4. The method according to claim 1, characterized in that, The display size of each 3D point is determined based on the distance between the 3D point and its neighboring 3D points, as well as the conversion relationship between the distance in the pre-stored 3D space and the number of pixels in the 2D space.

5. The method according to claim 1, characterized in that, The process of mapping the point cloud, which determines the display size of each 3D point, from 3D space to 2D space to obtain a depth map from the current viewpoint includes: The point cloud, with the display size of each 3D point determined, is mapped from 3D space to 2D space to obtain the first mapping result; Perform a depth test on the first mapping result to obtain a depth map from the current viewpoint.

6. The method according to claim 1, characterized in that, Determining the mapping position of each 3D point in the point cloud in the depth map includes: The point cloud is then mapped from three-dimensional space back to two-dimensional space to obtain a second mapping result; Based on the second mapping result, the mapping position of each 3D point in the point cloud in the depth map is determined.

7. The method according to claim 1, characterized in that, Also includes: Display the target 3D point on the display interface with a specified color and / or a specified texture; or Set the target 3D point to the selected state; or Delete the target 3D point.

8. A point cloud processing device, characterized in that, include: The depth map acquisition module is used to respond to the selection operation of the point cloud, and map the point cloud from three-dimensional space to two-dimensional space according to the distance between each three-dimensional point in the point cloud and its adjacent three-dimensional points to obtain a depth map from the current viewpoint; wherein, the depth value in the depth map is the depth value of at least a portion of the three-dimensional points on the surface of the point cloud from the current viewpoint. The mapping position determination module is used to determine the mapping position of each 3D point in the point cloud in the depth map; The surface 3D point determination module is used to determine all 3D points on the surface of the point cloud from the current viewpoint based on the difference between the depth value of each 3D point in the point cloud and the depth value of the 3D point at the mapping position in the depth map. The target 3D point determination module is used to determine the target 3D point within the selection area indicated by the selection operation from all 3D points on the point cloud surface from the current viewpoint. The depth map acquisition module is specifically used to determine the display size of each 3D point based on the distance between each 3D point in the point cloud and its adjacent 3D points. The display size is used to ensure that there are no gaps between the 3D point and its adjacent 3D points when displayed. The point cloud with the determined display size of each 3D point is mapped from 3D space to 2D space to obtain a depth map from the current viewpoint. The pixels in the depth map are the mapping results of some 3D points on the surface of the point cloud from the current viewpoint.

9. An electronic device, characterized in that, This includes memory, processor, and executable instructions stored in memory and capable of running on the processor; Wherein, when the processor executes the executable instructions, it implements the steps in the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images