Method, apparatus, scanner and scanning system for point cloud data processing

Abstract

This application provides a method, apparatus, scanner, and scanning system for point cloud data processing, relating to the field of point cloud data processing. The method includes: calculating the gravity direction in the camera coordinate system based on IMU data; determining a first set of planes based on acquired first point cloud data; filtering a second set of planes from the first set of planes based on the gravity direction, wherein the angle between the normal vector of the planes in the second set and the gravity direction is within a preset angle range; and removing point cloud data corresponding to the first and second planes from the first point cloud data, wherein in the second set of planes, the first plane is the plane with the greatest vertical distance to the depth camera, and the second plane is the plane parallel to the first plane and closest to it in terms of vertical distance. This method has no specific requirements on the shape of the turntable and will not incorrectly remove the planar structure of the object to be scanned.

Description

Technical Field

[0001] This application relates to the field of scanning technology, and more specifically, to methods, apparatus, scanners, and scanning systems for point cloud data processing. Background Technology

[0002] Scanners are widely used in fields such as 3D printing, object reconstruction, and reverse engineering, specifically for scanning target objects from all angles.

[0003] Currently, to improve scanning efficiency, the target object is typically placed on a rotating device. This device rotates the object within the scanner's field of vision, allowing for a comprehensive scan. The rotating device is usually a turntable mounted on a substrate, upon which the target object rests. While rotating devices increase scanning efficiency, they can also negatively impact the target object; for example, they might misidentify the substrate as part of the target object.

[0004] Therefore, how to accurately remove the image data corresponding to the rotating device from the image data acquired by the scanner is an urgent problem to be solved at this stage. Summary of the Invention

[0005] This application provides a method, apparatus, scanner, and scanning system for point cloud data processing, which can remove the point clouds corresponding to the base plane and the turntable plane from the first point cloud data in the point cloud data acquired by the scanner, thereby helping to improve the recognition accuracy of target objects.

[0006] In a first aspect, a method for processing point cloud data is provided, applied to a scanner. The scanner includes a depth camera equipped with an inertial measurement unit (IMU). The method includes: acquiring first point cloud data and IMU data, wherein the first point cloud data is used to represent a target object, the target object including an object to be scanned, a substrate, and a turntable, wherein the turntable is located on the substrate and the object to be scanned is located on the turntable; calculating the direction of gravity in the camera coordinate system based on the IMU data; determining a first set of planes based on the first point cloud data; selecting a second set of planes from the first set of planes based on the direction of gravity, wherein the angle between the normal vector direction of the planes in the second set of planes and the direction of gravity is within a preset angle range; removing the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second set of planes that is farthest from the depth camera in perpendicular distance, and the second plane is the plane in the second set of planes that is parallel to the first plane and has the shortest perpendicular distance to the first plane, and the first target point cloud data is used to identify the object to be scanned.

[0007] For example, depth image data including the target object can be acquired using a depth camera, and the first point cloud data can be determined based on the depth image data and the camera intrinsics of the depth camera.

[0008] It should be understood that camera intrinsics can be obtained directly from a depth camera and are known parameters.

[0009] For example, the aforementioned base can be a pedestal, the ground, or other platform used to hold objects.

[0010] For example, the first point cloud data can be detected using a plane detection algorithm to determine a first set of planes. Therefore, the first set of planes can include all detected planes related to the target object. This plane detection algorithm can be a random sample consensus (RANSAC) algorithm.

[0011] For example, ideally, the preset angle range could be a specific angle value: 0°. However, in practical applications, the base plane is usually not absolutely perpendicular to the direction of gravity in the camera coordinate system. Therefore, the angle between the normal vector of the base plane and the direction of gravity should be within a reasonable angle range, i.e., the preset angle range, for example, within the angle range of [-5, 5]°.

[0012] For example, after determining the first target point cloud data, the first target point cloud data can be sent to a downstream functional module, which then identifies the object to be identified based on the first target point cloud data. This downstream functional module can be a functional module of a processor used in 3D printing, object reconstruction, or reverse engineering scenarios.

[0013] For example, when the number of second planes is 0, the point cloud corresponding to the first plane can be removed from the first point cloud data to obtain the second target point cloud data. It should be understood that the second target point cloud data can also be used to identify the object to be scanned, but the accuracy of the identification result based on the second target point cloud data is relatively low. It can be combined with the subsequently determined target point cloud data to perform joint identification of the object to be scanned.

[0014] Based on the above technical solution, the gravity direction of the camera coordinate system is calculated using the acquired IMU data. Based on this gravity direction, point clouds corresponding to the base plane and the turntable plane are progressively filtered out from the point cloud data, and these point clouds corresponding to these two planes are removed from the point cloud data. This method has no specific requirements on the type and shape of the turntable, thus ensuring its versatility. Furthermore, in the process of removing point clouds corresponding to the base plane and the turntable plane, the planar structure of the object being scanned itself is not mistakenly removed, thus ensuring the accuracy of the method.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, when the first plane set is empty, the second point cloud data is obtained; based on the second point cloud data, the first plane set is determined.

[0016] Based on the above technical solution, even if the first plane set is empty, the point cloud data processing process will not be blocked, and the point cloud data can be reacquired to continue the point cloud data processing process.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, before acquiring the second point cloud data, the pose of the depth camera is adjusted so that the first set of planes determined based on the second point cloud data is not empty.

[0018] Based on the above technical solution, the problem of inaccurate plane detection caused by improper pose of the depth camera can be effectively avoided.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the first plane is the plane corresponding to the base, and the second plane is the plane corresponding to the turntable.

[0020] Secondly, a point cloud data processing apparatus is provided for use in a scanner. The scanner includes a depth camera equipped with an IMU. The apparatus includes: an acquisition unit for acquiring first point cloud data and IMU data, wherein the first point cloud data represents a target object, the target object including an object to be scanned, a substrate, and a turntable, wherein the turntable is located on the substrate and the object to be scanned is located on the turntable; a processing unit for calculating the direction of gravity in the camera coordinate system based on the IMU data; determining a first set of planes based on the first point cloud data; filtering a second set of planes from the first set of planes based on the direction of gravity, wherein the angle between the normal vector direction of the planes in the second set of planes and the direction of gravity is within a preset angle range; removing point cloud data corresponding to the first plane and point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second set of planes that is farthest from the depth camera in perpendicular distance, and the second plane is the plane in the second set of planes that is parallel to the first plane and has the shortest perpendicular distance to the first plane, and the first target point cloud data is used to identify the object to be scanned.

[0021] In conjunction with the second aspect, in some implementations of the second aspect, when the first plane set is empty, the above-mentioned acquisition unit is specifically used to: acquire the second point cloud data; the above-mentioned processing unit is specifically used to: determine the first plane set based on the second point cloud data.

[0022] In conjunction with the second aspect, in some implementations of the second aspect, the above-mentioned apparatus further includes: a control unit, configured to adjust the pose of the depth camera before the acquisition unit acquires the second point cloud data, so that the first set of planes determined based on the second point cloud data is not empty.

[0023] In conjunction with the second aspect, in some implementations of the second aspect, the first plane mentioned above is the plane corresponding to the base, and the second plane mentioned above is the plane corresponding to the turntable.

[0024] Thirdly, an apparatus for processing point cloud data is provided, including a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute the method in any possible implementation of the method design of the first aspect above.

[0025] Fourthly, a scanner is provided, which is the device in any possible implementation of the device design of the second aspect above.

[0026] Fifthly, a scanning system is provided, comprising: a scanner, a rotating device, and a processor, wherein the scanner includes a depth camera equipped with an IMU, the rotating device includes a base and a turntable, the turntable being disposed on the base and used to place an object to be scanned; wherein the processor is configured to send a first instruction to the scanner; the scanner is configured to acquire first point cloud data and IMU data according to the first instruction, and send the first point cloud data and IMU data to the processor, the first point cloud data being used to represent a target object, the target object including the object to be scanned, the base, and the turntable; the processor is further configured to calculate the coordinates in the camera coordinate system based on the IMU data. Gravity direction; Based on the first point cloud data, determine the first set of planes; Based on the gravity direction, filter from the first set of planes to obtain the second set of planes, where the angle between the normal vector of the planes in the second set and the gravity direction is within a preset angle range; Remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain the first target point cloud data. The first plane is the plane in the second set that is farthest from the depth camera in perpendicular distance, and the second plane is the plane in the second set that is parallel to the first plane and has the shortest perpendicular distance to the first plane. The first target point cloud data is used to identify the object to be identified.

[0027] For example, when the number of second planes is 0, the processor can simply remove the point cloud corresponding to the first plane from the first point cloud data to obtain the second target point cloud data. It should be understood that the second target point cloud data can also be used to identify the object to be scanned, but the accuracy of the identification result based on the second target point cloud data is relatively low. It can be combined with subsequently determined target point cloud data to perform joint identification of the object to be scanned.

[0028] In conjunction with the fifth aspect, in some implementations of the fifth aspect, when the first plane set is empty, the scanner is specifically used to acquire second point cloud data and send the second point cloud data to the processor; the processor is specifically used to determine the first plane set based on the second point cloud data.

[0029] In conjunction with the fifth aspect, in some implementations of the fifth aspect, before the scanner acquires the second point cloud data, the processor is further configured to: send a second instruction to the scanner, the second instruction being configured to instruct the adjustment of the pose of the depth camera so that the first set of planes determined based on the second point cloud data is not empty.

[0030] In a sixth aspect, a chip system is provided, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and processors are interconnected by lines; the interface circuits are used to receive echo signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes any possible implementation of the method design of the first aspect above.

[0031] In a seventh aspect, a computer-readable storage medium is provided storing a computer program or instructions for implementing the method in any possible implementation of the method design of the first aspect.

[0032] Eighthly, a computer program product is provided, wherein when the computer program code or instructions are executed on a computer, the computer performs the method in any possible implementation of the method design of the first aspect described above. Attached Figure Description

[0033] Figure 1 This is a schematic block diagram of a point cloud data processing method 100 proposed in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram illustrating an application scenario of a point cloud data processing method 100 proposed in an embodiment of this application;

[0035] Figure 3This is a schematic block diagram of a method 300 for removing planar point cloud data provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram showing the vertical distance between the plane of the second plane set proposed in the embodiments of this application and the depth camera;

[0037] Figure 5 This is a schematic diagram showing the perpendicular distance between the plane of the third plane set proposed in the embodiments of this application and the base plane;

[0038] Figure 6 This is a schematic diagram of the process for determining the first target point cloud data according to an embodiment of this application;

[0039] Figure 7 This is a schematic block diagram of a point cloud data processing apparatus 700 proposed in an embodiment of this application;

[0040] Figure 8 This is a schematic block diagram of a scanning system 800 proposed in an embodiment of this application;

[0041] Figure 9 This is a schematic block diagram of a computer device 900 according to an embodiment of this application;

[0042] Figure 10 This is a schematic block diagram of a computer-readable storage medium 1000 proposed in an embodiment of this application. Detailed Implementation

[0043] In the description of the embodiments in this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In this application, "at least one" means one or more, and "more" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0044] The use of prefixes such as "first" and "second" in this application embodiment is solely for distinguishing different descriptive objects and does not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes to distinguish descriptive objects in this application embodiment does not constitute a limitation on the described objects. The description of the described objects is found in the claims or the context of the embodiments, and the use of such prefixes should not constitute unnecessary restrictions.

[0045] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0046] In current scanning systems, the target object is typically placed on a rotating device. This device rotates the object within the scanner's field of vision, allowing for efficient omnidirectional scanning and the acquisition of point cloud images of the object facing different directions. The rotating device is usually a turntable mounted on a base, upon which the target object rests. The base can be a pedestal or the ground. During the scanning algorithm's processing, the point clouds corresponding to the base and turntable in the acquired point cloud image need to be removed to achieve more accurate target object reconstruction. Since the point clouds corresponding to the base plane and turntable plane have a significant impact on the target object's scanning and reconstruction, this can specifically involve removing the point clouds corresponding to the base plane and turntable plane. Specifically, the base plane refers to the upper surface of the pedestal, and the turntable plane refers to the upper surface of the turntable.

[0047] For the above application scenarios, the following two methods can be used to remove the point clouds corresponding to the base plane and the turntable plane from the point cloud image.

[0048] Option 1: Use a specially designed turntable.

[0049] The specially designed turntable mentioned above can be a turntable with rich textures and geometric shapes. During the recognition process of point cloud images, objects with rich textures and geometric shapes are relatively easy to identify. Therefore, in the point cloud data processing, specific textures and specific geometric shapes are used as image recognition constraints. When an object with a specific texture and specific geometric shape is identified, the system can determine that the object is the base or the turntable, and then remove the point cloud corresponding to the object from the point cloud image.

[0050] Option 2: Eliminate all circumscribed surfaces using point cloud data processing algorithms.

[0051] Since the point clouds corresponding to the base and turntable in the point cloud image are removed, specifically the point clouds corresponding to the base plane and the turntable plane, the point cloud data processing algorithm can be used to detect the point clouds corresponding to all circumscribed surfaces in the current point cloud image. After removing the point clouds corresponding to all circumscribed surfaces, what remains should be the point cloud corresponding to the target object.

[0052] However, for Scheme 1, since the image recognition constraints corresponding to objects with different textures and geometric shapes differ during point cloud data processing, and the turntables on the market vary in texture and shape, it is impossible to use a universal image recognition constraint to identify different turntables. Therefore, this scheme lacks universality. For Scheme 2, since the target object placed on the turntable may also have a planar structure, directly removing the point cloud corresponding to all circumscribed surfaces in the point cloud image may incorrectly remove the planar structure of the target object itself. Therefore, the current methods for removing the point cloud corresponding to the base plane and the turntable plane from the point cloud image suffer from poor universality and low accuracy.

[0053] In view of this, this application provides a method, apparatus, scanner, and scanning system for point cloud data processing. By acquiring IMU data from a camera, calculating the gravity direction in the camera coordinate system based on the IMU data, and then, based on this gravity direction, progressively filtering out the base plane and turntable plane in the point cloud image, and removing the point clouds corresponding to these two planes from the point cloud data. This method has no specific requirements on the type and shape of the turntable, thus ensuring the method's versatility; and during the process of removing the base plane and turntable plane, it avoids mistakenly removing the planar structure of the target object itself, thus ensuring the method's accuracy.

[0054] Figure 1 This is a schematic block diagram of a point cloud data processing method 100 proposed in an embodiment of this application.

[0055] Figure 2 This is a schematic diagram illustrating an application scenario of a point cloud data processing method 100 proposed in an embodiment of this application.

[0056] The method 100 is applied to a scanner that includes a depth camera equipped with an IMU. The depth camera can be a depth camera based on structured light technology, binocular technology, or time-of-flight (TOF) technology, and can simultaneously acquire spatial distance data of the scene and camera pose data.

[0057] S110: Acquire first point cloud data and IMU data.

[0058] The aforementioned first point cloud data is used to represent the target object, which is an object included in the image captured by the depth camera at the current pose. (Reference) Figure 2 As shown, the target object includes the object to be scanned, a substrate, and a turntable, wherein the turntable is located on the substrate and the object to be scanned is located on the turntable.

[0059] In some possible embodiments, depth image data including the target object can be acquired by a depth camera, and the first point cloud data can be determined based on the depth image data and the camera intrinsic parameters of the depth camera.

[0060] In some possible embodiments, the first point cloud data may include all image elements in the scene corresponding to the depth image data, that is, in addition to the object to be scanned, the base and the turntable, it may also include other environmental objects; or it may only include image elements corresponding to the region of interest (ROI) in the depth image data, that is, the depth image data may include only the object to be scanned, the base and the turntable as much as possible.

[0061] It should be understood that camera intrinsics can be obtained directly from a depth camera and are known parameters.

[0062] It should be understood that, due to the perspective of a depth camera in a specific pose, the captured images may contain missing elements of the target scene. For example, if the depth camera is shooting the object to be scanned vertically, and the top area of ​​the object is large, it may obstruct the turntable, resulting in the turntable plane not appearing in the captured image. For details on how to handle this situation, please refer to the description of the corresponding embodiments below.

[0063] In some possible embodiments, the aforementioned base can be a pedestal, the ground, or other platforms for holding objects; this application does not limit this.

[0064] S120: Calculate the direction of gravity in the camera coordinate system based on IMU data.

[0065] In some possible embodiments, since the IMU is mounted on the depth camera, the direction of gravity in the camera coordinate system of the depth camera can be calculated from the IMU data (see reference). Figure 2 The gravity direction in the camera coordinate system is established with the depth camera as the origin. In the process of 3D reconstruction, the origin of the world coordinate system coincides with that of the camera coordinate system. The point cloud data is the 3D data under the camera coordinate system. Therefore, it is only necessary to calculate the gravity direction under the camera coordinate system. Based on the gravity direction, the plane in the point cloud data can be identified.

[0066] S130: Determine the first plane set based on the first point cloud data mentioned above.

[0067] In some possible embodiments, the first point cloud data can be detected using a plane detection algorithm to determine a first set of planes. This plane detection algorithm can be the RANSAC algorithm. Therefore, the first set of planes can include all detected planes related to the target object.

[0068] It should be understood that the aforementioned first set of planes may specifically include the base plane, the turntable plane, and the planes that the object to be scanned may include.

[0069] In some possible embodiments, the first set of planes may include multiple planes, one plane, or even zero planes. For the processing flow when the first set of planes includes one or zero planes, please refer to the corresponding embodiments below. When the first set of planes includes multiple planes, the following S140 can be executed.

[0070] S140: Based on the direction of gravity, select a second set of planes from the first set of planes. The angle between the normal vector of the plane in the second set of planes and the direction of gravity is within a preset angle range.

[0071] It should be understood that since the base is usually the ground or a base placed horizontally on the ground, the direction of the normal vector of the base plane usually coincides with the direction of gravity, or there is an angle between the direction of the normal vector and the direction of gravity within a certain range.

[0072] In some possible embodiments, ideally, the preset angle range could be a specific angle value: 0°. However, in practical applications, the base plane is usually not absolutely perpendicular to the direction of gravity in the camera coordinate system. Therefore, the angle between the normal vector of the base plane and the direction of gravity should be within a reasonable angle range, i.e., the preset angle range, for example, within the range of [-5, 5]°. The turntable is usually placed horizontally on the base, so the turntable plane is usually parallel to the base plane. Therefore, the angle between the normal vector of the turntable plane and the direction of gravity should also be within the preset angle range. Thus, provided the depth camera pose is reasonable and the turntable plane is not obstructed, the second set of planes selected based on the above conditions can simultaneously include the base plane and the turntable plane.

[0073] S150: Remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain the first target point cloud data. The first plane is the plane in the second set of planes that is furthest from the depth camera in terms of vertical distance, and the second plane is the plane in the second set of planes that is parallel to the first plane and has the closest vertical distance to it.

[0074] It should be understood that the first plane can be the base plane, the second plane can be the turntable plane, and the first target point cloud data is used to identify the object to be scanned.

[0075] In some possible embodiments, both the first plane and the second plane are selected from the set of second planes. Therefore, based on the spatial location information of the planes in the set of second planes—that is, the vertical distance from the plane to the depth camera or the vertical distance between the planes—the set of second planes is filtered. The filtering result can include the following five cases:

[0076] 1) Select one first plane from the set of second planes;

[0077] 2) Select multiple first planes from the set of second planes;

[0078] 3) Select one first plane and one second plane from the set of second planes;

[0079] 4) Select multiple first planes and 1 second plane from the set of second planes;

[0080] 5) Select multiple first planes and multiple second planes from the set of second planes.

[0081] It should be understood that the reason multiple first planes, or multiple base planes, are selected is because the base can be the ground, and the ground has a large planar area. Therefore, the ground is divided into multiple sub-planes, and these sub-planes are the multiple first planes that can be selected. The reason for selecting multiple second planes is the same. In general, at least one first plane and at least one second plane can be selected from the above set of second planes.

[0082] In some possible embodiments, after determining the first target point cloud data, the first target point cloud data can also be sent to a downstream functional module, which identifies the object to be identified based on the first target point cloud data. This downstream functional module can be a functional module of a processor used in 3D printing, object reconstruction, or reverse engineering scenarios.

[0083] Based on the above technical solution, by acquiring IMU data, the gravity direction of the camera coordinate system is calculated. Based on this gravity direction, point clouds corresponding to the base plane and the turntable plane are progressively filtered from the point cloud data, and these point clouds corresponding to these two planes are removed from the point cloud data. This method has no specific requirements on the type and shape of the turntable, thus ensuring its versatility. Furthermore, in the process of removing point clouds corresponding to the base plane and the turntable plane, the planar structure of the object being scanned itself is not mistakenly removed, thus ensuring the accuracy of the method.

[0084] In some possible embodiments, the first set of planes can be filtered using the following method.

[0085] Figure 3 This is a schematic block diagram of a method 300 for removing planar point cloud data provided in an embodiment of this application.

[0086] S310: Select a second set of planes from the first set of planes, wherein the angle between the normal vector of the plane in the second set of planes and the direction of gravity is within a preset angle range.

[0087] In some possible embodiments, based on the spatial relationship between the base plane and the depth camera, it is known that the depth camera usually takes pictures of the target scene from above, so the depth camera is usually located on the base. Therefore, in the process of filtering out the second set of planes, only the planes whose spatial position is below the horizontal position of the depth camera can be filtered, thereby reducing the computational overhead caused by plane filtering.

[0088] S320: Determine the plane in the second set of planes that is furthest from the depth camera in terms of vertical distance as the first plane.

[0089] In some possible embodiments, reference Figure 4 As shown, when calculating the vertical distance between each plane in the second plane set and the depth camera, the depth camera can be abstracted as a depth camera mass point A. The vertical distance between each plane in the second plane set and the depth camera can specifically be the vertical distance L between the center O of each plane in the second plane set and the depth camera mass point A.

[0090] In some possible embodiments, when the number of planes included in the second set of planes is equal to 1, for ease of description, the planes included in the second set of planes are referred to as candidate planes. Since there is only 1 candidate plane, the candidate plane must also be the candidate plane with the farthest vertical distance from the depth camera. There is no process of comparing the above-mentioned vertical distances, and the candidate plane can be directly determined as the first plane, that is, determined as the base plane.

[0091] The reason for selecting the base plane rather than the turntable plane as the candidate plane is as follows: When the base is a pedestal, the area of ​​the base plane is usually larger than that of the turntable plane to ensure the stability of the rotation of the turntable and the object to be scanned. When the base is the ground or other platform, the area of ​​the base plane will be much larger than that of the turntable plane. Therefore, in either case, even if the object to be scanned is photographed perpendicularly, the object will at most obscure the turntable plane, not the base plane. Furthermore, when the depth camera is photographing the target object, its position is usually not directly above the target object. Therefore, in typical application scenarios, neither the base plane nor the turntable plane will usually be obscured. If the number of planes in the second plane set is greater than one, it is necessary to compare the vertical distances of each plane in the second plane set from the depth camera, and determine the plane with the greatest vertical distance from the depth camera as the base plane.

[0092] S330: Remove the point cloud data corresponding to the first plane from the first point cloud data.

[0093] S340: Determine the second plane from the set of second planes as the plane that is parallel to the first plane and has the closest perpendicular distance to the first plane.

[0094] S340 above can be broken down into two sub-steps. First, based on the first plane, a third set of planes is obtained by filtering from the second set of planes, where all planes in the third set are parallel to the first plane. Then, the plane in the third set that has the closest vertical distance to the first plane is determined as the second plane. Furthermore, "closest vertical distance" means that the plane that is closest to the first plane outside of the minimum distance is determined as the minimum plane, ensuring that the selected second plane does not belong to the first plane.

[0095] Figure 5 This is a schematic diagram showing the perpendicular distance between the plane of the third plane set proposed in the embodiments of this application and the first plane.

[0096] In some possible embodiments, reference Figure 5 As shown, when calculating the perpendicular distance between each plane in the third plane set and the first plane (base plane), the center O of the first plane can be used as the reference to calculate the perpendicular distance M between the center P of each plane in the third plane set and the center O. When there are multiple first planes, the center O of any one of the first planes can be selected as the reference.

[0097] It should be understood that since the turntable is placed on the base, the base plane (first plane) and the turntable plane (second plane) are parallel. Therefore, the selected set of third planes includes the second plane. Furthermore, because the turntable is placed on the base, the selection of the third set of planes can be performed only on planes spatially located above the first plane, thus reducing the computational overhead of plane selection.

[0098] In some possible embodiments, considering the possibility of missed detections during the plane detection process, a third plane set can be obtained by filtering from the first plane set. Alternatively, the first plane set can be re-determined using the plane detection algorithm, and then the third plane set can be obtained by filtering from the first plane set.

[0099] S350: Remove the point cloud data corresponding to the second plane from the first point cloud data.

[0100] In some possible embodiments, after S320 is completed, the process can directly jump to S340, and after S340 is completed, the following steps are performed:

[0101] S360: Remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data.

[0102] It should be understood that since the turntable is placed directly on the base, the turntable should be the object closest to the base in the entire target scene. Therefore, the turntable plane should be the plane with the closest vertical distance to the base plane in the third plane set.

[0103] Based on the above technical solution, the base plane and turntable plane in the target scene can be accurately identified, so that the point clouds corresponding to the base plane and turntable plane can be removed from the first point cloud data, thereby increasing the accuracy of target scanning.

[0104] In some possible embodiments, since the vertical distance between the second plane and the first plane is the shortest compared to the other planes in the third plane set, this application also proposes a method for fuzzy plane detection. This method can replace S340 in the above method 300, and the method can be:

[0105] The plane in the second set of planes that is the second furthest from the depth camera in perpendicular distance is identified as the second plane. It is evident that the plane selection constraints of this method are more relaxed and simpler than those of S340 above. Therefore, the computational cost of selecting the second plane is lower, and it does not significantly reduce the accuracy of selecting the second plane, achieving a good trade-off between computational cost and plane detection accuracy.

[0106] In some possible embodiments, if the number of planes parallel to the first plane in the second plane set is 0 (i.e., the second plane is not included in the second plane set), or if the number of planes in the aforementioned third plane set is 0, it indicates that no turntable plane has been identified. In this case, the point cloud data corresponding to the first plane can be removed from the first point cloud data to obtain the second target point cloud data. It should be understood that the second target point cloud data can also be used to identify the object to be scanned, but the accuracy of the identification result based on the second target point cloud data is relatively low. It can be combined with the subsequently determined target point cloud data for joint identification of the object to be scanned.

[0107] Therefore, even if the second plane is not identified based on the above method 300 and the corresponding embodiment, the point cloud data processing flow will not be blocked, but will continue to process the next frame of point cloud data.

[0108] In some possible embodiments, if the first plane set is empty, second point cloud data is acquired; based on the second point cloud data, the aforementioned first plane set is determined. Then, the subsequent plane filtering process in method 300 is performed based on the newly determined first plane set.

[0109] It should be understood that when the number of planes in the first plane set is 0, it means that no plane can be detected based on the current point cloud data. Therefore, the processing flow for the next frame of point cloud data can be directly initiated. In this case, the parameters of the plane detection algorithm or the pose of the depth camera can be adjusted to ensure that the first plane set determined based on the second point cloud data is not empty. Based on this, the problem of inaccurate plane detection caused by improper depth camera pose or abnormal plane detection algorithm parameters can be effectively avoided.

[0110] In some possible embodiments, when the number of planes in the first set of planes is 1, it is first necessary to determine whether the normal vector direction of the plane is parallel to the direction of gravity. If it is parallel to the direction of gravity, the plane can be determined as the first plane, and then the point cloud corresponding to the first plane can be directly removed from the first point cloud data to obtain the second target point cloud data. If it is not parallel to the direction of gravity, the second point cloud data is obtained, and the first set of planes is re-determined based on the second point cloud data. Then, the subsequent plane filtering process in method 300 is performed based on the re-determined first set of planes.

[0111] Furthermore, when the number of planes in the second plane set is 0, it indicates that the subsequent process of identifying the base plane and the turntable plane cannot be executed based on the point cloud data of the current frame. In this case, the process can directly proceed to the processing flow for the point cloud data of the next frame. In this situation, adjustments can also be made to a preset angle range or to adjust the pose of the depth camera.

[0112] It should be understood that the process described in method 300 and its corresponding embodiments is a cyclical process. When the current cyclic step is completed, or when the current cyclic step has not achieved its intended purpose, the processor's execution flow will actively jump out of the current cyclic step and enter the cyclic step for the next frame of point cloud data. That is, there will be subsequent processing of the third point cloud data, the fourth point cloud data, etc., with similar specific processes, which will not be elaborated here.

[0113] In subsequent point cloud data processing, because the depth camera's pose typically remains unchanged while the scanner is scanning the target object, the processor used for point cloud data processing does not need to simultaneously acquire IMU data at that moment. The gravity direction determined during the processing of the first point cloud data can be used for subsequent planar detection operations on the second point cloud data.

[0114] In some possible embodiments, the depth camera receives a first signal at a first moment, indicating that the camera's pose has changed. After the first moment, the depth camera acquires depth image data and IMU data at a second moment, and sends both the depth image data and IMU data to the processor. Upon receiving the reacquired IMU data, the processor recalculates the gravity direction in the camera coordinate system based on the IMU data.

[0115] It should be understood that the steps described in the above methods are merely exemplary steps, and the order of the steps can be reasonably adjusted for different situations. The embodiments of this application do not limit the specific order of the steps. Therefore, the specific process for determining the second point cloud data in this application is as follows.

[0116] Figure 6 This is a schematic diagram of the process for determining the first target point cloud data according to an embodiment of this application.

[0117] In some possible embodiments, the second point cloud data can be determined through the following two approaches.

[0118] Approach 1:

[0119] S611: Acquire image frame data.

[0120] S612: Determine the first point cloud data based on the depth image data and camera intrinsic parameters in the image frame data.

[0121] S613: Calculate the direction of gravity in the camera coordinate system based on the IMU data in the image frame data.

[0122] S614: Determine the first set of planes using a plane detection algorithm.

[0123] If the number of planes in the first plane set is greater than 0, proceed to S615; if the number of planes in the first plane set is equal to 0, proceed to S611.

[0124] S615: Based on the above method 300, determine the base plane.

[0125] If the base plane is determined, proceed to S616; if the base plane is not determined, proceed to S611.

[0126] S616: Based on the above method 300, determine the turntable plane.

[0127] If the turntable plane is determined, proceed to S617; if the turntable plane is not determined, proceed to S618.

[0128] S617: Remove the point clouds corresponding to the base plane and the turntable plane from the first point cloud data to determine the first target point cloud data.

[0129] S618: Remove the point cloud corresponding to the base plane from the first point cloud data to determine the first target point cloud data.

[0130] After executing S617 or S618, the second point cloud data is sent to the downstream functional module, and then the process proceeds to S611.

[0131] Approach 2:

[0132] S611: Acquire image frame data.

[0133] S612: Determine the first point cloud data based on the depth image data and camera intrinsic parameters in the image frame data.

[0134] S613: Calculate the corresponding gravity direction in the camera coordinate system based on the IMU data in the image frame data.

[0135] S614: Determine the first set of planes using a plane detection algorithm.

[0136] If the number of planes in the first plane set is greater than 0, proceed to S625; if the number of planes in the first plane set is equal to 0, proceed to S611.

[0137] S625: Based on the above method 300, determine the base plane.

[0138] If the base plane is determined, proceed to S626; if the base plane is not determined, proceed to S611.

[0139] S626: Remove the point cloud corresponding to the base plane from the first point cloud data to determine the first quasi-target point cloud data.

[0140] S627: Based on the above method 300, determine the turntable plane.

[0141] If the turntable plane is determined, proceed to S628; if the turntable plane is not determined, proceed to S629.

[0142] S628: Remove the point clouds corresponding to the above turntable planes from the first target point cloud data to determine the first target point cloud data.

[0143] S629: The first quasi-target point cloud data is determined as the first target point cloud data.

[0144] After executing S628 or S629, the first target point cloud data is sent to the downstream functional module, and then proceeds to S611.

[0145] Based on the above technical solutions, multiple approaches to determining target point cloud data are provided, thereby increasing the flexibility of the method.

[0146] In addition, embodiments of this application also provide an apparatus for implementing any of the above methods. For example, an apparatus for point cloud data processing is provided, which includes a unit (or means) for implementing any of the above point cloud data processing methods.

[0147] Figure 7 This is a schematic block diagram of a point cloud data processing apparatus 700 provided in an embodiment of this application.

[0148] The device 700 is used in a scanner that includes a depth camera equipped with an IMU. The device 700 includes:

[0149] The acquisition unit 710 is used to acquire first point cloud data and IMU data. The first point cloud data is used to represent the target object. The target object includes the object to be scanned, a substrate, and a turntable, wherein the turntable is located on the substrate and the object to be scanned is located on the turntable.

[0150] The processing unit 720 is used to calculate the gravity direction in the camera coordinate system based on IMU data; determine a first set of planes based on first point cloud data; filter a second set of planes from the first set of planes based on the gravity direction, wherein the angle between the normal vector direction of the planes in the second set of planes and the gravity direction is within a preset angle range; remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second set of planes that is farthest from the depth camera in perpendicular distance, and the second plane is the plane in the second set of planes that is parallel to the first plane and has the shortest perpendicular distance to the first plane, and the first target point cloud data is used to identify the object to be scanned.

[0151] In some possible embodiments, when the first plane set is empty, the acquisition unit 710 is specifically used to: acquire second point cloud data; the processing unit 720 is specifically used to: determine the first plane set based on the second point cloud data.

[0152] In some possible embodiments, the above-described device 700 further includes a control unit 730, configured to adjust the pose of the depth camera before the acquisition unit 710 acquires the second point cloud data, so that the first set of planes determined based on the second point cloud data is not empty.

[0153] This application also proposes a scanner that includes a point cloud data processing apparatus 700 as described in any of the above embodiments.

[0154] This application also proposes a scanning system.

[0155] Figure 8 This is a schematic block diagram of a scanning system 800 proposed in an embodiment of this application.

[0156] The scanning system 800 includes a scanner 810, a substrate 821, a turntable 822, and a processor 830. The scanner 810 includes a depth camera 811 equipped with an IMU 812. The turntable 822 is disposed on the substrate 821 and is used to place the object to be scanned. The substrate 821 can be the ground or a base.

[0157] The processor 830 is used to send the first instruction to the scanner 810.

[0158] The scanner 810 is used to: acquire first point cloud data and IMU data according to a first instruction, and send the first point cloud data and IMU data to the processor 830. The first point cloud data is used to represent a target object, which includes the object to be scanned, a substrate 821, and a turntable 822.

[0159] The processor 830 is used to: calculate the direction of gravity in the camera coordinate system based on IMU data; determine a first set of planes based on first point cloud data; filter a second set of planes from the first set of planes based on the direction of gravity, wherein the angle between the normal vector of the planes in the second set of planes and the direction of gravity is within a preset angle range; remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain the first target point cloud data, wherein the first plane is the plane in the second set of planes that is farthest from the depth camera in perpendicular distance, and the second plane is the plane in the second set of planes that is parallel to the first plane and has the shortest perpendicular distance to the first plane. The first target point cloud data is used to identify the object to be identified.

[0160] The first plane can be the base plane, and the second plane can be the turntable plane.

[0161] In some possible embodiments, the processor 830 described above may include all the functions of the processing unit 730 in the device 700 described above.

[0162] Figure 9 This is a schematic block diagram of a computer device 900 provided in an embodiment of this application. Figure 9 The computer device 900 shown includes a memory 910, a processor 920, and a bus 940. Optionally, the computer device 900 also includes a communication interface 930. The memory 910, processor 920, and communication interface 930 are interconnected via the bus 940.

[0163] The memory 910 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 910 can store programs, and when the program stored in the memory 910 is executed by the processor 920, the processor 920 performs the various steps of the VIO system initialization method of this embodiment. For example, the processor 920 can execute the steps described above. Figures 2 to 6 The method shown.

[0164] The processor 920 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the point cloud data processing method proposed in the method embodiments of this application.

[0165] The processor 920 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the VIO system initialization method of this application can be completed by the hardware integrated logic circuit in the processor 920 or by software instructions.

[0166] The processor 920 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods involved in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 910, and the processor 920 reads the information in memory 910 and completes the execution in conjunction with its hardware. Figure 7 The apparatus shown includes units that are required to perform functions, or to perform the methods described in this application. Figures 1 to 6 The method shown.

[0167] The communication interface 930 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 900 and other devices or communication networks.

[0168] Bus 940 may include a pathway for transmitting information between various components of device 900 (e.g., memory 10, processor 920, communication interface 930).

[0169] It should be understood that although the above-described device 900 only shows a memory, processor, and communication interface, those skilled in the art should understand that in specific implementations, device 900 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that device 900 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that device 900 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 9 All the devices shown.

[0170] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0171] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0172] Figure 10 This is a schematic block diagram of a computer-readable storage medium 1000 provided for an embodiment of this application. Figure 10The computer-readable storage medium 1000 shown stores computer instructions 1010. When executed by a processor, the computer instructions 1010 can implement the methods corresponding to the above embodiments.

[0173] In some possible embodiments, the computer-readable storage medium 1000 can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more sets of available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives (SSDs).

[0174] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0175] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0179] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0180] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing point cloud data, characterized in that, Applied to a scanner, the scanner including a depth camera equipped with an inertial measurement unit (IMU), the method includes: Acquire first point cloud data and IMU data. The first point cloud data is used to represent a target object. The target object includes an object to be scanned, a substrate, and a turntable. The turntable is located on the substrate, and the object to be scanned is located on the turntable. Based on the IMU data, calculate the direction of gravity in the camera coordinate system; Based on the first point cloud data, determine the first plane set; Based on the direction of gravity, a second set of planes is obtained by filtering from the first set of planes, wherein the angle between the normal vector of the plane in the second set of planes and the direction of gravity is within a preset angle range; The first target point cloud data is obtained by removing the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data. The first plane is the plane in the second set of planes that is farthest from the depth camera in terms of vertical distance. The second plane is the plane in the second set of planes that is parallel to the first plane and has the closest vertical distance to the first plane. The first target point cloud data is used to identify the object to be scanned.

2. The method according to claim 1, characterized in that, The step of determining the first plane set based on the first point cloud data includes: If the first plane set is empty, obtain the second point cloud data; The first set of planes is determined based on the second point cloud data.

3. The method according to claim 2, characterized in that, Before acquiring the second point cloud data, the method further includes: The pose of the depth camera is adjusted so that the first set of planes determined based on the second point cloud data is not empty.

4. The method according to any one of claims 1 to 3, characterized in that, The first plane is the plane corresponding to the base, and the second plane is the plane corresponding to the turntable.

5. An apparatus for processing point cloud data, characterized in that, Applied to a scanner, the scanner including a depth camera equipped with an inertial measurement unit (IMU), the device comprising: An acquisition unit is used to acquire first point cloud data and IMU data. The first point cloud data is used to represent a target object. The target object includes an object to be scanned, a substrate, and a turntable. The turntable is located on the substrate, and the object to be scanned is located on the turntable. The processing unit is configured to: calculate the gravity direction in the camera coordinate system based on the IMU data; determine a first set of planes based on the first point cloud data; filter a second set of planes from the first set of planes based on the gravity direction, wherein the angle between the normal vector of the planes in the second set of planes and the gravity direction is within a preset angle range; and remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second set of planes that is farthest from the depth camera in perpendicular distance, and the second plane is the plane in the second set of planes that is parallel to the first plane and has the shortest perpendicular distance to the first plane, and the first target point cloud data is used to identify the object to be scanned.

6. The apparatus according to claim 5, characterized in that, When the first plane set is empty, the acquisition unit is specifically used to: acquire second point cloud data; The processing unit is specifically used to: determine the first plane set based on the second point cloud data.

7. The apparatus according to claim 6, characterized in that, The device further includes: A control unit is configured to adjust the pose of the depth camera before the acquisition unit acquires the second point cloud data, so that the first set of planes determined based on the second point cloud data is not empty.

8. A scanner, characterized in that, Includes the apparatus as described in any one of claims 5 to 7.

9. A scanning system, characterized in that, include: The scanner, rotating device, and processor are provided, wherein the scanner includes a depth camera equipped with an inertial measurement unit (IMU), and the rotating device includes a base and a turntable disposed on the base for placing an object to be scanned. The processor is used to send a first instruction to the scanner; The scanner is used to acquire first point cloud data and IMU data according to the first instruction, and send the first point cloud data and the IMU data to the processor. The first point cloud data is used to represent a target object, and the target object includes the object to be scanned, the substrate, and the turntable. The processor is further configured to: calculate the gravity direction in the camera coordinate system based on the IMU data; determine a first set of planes based on the first point cloud data; filter a second set of planes from the first set of planes based on the gravity direction, wherein the angle between the normal vector direction of the planes in the second set of planes and the gravity direction is within a preset angle range; remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second set of planes that is farthest from the depth camera in perpendicular distance, and the second plane is the plane in the second set of planes that is parallel to the first plane and has the shortest perpendicular distance to the first plane, and the first target point cloud data is used to identify the object to be scanned.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 4.

Images