Point cloud data processing method and apparatus, and scanner and scanning system

Abstract

The present application relates to the field of point cloud data processing. Provided are a point cloud data processing method and apparatus, and a scanner and a scanning system. The method comprises: calculating the direction of gravity in a camera coordinate system on the basis of IMU data; determining a first plane set on the basis of acquired first point cloud data; on the basis of the direction of gravity, screening the first plane set to obtain a second plane set, wherein the included angle between the direction of the normal vector of a plane in the second plane set and the direction of gravity is within a preset included-angle range; and removing from the first point cloud data point cloud data corresponding to a first plane and a second plane, wherein in the second plane set, the first plane is a plane that is furthest from a depth camera in terms of vertical distance, and the second plane is a plane that is parallel to the first plane and is nearest the first plane in terms of vertical distance. In the method, there is no specific requirement for the shape of a turntable, and a plane structure of an object to be scanned may not be mistakenly removed.

Description

Point cloud data processing method, device, scanner and scanning system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 31, 2023, with application number 202311128435.X, and priority to the Chinese patent application entitled “Method, device, scanner and scanning system for point cloud data processing”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of scanning technology, and more specifically, to a method, device, scanner, and scanning system for point cloud data processing. Background Art

[0003] Scanners are widely used in 3D printing, object reconstruction, reverse engineering and other fields, and are specifically used to perform all-round scanning of target objects.

[0004] Currently, to improve scanning efficiency, the target object is typically placed on a rotating device, which rotates the target object within the scanner's field of view, allowing the scanner to scan the target object in all directions. The rotating device is typically a turntable mounted on a substrate, upon which the target object is placed. While the rotating device can increase scanning efficiency, its inclusion can also affect the target object, for example, misidentifying the rotating device or even the substrate as part of the target object.

[0005] In view of this, how to accurately remove the image data corresponding to the rotating device from the image data acquired by the scanner is an urgent problem to be solved at this stage.

[0006] Summary of the Invention

[0007] The present application provides a method, device, scanner and scanning system for processing point cloud data, which can remove the point clouds corresponding to the base plane and turntable plane in the point cloud data obtained by the scanner from the first point cloud data, thereby helping to improve the recognition accuracy of the target object.

[0008] In a first aspect, a method for processing point cloud data is provided, which is applied to a scanner, the scanner including a depth camera equipped with an inertial measurement unit (IMU), the method including: acquiring first point cloud data and IMU data, the first point cloud data being used to represent a target object, the target object including an object to be scanned, a base, and a turntable, wherein the turntable is located on the base and the object to be scanned is located on the turntable; solving the gravity direction in the camera coordinate system based on the IMU data; determining a first plane set based on the first point cloud data; screening a second plane set from the first plane set based on the gravity direction, wherein the angle between the normal vector direction of the plane in the second plane set and the gravity direction is within a preset angle range; eliminating the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is a plane in the second plane set that is farthest from the depth camera in vertical distance, and the second plane is a plane in the second plane set that is parallel to the first plane and closest to the first plane in vertical distance, and the first target point cloud data is used to identify the object to be scanned.

[0009] For example, depth image data including the target object may be acquired through a depth camera, and the first point cloud data may be determined based on the depth image data and the camera intrinsic parameters of the depth camera.

[0010] It should be understood that the camera intrinsic parameters can be directly obtained through the depth camera and are known parameters.

[0011] For example, the base may be a pedestal, or the ground, or other platforms for placing objects.

[0012] For example, the first point cloud data may be detected using a plane detection algorithm to determine a first plane set, whereby the first plane set may include all planes detected about the target object. The plane detection algorithm may be a random sample consensus (RANSAC) algorithm.

[0013] For example, under ideal conditions, the preset angle range can be a specific angle value: 0°. However, in practical applications, the base plane is usually not absolutely perpendicular to the direction of gravity in the camera coordinate system. Therefore, the angle between the base plane normal vector and the direction of gravity should be within a reasonable angle range, that is, the preset angle range, for example, the angle range of [-5, 5]°.

[0014] For example, after determining the first target point cloud data, the first target point cloud data may be sent to a downstream functional module, which then identifies the object to be identified based on the first target point cloud data. The downstream functional module may be a functional module of a processor used in 3D printing, object reconstruction, or reverse engineering scenarios.

[0015] For example, if the number of second planes is zero, only the point cloud corresponding to the first plane can be removed from the first point cloud data to obtain the second target point cloud data. It should be understood that this second target point cloud data can also be used to identify the object to be scanned, but the accuracy of the recognition result based on the second target point cloud data is relatively low. It can be combined with the subsequently determined target point cloud data to perform joint recognition of the object to be scanned.

[0016] Based on the above technical solution, the direction of gravity in the camera coordinate system is calculated using acquired IMU data. Based on this gravity direction, the point clouds corresponding to the base plane and turntable plane in the point cloud data are gradually filtered out and then removed from the point cloud data. This method has no specific requirements for the type and shape of the turntable, thus ensuring its versatility. In addition, when removing point clouds corresponding to the base plane and turntable plane, the planar structure of the object being scanned is not mistakenly removed, thus ensuring the accuracy of the method.

[0017] In combination with the first aspect, in certain implementations of the first aspect, when the first plane set is empty, second point cloud data is acquired; and the first plane set is determined based on the second point cloud data.

[0018] Based on the above technical solution, even when the above first plane set is empty, the point cloud data processing process will not be blocked, and the point cloud data can be reacquired and the point cloud data processing process can continue.

[0019] In combination with the first aspect, in some implementations of the first aspect, before acquiring the second point cloud data, the posture of the depth camera is adjusted so that the first plane set determined according to the second point cloud data is not empty.

[0020] Based on the above technical solution, the problem of being unable to accurately detect planes due to improper depth camera posture can be effectively avoided.

[0021] In combination with the first aspect, in some implementations of the first aspect, the first plane is a plane corresponding to the base, and the second plane is a plane corresponding to the turntable.

[0022] In a second aspect, a device for processing point cloud data is provided, which is applied to a scanner, wherein the scanner includes a depth camera equipped with an IMU, and the device includes: an acquisition unit, configured to acquire first point cloud data and IMU data, the first point cloud data being used to represent a target object, the target object including an object to be scanned, a base and a turntable, wherein the turntable is located on the base and the object to be scanned is located on the turntable; a processing unit, configured to solve the gravity direction in the camera coordinate system according to the IMU data; determining a first plane set according to the first point cloud data; screening a second plane set from the first plane set according to the gravity direction, wherein the angle between the normal vector direction of the plane in the second plane set and the gravity direction is within a preset angle range; removing the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second plane set that is farthest from the depth camera in vertical distance, and the second plane is the plane in the second plane set that is parallel to the first plane and closest to the first plane in vertical distance, and the first target point cloud data is used to identify the object to be scanned.

[0023] In combination with the second aspect, in certain implementations of the second aspect, when the first plane set is empty, the acquisition unit is specifically used to: acquire the second point cloud data; and the processing unit is specifically used to: determine the first plane set based on the second point cloud data.

[0024] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned device also includes: a control unit, used to adjust the posture of the depth camera before the acquisition unit acquires the second point cloud data, so that the first plane set determined according to the second point cloud data is not empty.

[0025] In combination with the second aspect, in some implementations of the second aspect, the first plane is a plane corresponding to the base, and the second plane is a plane corresponding to the turntable.

[0026] In a third aspect, a device for processing point cloud data is provided, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute a method in any possible implementation of the method design of the first aspect above.

[0027] In a fourth aspect, a scanner is provided, which is a device in any possible implementation of the device design of the second aspect.

[0028] In a fifth aspect, a scanning system is provided, comprising: a scanner, a rotating device and a processor, wherein the scanner comprises a depth camera equipped with an IMU, the rotating device comprises a base and a turntable, the turntable is arranged on the base, and the turntable is used to place the object to be scanned; wherein the processor is used to send a first instruction to the scanner; the scanner is used to obtain first point cloud data and IMU data according to the first instruction, and send the first point cloud data and the IMU data to the processor, the first point cloud data is used to represent the target object, and the target object comprises the object to be scanned, the base and the turntable; the processor is also used to solve the camera coordinate system according to the IMU data direction of gravity; according to the first point cloud data, determine the first plane set; according to the direction of gravity, filter out the second plane set from the first plane set, and the angle between the normal vector direction of the plane in the second plane set and the gravity direction is within the preset angle range; remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain the first target point cloud data, the first plane is the plane in the second plane set with the farthest vertical distance from the depth camera, the second plane is the plane in the second plane set that is parallel to the first plane and has the closest vertical distance to the first plane, and the first target point cloud data is used to identify the object to be identified.

[0029] For example, if the number of second planes is zero, the processor may remove only the point cloud corresponding to the first plane from the first point cloud data to obtain second target point cloud data. It should be understood that this second target point cloud data can also be used to identify the object to be scanned, but the accuracy of the recognition result based on the second target point cloud data is relatively low. Therefore, the object to be scanned can be jointly identified with the subsequently determined target point cloud data.

[0030] In combination with the fifth aspect, in certain implementations of the fifth aspect, when the first plane set is empty, the above-mentioned scanner is specifically used to obtain second point cloud data and send the second point cloud data to the processor; the above-mentioned processor is specifically used to: determine the first plane set based on the second point cloud data.

[0031] In combination with the fifth aspect, in certain implementations of the fifth aspect, before the above-mentioned scanner obtains the second point cloud data, the above-mentioned processor is also used to: send a second instruction to the scanner, and the second instruction is used to instruct to adjust the posture of the depth camera so that the first plane set determined according to the second point cloud data is not empty.

[0032] In the sixth aspect, a chip system is provided, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to echo a signal from the memory of the electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method in any possible implementation of the method design of the first aspect above.

[0033] In a seventh aspect, a computer-readable storage medium is provided, storing a computer program or instruction, which is used to implement the method in any possible implementation manner in the method design of the first aspect.

[0034] In an eighth aspect, a computer program product is provided. When the computer program code or instructions are executed on a computer, the computer executes a method in any possible implementation of the method design of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] FIG1 is a schematic block diagram of a method 100 for processing point cloud data proposed in an embodiment of the present application;

[0036] FIG2 is a schematic diagram of an application scenario of a method 100 for processing point cloud data proposed in an embodiment of the present application;

[0037] FIG3 is a schematic block diagram of a method 300 for removing planar point cloud data provided in an embodiment of the present application;

[0038] FIG4 is a schematic diagram of the vertical distance between a plane of the second plane set and a depth camera according to an embodiment of the present application;

[0039] FIG5 is a schematic diagram of the vertical distance between the planes of the third plane set and the base plane proposed in an embodiment of the present application;

[0040] FIG6 is a schematic diagram of a process for determining first target point cloud data proposed in an embodiment of the present application;

[0041] FIG7 is a schematic block diagram of a point cloud data processing device 700 proposed in an embodiment of the present application;

[0042] FIG8 is a schematic block diagram of a scanning system 800 proposed in an embodiment of the present application;

[0043] FIG9 is a schematic block diagram of a computer device 900 according to an embodiment of the present application;

[0044] FIG10 is a schematic block diagram of a computer-readable storage medium 1000 proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is a kind of association relationship that describes associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In this application, "at least one" refers to one or more, and "more than one" refers to two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0046] In the embodiments of this application, prefixes such as "first" and "second" are used only to distinguish different description objects and have no limiting effect on the position, order, priority, quantity, or content of the described objects. The use of prefixes such as ordinal numbers in the embodiments of this application to distinguish description objects does not constitute a limitation on the described objects. For a statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0047] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0048] In the current scanning system, the target object is usually placed on a rotating device, which drives the target object to rotate within the visual range of the scanner, so that the scanner can perform an all-round and efficient scan of the target object, and then obtain point cloud images of the target object facing different directions. The rotating device can usually be a turntable, which is set on a base, and the target object is placed on the turntable, wherein the base can be a base or the ground. During the processing of the scanning algorithm, the point clouds corresponding to the base and the turntable in the acquired point cloud image need to be eliminated to achieve more accurate reconstruction of the target object. Since the point clouds corresponding to the base plane and the turntable plane have a greater influence on the scanning and reconstruction of the target object, the point clouds corresponding to the base plane and the turntable plane can be specifically eliminated, wherein the base plane specifically refers to the upper surface of the base, and the turntable plane specifically refers to the upper surface of the turntable.

[0049] For the above application scenarios, the following two solutions can be used at this stage to remove the point clouds corresponding to the base plane and the turntable plane from the point cloud image.

[0050] Option 1: Use a special turntable.

[0051] The aforementioned specially designed turntable can be one with rich textures and geometric shapes. During point cloud image recognition, objects with rich textures and geometric shapes are relatively easy to identify. Therefore, during point cloud data processing, specific textures and geometric shapes are used as image recognition constraints. When an object with these specific textures and geometric shapes is identified, the system can determine that the object is a substrate or a turntable, and then remove the point cloud corresponding to the object from the point cloud image.

[0052] Solution 2: Eliminate all external surfaces through point cloud data processing algorithms.

[0053] Since the point clouds corresponding to the base and turntable in the point cloud image are eliminated, specifically the point clouds corresponding to the base plane and the turntable plane are eliminated, the point clouds corresponding to all circumscribed surfaces in the current point cloud image can be detected through the point cloud data processing algorithm. After eliminating the point clouds corresponding to all circumscribed surfaces, what remains should be the point cloud corresponding to the target object.

[0054] However, for solution 1, since the image recognition constraints corresponding to objects with different textures and different geometric shapes are different during the point cloud data processing process, and the turntables on the market are diverse, with different textures and shapes, it is impossible to identify different turntables through a universal image recognition constraint, so this solution is not universal. For solution 2, since the target object placed on the turntable may also have a structure with planar characteristics, if the point clouds corresponding to all circumscribed surfaces in the point cloud image are directly removed, there is a certain probability that the target object's own planar structure will be mistakenly removed. Therefore, at this stage, the method of removing the point clouds corresponding to the base plane and the turntable plane from the point cloud image has the problems of poor universality and low accuracy.

[0055] In view of this, the present application provides a method, apparatus, scanner, and scanning system for processing point cloud data. These methods obtain IMU data from a camera, calculate the direction of gravity in the camera coordinate system based on the IMU data, and then, based on this gravity direction, gradually filter out the base plane and turntable plane in the point cloud image, and remove the point clouds corresponding to these two planes from the point cloud data. This method has no specific requirements for the type or shape of the turntable, thereby ensuring the method's versatility. Furthermore, in the process of removing the base plane and turntable plane, the target object's own planar structure will not be mistakenly removed, thereby ensuring the method's accuracy.

[0056] FIG1 is a schematic block diagram of a method 100 for processing point cloud data proposed in an embodiment of the present application.

[0057] FIG2 is a schematic diagram of an application scenario of a method 100 for processing point cloud data proposed in an embodiment of the present application.

[0058] The method 100 is applied to a scanner including a depth camera equipped with an IMU. The depth camera can be a depth camera based on structured light technology, binocular technology, or time of flight (TOF) technology, and can simultaneously obtain spatial distance data of a scene and the camera's position data.

[0059] S110: Acquire first point cloud data and IMU data.

[0060] The first point cloud data is used to represent a target object, which is an object included in an image captured by the depth camera at its current position. Referring to FIG2 , the target object includes an object to be scanned, a substrate, and a turntable, wherein the turntable is located on the substrate and the object to be scanned is located on the turntable.

[0061] In some possible embodiments, depth image data including the target object may be acquired by a depth camera, and the first point cloud data may be determined based on the depth image data and the camera intrinsic parameters of the depth camera.

[0062] In some possible embodiments, the first point cloud data may include all image elements in the scene corresponding to the depth image data, that is, in addition to the object to be scanned, the substrate and the turntable, it also includes other environmental objects; it may also include only image elements corresponding to the region of interest (ROI) in the depth image data, that is, the depth image data includes only the object to be scanned, the substrate and the turntable as much as possible.

[0063] It should be understood that the camera intrinsic parameters can be directly obtained through the depth camera and are known parameters.

[0064] It should be understood that due to the depth camera's viewing angle at a specific pose, the captured image may contain missing elements of the target scene. For example, if the depth camera is vertically photographing the object to be scanned, if the top of the object to be scanned is large, the turntable may be obscured, resulting in the turntable plane not being present in the captured image. For details on how to handle this situation, please refer to the corresponding embodiments below.

[0065] In some possible embodiments, the base may be a pedestal, or may be the ground, or other platforms for placing objects, which is not limited in the embodiments of the present application.

[0066] S120: Calculate the gravity direction in the camera coordinate system based on the IMU data.

[0067] In some possible embodiments, since the IMU is mounted on the depth camera, the gravity direction in the camera coordinate system of the depth camera can be calculated based on the IMU data (refer to the gravity direction 1 in Figure 2). The camera coordinate system is a coordinate system established with the depth camera as the origin. In the three-dimensional reconstruction process, the world coordinate system coincides with the origin of the camera coordinate system, and the point cloud data is the three-dimensional data in the camera coordinate system. Therefore, it is only necessary to calculate the gravity direction in the camera coordinate system, and the plane in the point cloud data can be identified based on the gravity direction.

[0068] S130: Determine a first plane set based on the first point cloud data.

[0069] In some possible embodiments, the first point cloud data may be detected using a plane detection algorithm to determine a first set of planes. The plane detection algorithm may be a RANSAC algorithm. Thus, the first set of planes may include all detected planes of the target object.

[0070] It should be understood that the first plane set mentioned above may specifically include a base plane, a turntable plane, and planes that may be included in the object to be scanned.

[0071] In some possible embodiments, the first plane set may include multiple planes, 1 plane, or even 0 planes. For the processing flow when the number of planes included in the first plane set is 1 or 0, please refer to the subsequent corresponding embodiments for details. When the number of planes included in the first plane set is multiple, the following S140 can be executed.

[0072] S140: According to the gravity direction, a second plane set is obtained by screening from the first plane set, wherein the angles between the normal vector directions of the planes in the second plane set and the gravity direction are within a preset angle range.

[0073] It should be understood that since the base is usually the ground or a base placed horizontally on the ground, the normal vector direction of the base plane usually coincides with the above-mentioned gravity direction, or there is an angle within a certain range between the normal vector direction and the gravity direction.

[0074] In some possible embodiments, under ideal conditions, the preset angle range can be a specific angle value: 0°. However, in actual applications, the base plane is usually not absolutely perpendicular to the gravity direction in the camera coordinate system, so the angle between the normal vector of the base plane and the gravity direction should be within a reasonable angle range, that is, the preset angle range, for example, it can be within the angle range of [-5,5]°. The turntable is usually placed horizontally on the base, so the turntable plane is usually parallel to the base plane, so the angle between the normal vector direction of the turntable plane and the gravity direction should also be within the preset angle range. Therefore, when the posture of the depth camera is reasonable and the turntable plane is not blocked, the second plane set selected based on the above conditions can include both the base plane and the turntable plane.

[0075] S150: Delete the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data. The first plane is the plane in the second plane set that is the farthest perpendicular to the depth camera, and the second plane is the plane in the second plane set that is parallel to the first plane and closest perpendicularly to the first plane.

[0076] It should be understood that the first plane may be the base plane, the second plane may be the turntable plane, and the first target point cloud data is used to identify the object to be scanned.

[0077] In some possible embodiments, the first plane and the second plane are both screened from the second plane set. Then, based on the spatial position information of the planes in the second plane set, that is, the vertical distance from the plane to the depth camera or the vertical distance between the planes, the second plane set is screened. The screening result may include the following five cases:

[0078] 1) Selecting a first plane from the set of second planes;

[0079] 2) selecting a plurality of first planes from the set of second planes;

[0080] 3) selecting one first plane and one second plane from the second plane set;

[0081] 4) selecting a plurality of first planes and one second plane from the second plane set;

[0082] 5) Filtering out a plurality of first planes and a plurality of second planes from the second plane set.

[0083] It should be understood that the reason for selecting multiple first planes, that is, multiple base planes, is because the base can be the ground, and the ground has a large area, so it is divided into multiple sub-planes. These sub-planes are the multiple first planes that can be selected. The reason for selecting multiple second planes is similar. Generally, at least one first plane and at least one second plane can be selected from the above set of second planes.

[0084] In some possible embodiments, after determining the first target point cloud data, the first target point cloud data may be sent to a downstream functional module, which then identifies the object to be identified based on the first target point cloud data. The downstream functional module may be a functional module of a processor used in 3D printing, object reconstruction, or reverse engineering scenarios.

[0085] Based on the above technical solution, IMU data is acquired to calculate the direction of gravity in the camera coordinate system. Based on this direction of gravity, the point clouds corresponding to the base plane and turntable plane in the point cloud data are gradually filtered out and then removed from the point cloud data. This method has no specific requirements for the type and shape of the turntable, thus ensuring its versatility. In addition, when removing the point clouds corresponding to the base plane and turntable plane, the planar structure of the object being scanned is not mistakenly removed, thus ensuring the accuracy of the method.

[0086] In some possible embodiments, plane screening may be performed on the first plane set using the following method.

[0087] FIG3 is a schematic block diagram of a method 300 for removing planar point cloud data provided in an embodiment of the present application.

[0088] S310: Filtering the first plane set to obtain a second plane set, wherein the angles between the normal vector directions of the planes in the second plane set and the gravity direction are within a preset angle range.

[0089] In some possible embodiments, based on the spatial position relationship between the base plane and the depth camera, it can be seen that the depth camera usually takes a bird's-eye view of the target scene, so the depth camera is usually located above the base. Therefore, in the process of screening out the second plane set, only the planes whose spatial position is below the horizontal position of the depth camera can be screened, thereby reducing the computational overhead caused by plane screening.

[0090] S320: Determine a plane in the second plane set that has the longest vertical distance from the depth camera as a first plane.

[0091] In some possible embodiments, as shown in reference Figure 4, when calculating the vertical distance between each plane in the second plane set and the depth camera, the depth camera can be abstracted as a depth camera mass point A, and the vertical distance between each plane in the second plane set and the depth camera can be specifically the vertical distance L between the center O of each plane in the second plane set and the depth camera mass point A.

[0092] In some possible embodiments, when the number of planes included in the second plane set is equal to 1, for the convenience of description, the planes included in the second plane set are referred to as candidate planes below. Since there is only one candidate plane, this candidate plane must also be the candidate plane with the farthest vertical distance from the depth camera. There is no process of comparing the above-mentioned vertical distances, and the candidate plane can be directly determined as the first plane, that is, determined as the base plane.

[0093] It should be understood that the reason for determining the above-mentioned candidate plane as the base plane rather than the turntable plane is as follows: when the base is a pedestal, in order to ensure the stability of the rotation of the turntable and the object to be scanned, the area of ​​the base plane is usually larger than the area of ​​the turntable plane. When the base is the ground or other storage platform, the area of ​​the base plane will be much larger than the turntable plane. Therefore, in any of the above cases, even if the object to be scanned is photographed vertically, the object to be scanned will at most block the turntable plane, but not the base plane. Moreover, when the depth camera is photographing the target object, the position of the depth camera is usually not directly above the target object, so in conventional application scenarios, neither the base plane nor the turntable plane will usually be blocked. When the number of planes in the second plane set is greater than 1, it is necessary to compare the vertical distances of each plane in the second plane set from the depth camera, and determine the plane in the second plane set with the farthest vertical distance from the depth camera as the base plane.

[0094] S330: Eliminate the point cloud data corresponding to the first plane from the first point cloud data.

[0095] S340: Determine a plane in the second plane set that is parallel to the first plane and has the shortest vertical distance to the first plane as the second plane.

[0096] S340 described above can be divided into two sub-steps. First, based on the first plane, a third plane set is obtained by screening from the second plane set. All planes in the third plane set are parallel to the first plane. Then, the plane in the third plane set that is closest to the first plane in vertical distance is determined as the second plane. Furthermore, the "closest vertical distance" refers to the plane that is outside the minimum distance and has the closest vertical distance to be determined as the minimum plane, to ensure that the screened second plane does not belong to the first plane.

[0097] FIG5 is a schematic diagram of the vertical distance between the planes of the third plane set and the first plane proposed in an embodiment of the present application.

[0098] In some possible embodiments, referring to FIG5 , when calculating the vertical distance between each plane in the third plane set and the first plane (base plane), the center O of the first plane can be used as a reference to calculate the vertical distance M between the center P of each plane in the third plane set and the center O. When there are multiple first planes, the center O of any one of the first planes can be selected as the reference.

[0099] It should be understood that since the turntable is placed on the base, the base plane (first plane) and the turntable plane (second plane) are parallel. Therefore, the second plane is included in the filtered third plane set. Furthermore, since the turntable is placed on the base, when filtering the third plane set, only planes whose spatial positions are above the first plane can be filtered, thereby reducing the computational overhead caused by plane filtering.

[0100] In some possible embodiments, considering the possibility of missed detections during plane detection, a third plane set may be obtained by filtering the first plane set from the first plane set based on the first plane. Alternatively, the first plane set may be re-determined using a plane detection algorithm, and then the third plane set may be filtered from the first plane set based on the first plane.

[0101] S350: Eliminate the point cloud data corresponding to the second plane from the first point cloud data.

[0102] In some possible embodiments, after executing S320, the process may jump directly to S340. After executing S340, the following steps may be performed:

[0103] S360: Eliminate the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data.

[0104] It should be understood that since the turntable is placed directly on the base, it should be the object closest to the base in the entire target scene, so the turntable plane should be the plane with the closest vertical distance to the base plane in the third plane set.

[0105] Based on the above technical solution, the base plane and turntable plane in the target scene can be accurately identified, so that the point clouds corresponding to the base plane and turntable plane can be subsequently removed from the first point cloud data, thereby increasing the accuracy of target scanning.

[0106] In some possible embodiments, since the vertical distance between the second plane and the first plane is the shortest compared to other planes in the third plane set, the embodiment of the present application further proposes a method for detecting a blurred plane, which can replace S340 in the method 300. The method can be:

[0107] The plane in the second plane set with the second greatest vertical distance from the depth camera is determined as the second plane. As can be seen, the plane screening constraints of this method are looser and simpler than those in S340 above. Therefore, the computational overhead of screening the second plane is low, and the accuracy of screening the second plane is not significantly reduced, achieving a good compromise between computational overhead and plane detection accuracy.

[0108] In some possible embodiments, if the number of planes parallel to the first plane in the second plane set is zero, i.e., if the second plane set does not include the second plane, or if the number of planes in the third plane set is zero, this indicates that the turntable plane has not been identified. In this case, the point cloud data corresponding to the first plane can be removed from the first point cloud data to obtain second target point cloud data. It should be understood that this second target point cloud data can also be used to identify the object to be scanned, but the accuracy of the recognition result based on the second target point cloud data is relatively low. Therefore, it can be combined with the subsequently determined target point cloud data to perform joint recognition of the object to be scanned.

[0109] It can be seen from this that even if the second plane is not identified based on the above method 300 and the corresponding embodiment, the point cloud data processing process will not be blocked, but will continue to process the next frame of point cloud data.

[0110] In some possible embodiments, when the first plane set is empty, second point cloud data is acquired; the first plane set is determined based on the second point cloud data, and the subsequent plane screening process in the above method 300 is then performed based on the newly determined first plane set.

[0111] It should be understood that when the number of planes in the first plane set is 0, it indicates that no planes can be detected based on the current point cloud data, so the processing flow for the next frame of point cloud data can be directly entered. In this case, it is also possible to adjust the parameters of the plane detection algorithm or the position of the depth camera to ensure that the first plane set determined based on the second point cloud data is not empty. Based on this, it is possible to effectively avoid the problem of inaccurate plane detection caused by improper depth camera position or abnormal plane detection algorithm parameters.

[0112] In some possible embodiments, when the number of planes in the first plane set is 1, it is first necessary to determine whether the normal vector direction of the plane is parallel to the gravity direction. If it is parallel to the gravity direction, the plane can be determined to be the above-mentioned first plane, and then the point cloud corresponding to the first plane can be directly removed from the above-mentioned first point cloud data to obtain the second target point cloud data; if it is not parallel to the gravity direction, the second point cloud data is obtained, and the above-mentioned first plane set is re-determined according to the second point cloud data, and then the subsequent plane screening process in the above-mentioned method 300 is performed based on the re-determined first plane set.

[0113] In addition, if the number of planes in the second plane set is 0, it means that the subsequent process of identifying the base plane and turntable plane cannot be performed based on the point cloud data of the current frame. In this case, the processing flow for the next frame of point cloud data can be directly entered. In this case, adjustments can also be made to the preset angle range or the depth camera pose.

[0114] It should be understood that the process described in the above method 300 and the corresponding embodiments is a cyclic process. When the current loop step is completed or the current loop step does not achieve the expected result, the processor's execution process will automatically exit the current loop step and enter the loop step for the next frame of point cloud data. In other words, subsequent processing of the third and fourth point cloud data will be carried out. The specific process is similar and will not be repeated here.

[0115] During the subsequent point cloud data processing, the depth camera's pose typically remains unchanged while the scanner is scanning the target object. Therefore, the processor used for point cloud data processing does not need to synchronously acquire the IMU data at that moment. The gravity direction determined during the processing of the first point cloud data can be used for subsequent plane detection operations on the second point cloud data.

[0116] In some possible embodiments, the depth camera receives a first signal at a first moment, the first signal being used to indicate that the depth camera's position has changed. After the first moment, the depth camera acquires depth image data at a second moment, acquires IMU data at the second moment, and sends the depth image data and IMU data to the processor. After receiving the reacquired IMU data, the processor recalculates the gravity direction in the camera coordinate system based on the IMU data.

[0117] It should be understood that the steps described in the above methods are merely exemplary steps, and the order of the steps can be reasonably adjusted for different situations. The embodiments of this application do not limit the order of the specific steps. Therefore, the specific process for determining the second point cloud data in this application is as follows.

[0118] FIG6 is a schematic diagram of a process for determining first target point cloud data proposed in an embodiment of the present application.

[0119] In some possible embodiments, the second point cloud data may be determined through the following two approaches.

[0120] Path 1:

[0121] S611: Acquire image frame data.

[0122] S612: Determine first point cloud data based on the depth image data and camera intrinsic parameters in the image frame data.

[0123] S613: Calculate the gravity direction in the camera coordinate system based on the IMU data in the image frame data.

[0124] S614: Determine a first plane set using a plane detection algorithm.

[0125] When the number of planes in the first plane set is greater than 0, go to S615 ; when the number of planes in the first plane set is equal to 0, go to S611 .

[0126] S615: Based on the above method 300, determine the base plane.

[0127] When the base plane is determined, go to S616; when the base plane is not determined, go to S611.

[0128] S616: Based on the above method 300, determine the turntable plane.

[0129] When the turntable plane is determined, go to S617; when the turntable plane is not determined, go to S618.

[0130] S617: Eliminate the point clouds corresponding to the base plane and the turntable plane from the first point cloud data to determine first target point cloud data.

[0131] S618: Eliminate the point cloud corresponding to the base plane from the first point cloud data to determine the first target point cloud data.

[0132] After executing the above S617 or S618, the second point cloud data is sent to the downstream functional module, and then the process goes to S611.

[0133] Path 2:

[0134] S611: Acquire image frame data.

[0135] S612: Determine first point cloud data based on the depth image data and camera intrinsic parameters in the image frame data.

[0136] S613: Calculate the corresponding gravity direction in the camera coordinate system based on the IMU data in the image frame data.

[0137] S614: Determine a first plane set using a plane detection algorithm.

[0138] When the number of planes in the first plane set is greater than 0, go to S625 ; when the number of planes in the first plane set is equal to 0, go to S611 .

[0139] S625: Based on the above method 300, determine the base plane.

[0140] When the base plane is determined, go to S626; when the base plane is not determined, go to S611.

[0141] S626: Eliminate the point cloud corresponding to the base plane from the first point cloud data to determine first quasi-target point cloud data.

[0142] S627: Based on the above method 300, determine the turntable plane.

[0143] When the turntable plane is determined, go to S628; when the turntable plane is not determined, go to S629.

[0144] S628: Eliminate the point clouds corresponding to the above-mentioned turntable planes from the first quasi-target point cloud data to determine the first target point cloud data.

[0145] S629: Determine the first quasi-target point cloud data as the first target point cloud data.

[0146] After executing the above S628 or S629, the first target point cloud data is sent to the downstream functional module and the process goes to S611.

[0147] Based on the above technical solution, multiple ways of determining target point cloud data are provided, thereby increasing the flexibility of the method.

[0148] In addition, an embodiment of the present application also provides a device for implementing any of the above methods. For example, a device for point cloud data processing is provided, which includes a unit (or means) for implementing any of the above point cloud data processing methods.

[0149] FIG7 is a schematic block diagram of a point cloud data processing device 700 provided in an embodiment of the present application.

[0150] The device 700 is applied to a scanner, which includes a depth camera equipped with an IMU. The device 700 includes:

[0151] The acquisition unit 710 is used to acquire first point cloud data and IMU data. The first point cloud data is used to represent the target object. The target object includes an object to be scanned, a substrate and a turntable, wherein the turntable is located on the substrate and the object to be scanned is located on the turntable.

[0152] Processing unit 720 is used to solve the gravity direction in the camera coordinate system based on the IMU data; determine a first plane set based on the first point cloud data; according to the gravity direction, filter out a second plane set from the first plane set, wherein the angle between the normal vector direction of the plane and the gravity direction in the second plane set is within a preset angle range; remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second plane set that is farthest from the vertical distance of the depth camera, and the second plane is the plane in the second plane set that is parallel to the first plane and has the closest vertical distance to the first plane. The first target point cloud data is used to identify the object to be scanned.

[0153] In some possible embodiments, when the first plane set is empty, the acquisition unit 710 is specifically configured to acquire second point cloud data; and the processing unit 720 is specifically configured to determine the first plane set based on the second point cloud data.

[0154] In some possible embodiments, the apparatus 700 further includes: a control unit 730 configured to adjust the posture of the depth camera before the acquisition unit 710 acquires the second point cloud data, so that the first plane set determined according to the second point cloud data is not empty.

[0155] An embodiment of the present application further proposes a scanner, which includes a point cloud data processing device 700 proposed in any of the above embodiments.

[0156] The embodiment of the present application also proposes a scanning system.

[0157] FIG8 is a schematic block diagram of a scanning system 800 proposed in an embodiment of the present application.

[0158] The scanning system 800 includes a scanner 810, a base 821, a turntable 822, and a processor 830. The scanner 810 includes a depth camera 811 equipped with an IMU 812. The turntable 822 is mounted on the base 821 and is used to place an object to be scanned. The base 821 can be a ground surface or a base.

[0159] The processor 830 is configured to send the first instruction to the scanner 810 .

[0160] The scanner 810 is used to: obtain first point cloud data and IMU data according to the first instruction, and send the first point cloud data and IMU data to the processor 830, where the first point cloud data is used to represent the target object, which includes the object to be scanned, the base 821 and the turntable 822.

[0161] Processor 830 is used to: calculate the gravity direction in the camera coordinate system based on the IMU data; determine a first plane set based on the first point cloud data; filter out a second plane set from the first plane set based on the gravity direction, and the angle between the normal vector direction of the plane in the second plane set and the gravity direction is within a preset angle range; remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second plane set that is farthest from the depth camera in vertical distance, and the second plane is the plane in the second plane set that is parallel to the first plane and has the closest vertical distance to the first plane, and the first target point cloud data is used to identify the object to be identified.

[0162] The first plane may be a base plane, and the second plane may be a turntable plane.

[0163] In some possible embodiments, the processor 830 may include all functions of the processing unit 730 in the apparatus 700 .

[0164] Figure 9 is a schematic block diagram of a computer device 900 provided in an embodiment of the present application. The computer device 900 shown in Figure 9 includes a memory 910, a processor 920, and a bus 940. Optionally, the computer device 900 also includes a communication interface 930. The memory 910, processor 920, and communication interface 930 are connected to each other via bus 940.

[0165] The memory 910 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 910 can store programs. When the program stored in the memory 910 is executed by the processor 920, the processor 920 is used to perform the various steps of the VIO system initialization method of the embodiment of the present application. For example, the processor 920 can execute the method shown in Figures 2 to 6 above.

[0166] The processor 920 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the point cloud data processing method proposed in the method embodiment of the present application.

[0167] The processor 920 can also be an integrated circuit chip with signal processing capabilities. During the implementation process, the various steps of the VIO system initialization method of the present application can be completed through the hardware integrated logic circuit in the processor 920 or software instructions.

[0168] The processor 920 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods involved in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or the like. The storage medium is located in the memory 910, and the processor 920 reads the information in the memory 910 and, in combination with its hardware, completes the functions required to be executed by the units included in the apparatus shown in FIG. 7 , or executes the methods shown in FIG. 1 to FIG. 6 of the method embodiments of the present application.

[0169] The communication interface 930 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 900 and other devices or a communication network.

[0170] The bus 940 may include a path for transmitting information between various components of the device 900 (eg, the memory 900 , the processor 920 , and the communication interface 930 ).

[0171] It should be understood that although the above-mentioned device 900 only shows a memory, a processor, and a communication interface, during the specific implementation process, those skilled in the art will understand that the device 900 may also include other components necessary for normal operation. At the same time, according to specific needs, those skilled in the art will understand that the device 900 may also include hardware components that implement other additional functions. In addition, those skilled in the art will understand that the device 900 may also include only the components necessary to implement the embodiments of the present application, and does not necessarily include all the components shown in Figure 9.

[0172] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0173] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0174] Figure 10 is a schematic block diagram of a computer-readable storage medium 1000 provided in an embodiment of the present application. The computer-readable storage medium 1000 shown in Figure 10 stores computer instructions 1010. When the computer instructions 1010 are executed by a processor, the method corresponding to the above embodiment can be implemented.

[0175] In some possible embodiments, the computer-readable storage medium 1000 may be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes a collection of one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state drive.

[0176] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0177] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0181] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0182] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for processing point cloud data, characterized in that: Applied to a scanner, the scanner comprising a depth camera equipped with an inertial measurement unit (IMU), the method comprising: Acquire first point cloud data and IMU data, wherein the first point cloud data is used to represent a target object, wherein the target object includes an object to be scanned, a substrate, and a turntable, wherein the turntable is located on the substrate, and the object to be scanned is located on the turntable; According to the IMU data, the gravity direction in the camera coordinate system is calculated; Determine a first plane set according to the first point cloud data; According to the gravity direction, a second plane set is obtained by screening from the first plane set, wherein the angle between the normal vector direction of the plane in the second plane set and the gravity direction is within a preset angle range; The point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane are eliminated from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second plane set that is the farthest from the depth camera in vertical distance, and the second plane is the plane in the second plane set that is parallel to the first plane and has the closest vertical distance to the first plane, and the first target point cloud data is used to identify the object to be scanned.

2. The method according to claim 1, characterized in that: Determining a first plane set according to the first point cloud data includes: When the first plane set is empty, obtaining second point cloud data; The first plane set is determined according to the second point cloud data.

3. The method according to claim 2, characterized in that Before acquiring the second point cloud data, the method further includes: The posture of the depth camera is adjusted so that the first plane set determined according to the second point cloud data is not empty.

4. The method according to any one of claims 1 to 3, characterized in that The first plane is a plane corresponding to the base, and the second plane is a plane corresponding to the turntable.

5. A device for processing point cloud data, characterized in that: Applied to a scanner, the scanner includes a depth camera equipped with an inertial measurement unit IMU, the device includes: an acquisition unit, configured to acquire first point cloud data and IMU data, wherein the first point cloud data is used to represent a target object, wherein the target object includes an object to be scanned, a substrate, and a turntable, wherein the turntable is located on the substrate, and the object to be scanned is located on the turntable; A processing unit is used to solve the gravity direction in the camera coordinate system according to the IMU data; determine a first plane set according to the first point cloud data; screen a second plane set from the first plane set according to the gravity direction, wherein the angle between the normal vector direction of the plane in the second plane set and the gravity direction is within a preset angle range; remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second plane set that is the farthest from the depth camera in vertical distance, and the second plane is the plane in the second plane set that is parallel to the first plane and has the shortest vertical distance to the first plane, and the first target point cloud data is used to identify the object to be scanned.

6. The device according to claim 5, characterized in that When the first plane set is empty, the acquisition unit is specifically used to: acquire second point cloud data; The processing unit is specifically configured to determine the first plane set according to the second point cloud data.

7. The device according to claim 6, characterized in that The device also includes: A control unit is used to adjust the posture of the depth camera before the acquisition unit acquires the second point cloud data, so that the first plane set determined according to the second point cloud data is not empty.

8. A scanner, characterized in that: Comprising a device as claimed in any one of claims 5 to 7.

9. A scanning system, characterized in that: include: A scanner, a rotating device and a processor, wherein the scanner includes a depth camera equipped with an inertial measurement unit (IMU), the rotating device includes a base and a turntable, the turntable is arranged on the base, and the turntable is used to place an object to be scanned; The processor is used to send a first instruction to the scanner; The scanner is used to, according to the first instruction, obtain first point cloud data and IMU data, and send the first point cloud data and the IMU data to the processor, wherein the first point cloud data is used to represent a target object, and the target object includes an object to be scanned, the substrate, and the turntable; The processor is also used to solve the gravity direction in the camera coordinate system according to the IMU data; determine a first plane set according to the first point cloud data; screen a second plane set from the first plane set according to the gravity direction, wherein the angle between the normal vector direction of the plane in the second plane set and the gravity direction is within a preset angle range; remove the point cloud data corresponding to the first plane and the point cloud data corresponding to the second plane from the first point cloud data to obtain first target point cloud data, wherein the first plane is the plane in the second plane set that is the farthest from the depth camera in vertical distance, and the second plane is the plane in the second plane set that is parallel to the first plane and has the shortest vertical distance to the first plane, and the first target point cloud data is used to identify the object to be identified.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 4.

Images