Three-dimensional point cloud filtering method and electronic equipment

Abstract

The present invention discloses a three-dimensional point cloud filtering method and electronic device, the method comprising obtaining a first image of a light plane array projected by a light plane array transmitter through a binocular system; calculating a light plane expression corresponding to each light plane in the light plane array based on the first image; controlling the light plane array transmitter to project the light plane array toward an object to be measured, and obtaining a second image of the light plane array projected on the object to be measured through the binocular system; obtaining a three-dimensional point cloud to be filtered based on the second image; and filtering points in the corresponding three-dimensional point cloud to be filtered based on the light plane expression of each light plane to obtain a filtered three-dimensional point cloud. The present application obtains a filtered three-dimensional point cloud by filtering points in the corresponding three-dimensional point cloud to be filtered based on the light plane expression of each light plane, and can accurately filter out abnormal points in the three-dimensional point cloud based on the geometric principles of imaging.

Description

Technical Field

[0001] The present invention relates to the technical field of structured light reconstruction, and in particular to a three-dimensional point cloud filtering method and electronic equipment. Background Art

[0002] With the rapid development of 3D scanning technology, 3D reconstruction methods based on structured light have been widely used in many fields. Structured light-based 3D reconstruction methods project structured light (usually a light plane array) onto the object to be measured, and use a binocular system to obtain images of the structured light projected on the object to be measured from different angles for processing, thereby obtaining a 3D point cloud of the object to be measured. Based on the 3D point cloud of the object to be measured, a 3D model of the object to be measured is reconstructed. In practical applications, due to factors such as ambient light interference and surface scattering of the object to be measured, abnormal light spots often exist in the obtained structured light images, which can lead to abnormal points in the obtained 3D point cloud.

[0003] Therefore, existing technologies typically process acquired structured light images at the image processing level. These methods often rely on image processing algorithms, such as threshold segmentation, morphological operations, and optical flow methods, to identify and remove light spots that do not conform to the expected structured light pattern. These methods can, to a certain extent, reduce the impact of anomalous light spots on the 3D point cloud, thereby reducing the number of outliers in the 3D point cloud.

[0004] However, due to the dynamic changes of ambient light and the complex scattering characteristics of the object surface, the shape and distribution of abnormal light spots are often difficult to accurately predict and completely filter, which will still lead to some abnormal points in the obtained three-dimensional point cloud. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present application provides a three-dimensional point cloud filtering method and electronic device, which filters the points in the corresponding three-dimensional point cloud to be filtered according to the light plane expression of each light plane to obtain a filtered three-dimensional point cloud, and can accurately filter out abnormal points in the three-dimensional point cloud according to the geometric principles of imaging.

[0006] In order to solve the above problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a three-dimensional point cloud filtering method, comprising:

[0008] acquiring, through the binocular system, a first image of the light plane array projected by the light plane array emitter;

[0009] Obtaining a light plane expression corresponding to each light plane in the light plane array by calculation based on the first image;

[0010] controlling the light plane array emitter to project the light plane array toward the object to be measured, and acquiring a second image of the light plane array projected on the object to be measured through the binocular system;

[0011] Acquire a three-dimensional point cloud to be filtered according to the second image, wherein each point in the three-dimensional point cloud to be filtered corresponds to one of the light planes;

[0012] Points in the corresponding three-dimensional point cloud to be filtered are filtered according to the light plane expression of each light plane to obtain a filtered three-dimensional point cloud.

[0013] In some embodiments, calculating and obtaining a light plane expression corresponding to each light plane in the light plane array based on the first image includes:

[0014] identifying a plurality of fringe centers in the first image;

[0015] identifying fringe features of each fringe center in the first image, wherein each fringe center represents a light plane;

[0016] Determining the code of the light plane corresponding to each of the fringe centers according to the fringe characteristics of each of the fringe centers;

[0017] Obtaining a fitting point set for each light plane by calculation according to the codes of the plurality of light planes and the first image;

[0018] Plane fitting is performed on each light plane based on the fitting point set of each light plane to obtain a light plane expression corresponding to each light plane.

[0019] In some embodiments, identifying a plurality of fringe centers in the first image includes:

[0020] traversing pixel points in the first image one by one, and determining the pixel point as a pixel point belonging to the light streak when the brightness value of the pixel point is greater than a preset brightness threshold, and determining the pixel point as a pixel point not belonging to the light streak when the brightness value of the pixel point is not greater than the preset brightness threshold;

[0021] determining regions of the plurality of light streaks based on all the pixel points belonging to the light streaks and the pixel points not belonging to the light streaks;

[0022] The stripe center of each light stripe is obtained according to the brightness values and coordinates of all pixel points in the area of each light stripe, thereby obtaining the multiple stripe centers.

[0023] In some embodiments, the colors of the multiple light planes represented by the multiple fringe centers satisfy a preset rule, and the color of each fringe center is different from the color of an adjacent fringe center. Determining the code of the light plane corresponding to each fringe center based on the fringe characteristics of each fringe center includes:

[0024] The encoding of the light plane corresponding to each fringe center is determined according to the preset rule, the colors of the multiple fringe centers and the position of each fringe center in the first image, so as to obtain the encoding of the multiple light planes.

[0025] In some embodiments, the first image includes a first sub-image acquired by a first camera in the binocular system and a second sub-image acquired by a second camera in the binocular system, and calculating the fitting point set for each light plane based on the encodings of the plurality of light planes and the first image includes:

[0026] Obtaining an epipolar equation and a fundamental matrix in the epipolar equation;

[0027] Traversing each of the stripe centers in the first sub-image one by one, and calculating the coordinates of second pixels in the second sub-image that match each of the first pixels based on the coordinates of each first pixel in the current stripe center in the first sub-image, the epipolar equation, and the fundamental matrix;

[0028] determining the coordinates in space of each fitting point in a fitting point set of the light plane corresponding to the current fringe center based on the coordinates in space of the optical center of the first camera, the coordinates in space of each first pixel point, the coordinates in space of the optical center of the second camera, and the coordinates in space of a second pixel point matching each first pixel point, to obtain the fitting point set of the light plane corresponding to the current fringe center;

[0029] After traversing all the stripe centers in the first image, a fitting point set of each light plane is obtained.

[0030] In some embodiments, performing plane fitting on each light plane based on the fitting point set of each light plane to obtain a light plane expression corresponding to each light plane includes:

[0031] Traversing each fitting point set of the light plane one by one, setting N=1, and randomly selecting a preset number of fitting points from the current fitting point set of the light plane as a plurality of sample points, where N is a positive integer;

[0032] Performing plane fitting according to the plurality of sample points to obtain a light plane expression corresponding to the first plane;

[0033] Calculating a first distance from each fitting point in the current fitting point set of the light plane to the first plane;

[0034] When the first distance is less than a first preset threshold, determining that the fitting point is a point on the first plane, and adding the fitting point to the first point set;

[0035] Determine whether N is greater than a second preset threshold; if N is not greater than the second preset threshold, set N=N+1, and return to step 10 to randomly select a preset number of fitting points from the current fitting point set of the light plane as a plurality of sample points;

[0036] If N is greater than the second preset threshold, obtaining a plurality of first planes and a plurality of first point sets of the first planes;

[0037] taking the first point set with the largest number of points in the plurality of first point sets as a target point set;

[0038] Performing plane fitting based on all points in the target point set to obtain a light plane expression corresponding to the current light plane;

[0039] After traversing all the fitting point sets of the light planes, a light plane expression corresponding to each light plane is obtained.

[0040] In some implementations, performing plane fitting based on all points in the target point set to obtain a light plane expression corresponding to the current light plane includes:

[0041] Taking the average coordinate point of all points in the target point set as the initial calculation point of the current light plane;

[0042] Calculating an initial normal vector of the current light plane based on all points in the target point set;

[0043] Constructing a cost function according to all points in the target point set, an initial calculated point of the current light plane, and an initial normal vector of the current light plane;

[0044] Calculating and obtaining an optimized calculation point and an optimized normal vector of the current light plane that minimizes the value of the cost function;

[0045] Plane fitting is performed according to the optimized calculation points and optimized normal vectors of the current light plane to obtain a light plane expression corresponding to the current light plane.

[0046] In some embodiments, filtering the points in the corresponding three-dimensional point cloud to be filtered according to the light plane expression of each light plane to obtain a filtered three-dimensional point cloud includes:

[0047] Traversing each point in the three-dimensional point cloud to be filtered one by one, and calculating the distance from each point in the three-dimensional point cloud to be filtered to the corresponding light plane according to the light plane expression corresponding to each light plane;

[0048] When the distance between the point and the corresponding light plane is greater than a third preset threshold, deleting the point;

[0049] After traversing each point in the three-dimensional point cloud to be filtered, the filtered three-dimensional point cloud is obtained.

[0050] In some embodiments, when the distance between the point and the corresponding light plane is greater than a third preset threshold, deleting the point includes:

[0051] Determine the third preset threshold according to the distance from the point to the binocular system;

[0052] When the distance between the point and the corresponding light plane is greater than the third preset threshold, the point is deleted.

[0053] In a second aspect, an embodiment of the present application provides an electronic device, comprising:

[0054] at least one processor; and,

[0055] a memory communicatively connected to the at least one processor; wherein,

[0056] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the three-dimensional point cloud filtering method as described in the first aspect.

[0057] The present application provides a three-dimensional point cloud filtering method and electronic device. The present application obtains a filtered three-dimensional point cloud by filtering the points in the corresponding three-dimensional point cloud to be filtered according to the light plane expression of each light plane. The method can accurately filter out abnormal points in the three-dimensional point cloud according to the geometric principles of imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of the three-dimensional point cloud filtering method provided in an embodiment of the present application.

[0059] Figure 2 This is a schematic diagram of obtaining a first image provided in an embodiment of the present application.

[0060] Figure 3 yes Figure 1 Detailed flowchart of step S200.

[0061] Figure 4 yes Figure 3Detailed flowchart of step S250.

[0062] Figure 5 This is a structural diagram of an electronic device provided in an embodiment of the present application.

[0063] Figure 6 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0065] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0066] The present application provides a three-dimensional point cloud filtering method and electronic device, which filters the points in the corresponding three-dimensional point cloud to be filtered according to the light plane expression of each light plane to obtain a filtered three-dimensional point cloud, and can accurately filter out abnormal points in the three-dimensional point cloud according to the geometric principles of imaging.

[0067] The following will describe in detail the three-dimensional point cloud filtering method provided by this application with reference to the accompanying drawings.

[0068] See also Figure 1 , Figure 1 Schematic diagram of the process of the three-dimensional point cloud filtering method provided in the embodiment of the present application. Figure 1 As shown, the three-dimensional point cloud filtering method includes: steps S100 to S500.

[0069] Step S100: acquiring a first image of a light plane array projected by a light plane array emitter through a binocular system.

[0070] In some implementations, a binocular system includes a first camera and a second camera.

[0071] Optionally, the shooting positions of the first camera and the second camera are different.

[0072] In some embodiments, the binocular system further includes a light plane array emitter. To perform three-dimensional reconstruction of the target object, the light plane array emitter can project a light plane array onto the target object, wherein the light plane array includes a plurality of light planes.

[0073] Optionally, the number of light planes is more than 3, for example, 3, 4, 10, 20 or 30, etc.

[0074] See also Figure 2 , Figure 2 Schematic diagram of obtaining the first image provided by the embodiment of the present application. Figure 2 As shown, Figure 2 The light plane array 10 in FIG. 1 includes four light planes 11. When the light plane array 10 projects light onto the target object ( Figure 2 When the light plane array 10 is projected by the first camera 20 and the second camera 30 in the binocular system, a light stripe is formed on the target object. At this time, the target object is photographed by the first camera 20 and the second camera 30 in the binocular system, and the first image of the light plane array 10 projected by the light plane array emitter can be obtained.

[0075] Optionally, the target object may be an object used for image calibration, such as a calibration plate.

[0076] In some embodiments, the first camera in the binocular system may acquire a first sub-image of the light plane array projected onto the target object, and the second camera may acquire a second sub-image of the light plane array projected onto the target object.

[0077] In some embodiments, the first image includes a first sub-image acquired using a first camera in the binocular system and a second sub-image acquired using a second camera in the binocular system.

[0078] Optionally, the binocular system can be located in the 3D scanner.

[0079] Optionally, the light plane array emitter can be located in the 3D scanner.

[0080] Alternatively, the three-dimensional scanner may be an oral scanner, which is used to scan the user's oral cavity to obtain the user's oral scan data, and subsequently perform operations such as tooth restoration and replacement on the user based on the user's oral scan data.

[0081] According to the geometric principles of imaging, ideally, each point in the 3D point cloud obtained from the first image acquired by the binocular system corresponds to a light plane, and each point is located on the light plane corresponding to that point. Therefore, the distance from the point to the corresponding light plane can be used to determine whether the point is an outlier. To calculate the distance from a point to the corresponding light plane, it is necessary to first find the light plane expression corresponding to each light plane.

[0082] Step S200: Obtaining a light plane expression corresponding to each light plane in the light plane array by calculation based on the first image.

[0083] See also Figure 3 , Figure 3 yes Figure 1 Detailed flow chart of step S200. Figure 3 As shown, step S200 includes steps S210 to S250.

[0084] Step S210: Identify multiple fringe centers in the first image.

[0085] Because a light plane projected by the light plane array emitter forms a light stripe on the target object, each light stripe represents a light plane, and each light stripe indicates the position of a light plane.

[0086] Optionally, the stripe center is the center line of the area of the light stripe, and the stripe center of each light stripe represents a more accurate light plane, that is, the stripe center of each light stripe indicates a more accurate position of the light plane.

[0087] By identifying multiple stripe centers in the first image, compared to calculating the light plane expression based on the area of the light stripes, the light plane expression is subsequently calculated based on the multiple stripe centers, which can improve the accuracy of the light plane expression.

[0088] In some embodiments, a plurality of fringe centers are identified in the first sub-image and the second sub-image of the first image, respectively.

[0089] In some embodiments, step S210 includes the following steps (1) to (3).

[0090] (1) traverse the pixel points in the first image one by one, and when the brightness value of the pixel point is greater than a preset brightness threshold, determine the pixel point as a pixel point belonging to the light streak; when the brightness value of the pixel point is not greater than the preset brightness threshold, determine the pixel point as a pixel point not belonging to the light streak.

[0091] In some embodiments, the preset brightness threshold is in the range of 100 to 200, for example, the preset brightness threshold is 100, 150, or 200.

[0092] In some embodiments, when the first image is a color image and pixels in the first image include pixel values of multiple color channels, the first image is converted into a first grayscale image, and the pixel values of the pixels in the first grayscale image are brightness values.

[0093] (2) Determine the regions of the multiple light streaks based on all the pixels belonging to the light streaks and the pixels not belonging to the light streaks.

[0094] In some embodiments, an edge detection algorithm is used to detect the edge of the light stripe region based on all pixels belonging to the light stripe and pixels not belonging to the light stripe, and then the regions of the multiple light stripes are determined based on the edge of the light stripe region.

[0095] Optionally, the edge detection algorithm includes a gradient operator method, a high-pass filter method, and the like.

[0096] (3) The stripe center of each light stripe is obtained according to the brightness values and coordinates of all pixel points in the area of each light stripe, and multiple stripe centers are obtained.

[0097] In some embodiments, the first image includes a first coordinate axis and a second coordinate axis, the first coordinate axis and the second coordinate axis are perpendicular to each other, and each pixel has a coordinate on the first coordinate axis and a coordinate on the second coordinate axis.

[0098] Optionally, the first coordinate axis is one of the X axis and the Y axis, and the second coordinate axis is one of the X axis and the Y axis that is different from the first coordinate axis.

[0099] In some embodiments, step (3) includes the following steps (3.1) to (3.3):

[0100] (3.1) When the direction of the light streak is parallel to the direction of the first coordinate axis, a set of all pixel points with the same coordinates on the first coordinate axis in the area of the current light streak is determined as a second point set, and multiple second point sets are obtained.

[0101] (3.2) The stripe center pixel point corresponding to each second point set is calculated based on all pixel points in each second point set, and multiple stripe center pixel points are obtained.

[0102] Optionally, the coordinates of the pixel points in the current second point set on the first coordinate axis are used as the coordinates of a stripe center pixel point on the current stripe center on the first coordinate axis.

[0103] Optionally, coordinates of all pixel points in the current second point set on the second coordinate axis are weightedly calculated to obtain coordinates of a stripe center pixel point at the center of the current stripe on the second coordinate axis.

[0104] Optionally, a formula for weighted calculation of the coordinates of all pixel points in the current second point set on the second coordinate axis may be:

[0105]

[0106] in, represents the coordinates of a stripe center pixel point on the current stripe center on the second coordinate axis, a represents the coordinates of the pixel point in the current second point set on the second coordinate axis, and w represents the pixel value of the pixel point in the current second point set.

[0107] Optionally, the pixel value of the pixel point is a brightness value.

[0108] After obtaining the coordinates of the center pixel point of the stripe on the first coordinate axis and the coordinates of the center pixel point of the stripe on the second coordinate axis, the coordinates of the center pixel point of the stripe are determined, and thus the position of the center pixel point of the stripe in the image is determined.

[0109] (3.3) The set of multiple stripe center pixel points on the stripe center is taken as the stripe center of the area of the current light stripe.

[0110] In some implementations, a center line between two edge lines of the current light stripe region is used as the stripe center of the current light stripe region.

[0111] Step S220: Identify the stripe features at the center of each stripe in the first image.

[0112] The center of each stripe represents a light plane.

[0113] In some embodiments, the stripe feature at the stripe center includes a temporal feature or a spatial feature.

[0114] In some embodiments, the colors of the multiple light planes represented by the multiple stripe centers satisfy a preset rule.

[0115] In some embodiments, the spatial characteristic of the stripe centers is that the color of each stripe center is different from the color of the adjacent stripe centers.

[0116] Step S230: determining the code of the light plane corresponding to each fringe center according to the fringe characteristics of each fringe center.

[0117] In some embodiments, the colors of the multiple light planes represented by the multiple stripe centers satisfy a preset rule, and the color of each stripe center is different from the color of an adjacent stripe center.

[0118] Optionally, the preset rule is that the colors of the multiple light planes are arranged according to a preset color order.

[0119] Optionally, the preset color sequence includes repeated preset sequence units, each of which includes at least two different colors. For example, the preset sequence unit includes three colors, which are blue, green, and cyan.

[0120] Optionally, the preset color sequence includes the color and light plane code of each light plane.

[0121] Optionally, the color of one light plane is different from the color of an adjacent light plane in the preset color sequence. Therefore, the color of each stripe center is different from the color of the adjacent stripe centers.

[0122] Exemplarily, the preset color sequence can be blue, green, cyan, blue, green, cyan, blue, green, cyan, where the color of the light plane is different from the color of the adjacent light plane, and therefore the color of each stripe center is different from the color of the adjacent stripe center.

[0123] In the above manner, when the colors of the multiple light planes meet the preset rule, the color of each stripe center can be made different from the color of the adjacent stripe centers, which makes it easier to distinguish each stripe center.

[0124] In some embodiments, the encoding of the light plane corresponding to each fringe center is determined according to a preset rule, the colors of the multiple fringe centers, and the position of each fringe center in the first image, to obtain the encoding of the multiple light planes.

[0125] Optionally, when the stripes are vertical stripes, the position order of the stripe centers is determined according to the order from left to right of each stripe center in the first image. The position order of the colors is determined according to the order from left to right of the colors of each light plane in a preset color order.

[0126] Optionally, when the stripes are horizontal stripes, the position order of the stripe centers is determined according to the order of the stripe centers from top to bottom in the first image. The position order of the colors is determined according to the order of the colors of each light plane from top to bottom in a preset color order.

[0127] Optionally, based on the position order of each stripe center in the first image, it is detected whether the color of the stripe center in each position order is the same as the color of the same position order in the preset color order. If they are the same, the light plane encoding corresponding to the same position order in the preset color order is used as the encoding of the light plane corresponding to the stripe center.

[0128] For example, in the first image, the center of a stripe is the second stripe center from left to right, and it is detected whether the color of the stripe center is the same as the second color from left to right in the preset color sequence. If they are the same, the code of the light plane corresponding to the stripe center is determined to be 2.

[0129] Step S240: Calculate a fitting point set for each light plane according to the codes of the multiple light planes and the first image.

[0130] As described above, in some embodiments, the first image includes a first sub-image acquired by a first camera in the binocular system and a second sub-image acquired by a second camera in the binocular system.

[0131] In some embodiments, step S240 includes the following steps (1) to (4).

[0132] (1) Obtain the polar equation and the fundamental matrix in the polar equation.

[0133] In a binocular system, for a point on the target object, the corresponding points in the first sub-image and the second sub-image should be on a straight line (ie, an epipolar line) on the same plane.

[0134] Alternatively, the epipolar equation is:

[0135]

[0136] Where p1 represents the coordinates of the pixel in the first sub-image, represents the transpose of the coordinates of the pixel points in the second sub-image, and F represents the basic matrix in the epipolar equation.

[0137] Optionally, the fundamental matrix in the epipolar equation has been determined and can be obtained directly.

[0138] (2) Traverse each stripe center in the first sub-image one by one, and calculate the coordinates of the second pixel points in the second sub-image that match each first pixel point based on the coordinates of each first pixel point in the current stripe center in the first sub-image, the epipolar equation and the basic matrix.

[0139] Optionally, the coordinates of each first pixel point and the basic matrix are substituted into the epipolar equation to obtain the coordinates of the second pixel point that matches each first pixel point.

[0140] (3) Determine the coordinates of each fitting point in the fitting point set of the light plane corresponding to the current fringe center according to the coordinates of the optical center of the first camera in space, the coordinates of each first pixel point in space, the coordinates of the optical center of the second camera in space, and the coordinates of the second pixel point matching each first pixel point in space, and obtain the fitting point set of the light plane corresponding to the current fringe center.

[0141] Optionally, each of the first pixel point and the second pixel point has coordinates in space.

[0142] In some embodiments, a mathematical expression of a first straight line passing through the optical center of the first camera and the current first pixel is determined according to the coordinates of the optical center of the first camera in space and the coordinates of the current first pixel in space.

[0143] In some embodiments, a mathematical expression of a second straight line passing through the optical center of the second camera and the current second pixel point is determined according to the coordinates of the optical center of the second camera in space and the coordinates of the current second pixel point in space.

[0144] In some embodiments, if the first straight line intersects the second straight line, the intersection point of the first straight line and the second straight line is determined as a fitting point in the fitting point set of the light plane corresponding to the current fringe center.

[0145] In some embodiments, if the first straight line and the second straight line are skew lines, the midpoint of the common perpendicular segment between the first straight line and the second straight line is determined as a fitting point in the fitting point set of the light plane corresponding to the current fringe center.

[0146] According to mathematical principles, there is a unique common perpendicular between two skew lines. This common perpendicular is a line that is perpendicular to both lines. The common perpendicular segment between the first and second lines is determined by the intersection of the common perpendicular with the first line and the intersection of the common perpendicular with the second line.

[0147] (4) After traversing all the stripe centers in the first image, a set of fitting points for each light plane is obtained.

[0148] Step S250: performing plane fitting on each light plane based on the fitting point set of each light plane to obtain a light plane expression corresponding to each light plane.

[0149] See also Figure 4 , Figure 4 yes Figure 3 Detailed flow chart of step S250 in FIG. Figure 4 As shown, step S250 includes steps S251 to S259.

[0150] In some embodiments, the fitting point set of each light plane is traversed one by one, and N=1 is set, where N is a positive integer.

[0151] Step S251: randomly selecting a preset number of fitting points from the fitting point set of the current light plane as a plurality of sample points.

[0152] In some embodiments, the preset number is greater than 3, for example, the preset number is 3, 5, 10, or 20.

[0153] Step S252: performing plane fitting according to the plurality of sample points to obtain a light plane expression corresponding to the first plane.

[0154] In some implementations, the average coordinate point of the plurality of sample points is used as the calculation point of the first plane.

[0155] In some implementations, the normal vector of the first plane is calculated based on a plurality of sample points.

[0156] Optionally, the coordinates of multiple sample points are combined into a first matrix, and singular value decomposition (SVD) is performed on the first matrix to obtain a first orthogonal matrix and a second orthogonal matrix, where one column vector of the first orthogonal matrix or the second orthogonal matrix is a normal vector of the first plane.

[0157] In some embodiments, plane fitting is performed based on the calculation points of the first plane and the normal vector of the first plane to obtain a light plane expression corresponding to the first plane.

[0158] In the above manner, compared with a method of substituting the coordinates of multiple sample points into a light plane expression to be determined, the fitting error can be reduced and a more accurate light plane expression can be obtained.

[0159] In some embodiments, when the number of the plurality of sample points is 3, since 3 points determine 1 plane, the light plane expression corresponding to the first plane can be directly calculated according to the coordinates of the 3 sample points.

[0160] In some embodiments, the light plane expression may be: Ax+By+Cz+1=0, where x represents the x-axis in the spatial coordinate system, y represents the y-axis in the spatial coordinate system, z represents the z-axis in the spatial coordinate system, and A, B, and C are all constants.

[0161] Optionally, the light plane expression includes multiple light plane parameters, for example, the multiple light plane parameters include A, B and C.

[0162] Step S253: Calculate a first distance from each fitting point in the fitting point set of the current light plane to the first plane.

[0163] Optionally, the distance from the fitting point to the first plane is calculated according to the coordinates of each fitting point in space and the light plane expression corresponding to the first plane, and the distance is also the first distance.

[0164] Step S254: When the first distance is less than the first preset threshold, determine that the fitting point is a point on the first plane, and add the fitting point to the first point set.

[0165] Optionally, the first preset threshold value ranges from 0.1 mm (millimeter) to 1 mm, for example, the first preset threshold value is 0.1 mm, 0.2 mm, 0.3 mm, 0.4 mm, 0.5 mm or 1 mm.

[0166] Step S255: Determine whether N is greater than a second preset threshold.

[0167] Optionally, each first plane and each first point set of the first plane are recorded.

[0168] Optionally, a first point set with the largest number of points in all currently recorded first point sets is determined as the third point set, and the number of points in the third point set is divided by the number of fitting points in the fitting point set of the current light plane to obtain a first ratio.

[0169] Optionally, the second preset threshold is determined according to the first ratio.

[0170] In some implementations, the formula for calculating the second preset threshold may be:

[0171] lg(0.01) / lg(1-r 3 ),

[0172] Here, r represents the first ratio.

[0173] In step S255 , if N is not greater than the second preset threshold, set N=N+1 and return to step S251 .

[0174] Step S256: If N is greater than a second preset threshold, a plurality of first planes and a plurality of first point sets of the first planes are obtained.

[0175] Step S257: taking the first point set with the largest number of points among the multiple first point sets as the target point set.

[0176] Step S258: performing plane fitting based on all points in the target point set to obtain a light plane expression corresponding to the current light plane.

[0177] In some embodiments, step S258 includes the following steps (1) to (5).

[0178] (1) The average coordinate point of all points in the target point set is used as the initial calculation point of the current light plane.

[0179] (2) The initial normal vector of the current light plane is calculated based on all points in the target point set.

[0180] In some embodiments, the coordinates of all points in the target point set are combined into a second matrix, and the second matrix is subjected to singular value decomposition to obtain a third orthogonal matrix and a fourth orthogonal matrix, wherein one of the column vectors of the third orthogonal matrix or the fourth orthogonal matrix is the initial normal vector of the current light plane.

[0181] In some implementations, plane fitting is performed directly based on the initial calculation points and initial normal vectors of the current light plane to obtain a light plane expression corresponding to the current light plane.

[0182] In some implementations, the initial calculated point and initial normal vector of the current light plane may not be accurate enough, and the initial calculated point and initial normal vector of the current light plane need to be optimized.

[0183] In some embodiments, the method for optimizing the initial calculation point and the initial normal vector of the current light plane includes a linear optimization method, a nonlinear optimization method, a genetic algorithm, a particle swarm optimization algorithm, a Bayesian optimization method, or the like.

[0184] Optionally, a nonlinear optimization method is used to optimize the initial calculation point and the initial normal vector of the current light plane.

[0185] (3) Construct a cost function based on all points in the target point set, the initial calculation point of the current light plane, and the initial normal vector of the current light plane.

[0186] In some implementations, the cost function may be formulated as:

[0187] Where D represents the target point set, n represents the optimized normal vector of the current light plane, p represents the optimized calculation point of the current light plane, i represents the sequence number of the point in the target point set, and p i Represents the i-th point in the target point set. The initial value of n is the initial normal vector of the current light plane, and the initial value of p is the initial calculation point of the current light plane.

[0188] (4) Calculate the optimized calculation point and optimized normal vector of the current light plane that minimizes the value of the cost function.

[0189] In some embodiments, a gradient descent method is used to calculate the optimized calculation point and the optimized normal vector of the current light plane that minimize the value of the cost function according to the initial value of n and the initial value of p.

[0190] In multidimensional space, the gradient of a function is a vector whose direction points to the direction in which the function grows fastest, and its magnitude indicates the rate of growth.

[0191] Optionally, calculate the gradient of the current cost function, update the value of n and the value of p along the opposite direction of the gradient, and repeat the steps of calculating the gradient of the current cost function, updating the value of n and the value of p along the opposite direction of the gradient until the termination condition is met to obtain the final value of n and the value of p.

[0192] Optionally, the termination condition is that the difference between the gradient value and 0 is less than a fourth preset threshold, indicating that the minimum point may be approached.

[0193] Optionally, the termination condition is that the number of repeated executions reaches a fifth preset threshold.

[0194] (5) Perform plane fitting based on the optimized calculation points and optimized normal vector of the current light plane to obtain the light plane expression corresponding to the current light plane.

[0195] Through the above method, the accuracy of the light plane expression corresponding to the current light plane can be improved, and then the accuracy of subsequent filtering of the points in the corresponding three-dimensional point cloud to be filtered according to the light plane expression of each light plane can be improved, and abnormal points in the three-dimensional point cloud can be filtered out more accurately.

[0196] Step S259: After traversing the fitting point set of each light plane, the light plane expression corresponding to each light plane is obtained.

[0197] Step S300: controlling the light plane array emitter to project the light plane array onto the object to be measured, and acquiring a second image of the light plane array projected onto the object to be measured through the binocular system.

[0198] In some embodiments, the object to be measured is any object for which three-dimensional reconstruction is to be performed.

[0199] Optionally, the object to be tested is gums or teeth.

[0200] In some embodiments, the second image includes a third sub-image acquired by a first camera in the binocular system and a fourth sub-image acquired by a second camera in the binocular system.

[0201] Step S400: obtaining a three-dimensional point cloud to be filtered according to the second image.

[0202] In some embodiments, the method for obtaining the three-dimensional point cloud to be filtered based on the second image may refer to the method for calculating the fitting point set of each light plane based on the first image.

[0203] In some embodiments, step S400 includes the following steps (1) to (5).

[0204] (1) Identify multiple fringe centers in the second image.

[0205] In some embodiments, multiple fringe centers in the third sub-image and multiple fringe centers in the fourth sub-image are identified separately.

[0206] Optionally, step (1) refers to the description in step S210.

[0207] (2) Identify the stripe features at the center of each stripe in the second image.

[0208] Optionally, step (3) refers to the description in step S220.

[0209] (3) Determine the encoding of the light plane corresponding to each fringe center based on the fringe characteristics of each fringe center.

[0210] Optionally, step (3) refers to the description in step S230.

[0211] (4) A point set of each light plane is obtained by calculation based on the encoding of the multiple light planes and the second image.

[0212] Optionally, step (4) refers to the description in step S240.

[0213] (5) The point set of all light planes is determined as the three-dimensional point cloud to be filtered.

[0214] Each point in the three-dimensional point cloud to be filtered corresponds to a light plane.

[0215] Step S500: filtering points in the corresponding three-dimensional point cloud to be filtered according to the light plane expression of each light plane to obtain a filtered three-dimensional point cloud.

[0216] In some embodiments, step S500 includes the following steps (1) to (3).

[0217] (1) Each point in the three-dimensional point cloud to be filtered is traversed one by one, and the distance from each point in the three-dimensional point cloud to be filtered to the corresponding light plane is calculated according to the light plane expression corresponding to each light plane.

[0218] Optionally, the distance from the point to the corresponding light plane is calculated according to the coordinates of each point in the three-dimensional point cloud to be filtered in space and a light plane expression of the light plane corresponding to the point.

[0219] (2) When the distance from a point to the corresponding light plane is greater than a third preset threshold, the point is deleted.

[0220] In some embodiments, the third preset threshold is determined according to the distance from the point to the binocular system.

[0221] In some embodiments, the distance from the point to the optical center of the first camera and the distance from the optical center of the second camera in the binocular system are equal, and this distance is used as the distance from the point to the binocular system.

[0222] In some embodiments, the average of the distance from the point to the optical center of the first camera and the distance from the point to the optical center of the second camera is used as the distance from the point to the binocular system.

[0223] In some implementations, the formula for determining the third preset threshold may be:

[0224] thresh=0.001d+0.1,

[0225] Wherein, thresh represents the third preset threshold, d represents the distance from the point to the binocular system, and the unit of the third preset threshold is mm.

[0226] In some embodiments, d ranges from 50 mm to 200 mm, for example, the third preset threshold is 50 mm, 60 mm, 80 mm, 90 mm, 100 mm, 130 mm, 150 mm or 200 mm, etc.

[0227] (3) After traversing each point in the three-dimensional point cloud to be filtered, the filtered three-dimensional point cloud is obtained.

[0228] In some embodiments, the object to be measured can be three-dimensionally reconstructed based on the filtered three-dimensional point cloud to obtain an accurate three-dimensional model of the object to be measured.

[0229] In summary, the 3D point cloud filtering method provided by the embodiments of the present application has the following advantages:

[0230] 1. By filtering the points in the corresponding 3D point cloud to be filtered according to the light plane expression of each light plane, a filtered 3D point cloud is obtained, which can accurately filter out abnormal points in the 3D point cloud according to the geometric principle of imaging.

[0231] 2. By having the colors of multiple light planes meet preset rules, the color of each stripe center can be made different from the colors of adjacent stripe centers, making it easier to distinguish each stripe center.

[0232] 3. The method of performing plane fitting by calculating the calculation points and normal vectors of the first plane using multiple sample points can reduce the fitting error and obtain a more accurate light plane expression compared to the method of substituting the coordinates of multiple sample points into the undetermined light plane expression.

[0233] 4. By optimizing the calculation points and normal vectors of the current light plane, the accuracy of the light plane expression corresponding to the current light plane can be improved, thereby improving the accuracy of subsequent filtering of points in the corresponding three-dimensional point cloud to be filtered according to the light plane expression of each light plane, and more accurately filtering out abnormal points in the three-dimensional point cloud.

[0234] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 400 includes: one or more processors 410 and a memory 420, Figure 5 A processor 410 is taken as an example.

[0235] In some embodiments, the processor 410 and the memory 420 may be connected via a bus or other means. Figure 5 The bus connection is taken as an example.

[0236] In some embodiments, the processor 410 is configured to acquire a first image of a light plane array projected by a light plane array emitter through a binocular system; calculate a light plane expression corresponding to each light plane in the light plane array based on the first image; control the light plane array emitter to project the light plane array toward the object to be measured, and acquire a second image of the light plane array projected on the object to be measured through the binocular system; acquire a three-dimensional point cloud to be filtered based on the second image, wherein each point in the three-dimensional point cloud to be filtered corresponds to a light plane; and filter the points in the corresponding three-dimensional point cloud to be filtered based on the light plane expression of each light plane to obtain a filtered three-dimensional point cloud. The present application obtains a filtered three-dimensional point cloud by filtering the points in the corresponding three-dimensional point cloud to be filtered based on the light plane expression of each light plane, and can accurately filter out abnormal points in the three-dimensional point cloud based on the geometric principles of imaging.

[0237] In some embodiments, memory 420, as a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules for the 3D point cloud filtering method in the embodiments of this application. Processor 410 executes the non-volatile software programs, instructions, and modules stored in memory 420 to execute various functional applications and data processing of electronic device 400, thereby implementing the 3D point cloud filtering method in the aforementioned method embodiment.

[0238] In some embodiments, the memory 420 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device 400, etc. In addition, the memory 420 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 420 may optionally include a memory remotely located relative to the processor 410, and these remote memories may be connected to the controller via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0239] In some embodiments, one or more modules are stored in the memory 420, and when executed by one or more processors 410, perform the three-dimensional point cloud filtering method in any of the above method embodiments, for example, perform the above described Figure 1 Method steps S100 to S500.

[0240] Please refer to Figure 6 , Figure 6The computer-readable storage medium 500 stores program code 510, which can be called by a processor to execute the three-dimensional point cloud filtering method described in the above method embodiment.

[0241] The computer-readable storage medium 500 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium 500 has storage space for program code that executes any of the method steps in the above-described three-dimensional point cloud filtering method. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable format.

[0242] In summary, the present application provides a three-dimensional point cloud filtering method and electronic device, the three-dimensional point cloud filtering method comprising obtaining a first image of a light plane array projected by a light plane array transmitter through a binocular system; calculating a light plane expression corresponding to each light plane in the light plane array based on the first image; controlling the light plane array transmitter to project the light plane array onto the object to be measured, and obtaining a second image of the light plane array projected on the object to be measured through the binocular system; obtaining a three-dimensional point cloud to be filtered based on the second image, wherein each point in the three-dimensional point cloud to be filtered corresponds to a light plane; filtering the points in the corresponding three-dimensional point cloud to be filtered according to the light plane expression of each light plane to obtain a filtered three-dimensional point cloud. The present application obtains a filtered three-dimensional point cloud by filtering the points in the corresponding three-dimensional point cloud to be filtered according to the light plane expression of each light plane, and can accurately filter out abnormal points in the three-dimensional point cloud according to the geometric principles of imaging.

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A three-dimensional point cloud filtering method, characterized in that: include: acquiring, through the binocular system, a first image of the light plane array projected by the light plane array emitter; Obtaining a light plane expression corresponding to each light plane in the light plane array by calculation based on the first image; The calculating based on the first image to obtain a light plane expression corresponding to each light plane in the light plane array includes: identifying a plurality of fringe centers in the first image; identifying fringe features of each fringe center in the first image, wherein each fringe center represents a light plane; Determining the code of the light plane corresponding to each of the fringe centers according to the fringe characteristics of each of the fringe centers; Obtaining a fitting point set for each light plane by calculation according to the codes of the plurality of light planes and the first image; Performing plane fitting on each light plane based on a fitting point set of each light plane to obtain a light plane expression corresponding to each light plane; controlling the light plane array emitter to project the light plane array toward the object to be measured, and acquiring a second image of the light plane array projected on the object to be measured through the binocular system; Acquire a three-dimensional point cloud to be filtered according to the second image, wherein each point in the three-dimensional point cloud to be filtered corresponds to one of the light planes; Points in the corresponding three-dimensional point cloud to be filtered are filtered according to the light plane expression of each light plane to obtain a filtered three-dimensional point cloud.

2. The three-dimensional point cloud filtering method according to claim 1, characterized in that: The identifying a plurality of fringe centers in the first image comprises: traversing pixel points in the first image one by one, and determining the pixel point as a pixel point belonging to the light streak when the brightness value of the pixel point is greater than a preset brightness threshold, and determining the pixel point as a pixel point not belonging to the light streak when the brightness value of the pixel point is not greater than the preset brightness threshold; determining regions of the plurality of light streaks based on all the pixel points belonging to the light streaks and the pixel points not belonging to the light streaks; The stripe center of each light stripe is obtained according to the brightness values and coordinates of all pixel points in the area of each light stripe, thereby obtaining the multiple stripe centers.

3. The three-dimensional point cloud filtering method according to claim 1, characterized in that: The colors of the multiple light planes represented by the multiple fringe centers satisfy a preset rule, the color of each fringe center is different from the color of an adjacent fringe center, and the determining the code of the light plane corresponding to each fringe center according to the fringe characteristics of each fringe center includes: The encoding of the light plane corresponding to each fringe center is determined according to the preset rule, the colors of the multiple fringe centers and the position of each fringe center in the first image, so as to obtain the encoding of the multiple light planes.

4. The three-dimensional point cloud filtering method according to claim 1, characterized in that: The first image includes a first sub-image acquired by a first camera in the binocular system and a second sub-image acquired by a second camera in the binocular system, and the step of calculating a fitting point set for each light plane based on the codes of the plurality of light planes and the first image includes: Obtaining an epipolar equation and a fundamental matrix in the epipolar equation; Traversing each of the stripe centers in the first sub-image one by one, and calculating the coordinates of second pixels in the second sub-image that match each of the first pixels based on the coordinates of each first pixel in the current stripe center in the first sub-image, the epipolar equation, and the fundamental matrix; determining the coordinates in space of each fitting point in a fitting point set of the light plane corresponding to the current fringe center based on the coordinates in space of the optical center of the first camera, the coordinates in space of each first pixel point, the coordinates in space of the optical center of the second camera, and the coordinates in space of a second pixel point matching each first pixel point, to obtain the fitting point set of the light plane corresponding to the current fringe center; After traversing all the stripe centers in the first image, a fitting point set of each light plane is obtained.

5. The three-dimensional point cloud filtering method according to claim 1, characterized in that: The performing plane fitting on each light plane based on the fitting point set of each light plane to obtain a light plane expression corresponding to each light plane includes: Traversing each fitting point set of the light plane one by one, setting N=1, and randomly selecting a preset number of fitting points from the current fitting point set of the light plane as a plurality of sample points, where N is a positive integer; Performing plane fitting according to the plurality of sample points to obtain a light plane expression corresponding to the first plane; Calculating a first distance from each fitting point in the current fitting point set of the light plane to the first plane; When the first distance is less than a first preset threshold, determining that the fitting point is a point on the first plane, and adding the fitting point to the first point set; Determine whether N is greater than a second preset threshold; if N is not greater than the second preset threshold, set N=N+1, and return to step 10 to randomly select a preset number of fitting points from the current fitting point set of the light plane as a plurality of sample points; If N is greater than the second preset threshold, obtaining a plurality of first planes and a plurality of first point sets of the first planes; taking the first point set with the largest number of points in the plurality of first point sets as a target point set; Performing plane fitting based on all points in the target point set to obtain a light plane expression corresponding to the current light plane; After traversing all the fitting point sets of the light planes, a light plane expression corresponding to each light plane is obtained.

6. The three-dimensional point cloud filtering method according to claim 5, characterized in that: The performing plane fitting according to all points in the target point set to obtain a light plane expression corresponding to the current light plane includes: Taking the average coordinate point of all points in the target point set as the initial calculation point of the current light plane; Calculating an initial normal vector of the current light plane based on all points in the target point set; Constructing a cost function according to all points in the target point set, an initial calculated point of the current light plane, and an initial normal vector of the current light plane; Calculating and obtaining an optimized calculation point and an optimized normal vector of the current light plane that minimizes the value of the cost function; Plane fitting is performed according to the optimized calculation points and optimized normal vectors of the current light plane to obtain a light plane expression corresponding to the current light plane.

7. The three-dimensional point cloud filtering method according to claim 1, characterized in that: The filtering of the points in the corresponding three-dimensional point cloud to be filtered according to the light plane expression of each light plane to obtain a filtered three-dimensional point cloud includes: Traversing each point in the three-dimensional point cloud to be filtered one by one, and calculating the distance from each point in the three-dimensional point cloud to be filtered to the corresponding light plane according to the light plane expression corresponding to each light plane; When the distance between the point and the corresponding light plane is greater than a third preset threshold, deleting the point; After traversing each point in the three-dimensional point cloud to be filtered, the filtered three-dimensional point cloud is obtained.

8. The three-dimensional point cloud filtering method according to claim 7, characterized in that: When the distance between the point and the corresponding light plane is greater than a third preset threshold, deleting the point includes: Determine the third preset threshold according to the distance from the point to the binocular system; When the distance between the point and the corresponding light plane is greater than the third preset threshold, the point is deleted.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the three-dimensional point cloud filtering method according to any one of claims 1 to 8.

Images