Method, apparatus, and system for generating point cloud of three-dimensional model of object to be modeled

Abstract

The present invention discloses a method, device and system for generating a point cloud of a three-dimensional model of an object to be modeled. The method of the present invention respectively collects a structured light scanning point cloud and an MVS point cloud of the object to be modeled, and then uses the PCA algorithm and the ICP algorithm to perform rough registration and fine registration on the structured light scanning point cloud and the MVS point cloud in sequence, so that the MVS point cloud after rough registration and fine registration is accurately transformed into the coordinate system of the structured light scanning point cloud, effectively avoiding the disadvantage that the MVS point cloud obtained based on a monocular camera cannot reflect the true size of the object to be modeled and affects the fusion with the structured light scanning point cloud. Further, based on the structured light scanning point cloud, the registered MVS point cloud is used to supplement the missing areas in the structured light scanning point cloud, thereby improving the integrity of the supplemented structured light scanning point cloud and avoiding errors or failures in the three-dimensional modeling of the object to be modeled.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional measurement and modeling, and particularly relates to a method, device and system for generating a point cloud of a three-dimensional model of an object to be modeled. Background Art

[0002] Three-dimensional reconstruction is a hot issue in computer vision. Currently, there are two ways to obtain the point cloud of an object to be modeled, namely active and passive. Among them, the structured light scanning method is a typical active three-dimensional modeling technology. By projecting stripes or speckles to encode the object to be modeled, and then combining with the pre-calibrated system parameters to calculate the height information of the object surface, so as to obtain the point cloud of the object to be modeled. And the three-dimensional reconstruction of an object based on multiple views is the representative of passive three-dimensional reconstruction. This type of method usually uses a monocular camera to collect images of the object to be modeled, then restores the pose of the camera and the sparse point cloud of the scene through the SFM algorithm, and finally reconstructs the MVS point cloud describing the surface of the object to be modeled through the MVS algorithm.

[0003] However, the MVS point cloud obtained by three-dimensional reconstruction from the images collected by a monocular camera currently cannot obtain the true size of the object, so it cannot be fused with the point cloud obtained by the structured light scanning method to achieve accurate modeling of the object to be modeled. Summary of the Invention

[0004] Based on the above situation, the main purpose of the present invention is to provide a method, device and system for generating a point cloud of a three-dimensional model of an object to be modeled.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] A method for generating a point cloud of a three-dimensional model of an object to be modeled includes the following steps:

[0007] Calculating the structured light scanning point cloud of the object to be modeled by the structured light scanning method, using a monocular camera to capture images of the object to be modeled at multiple viewpoints, and calculating the initial MVS point cloud of the object to be modeled through the MVS algorithm, wherein the structured light scanning point cloud and the initial MVS point cloud have different scales;

[0008] Filtering the initial MVS point cloud to obtain a preprocessed MVS point cloud;

[0009] Using the PCA algorithm to analyze the initial scale estimation and initial pose transformation between the structured light scanning point cloud and the preprocessed MVS point cloud. Taking the structured light scanning point cloud as the target point cloud, based on the initial scale estimation and the initial pose transformation, transforming the preprocessed MVS point cloud into the coordinate system of the structured light scanning point cloud to obtain a first transformed MVS point cloud;

[0010] Iteratively analyze the precise scale estimation and precise pose transformation between the structured light scanned point cloud and the first transformed MVS point cloud through an ICP algorithm based on scale estimation. Based on the precise scale estimation and the precise pose transformation, transform the first transformed MVS point cloud to the coordinate system of the structured light scanned point cloud to obtain a second transformed MVS point cloud;

[0011] Traverse all the MVS points in the second transformed MVS point cloud. Respectively, with a single MVS point as the center of a sphere, construct a preset area corresponding to each MVS point within a preset radius;

[0012] Respectively calculate the total number of all scanned points of the structured light scanned point cloud within each preset area to obtain the number of scanned points in each preset area;

[0013] Select several preset areas with the number of scanned points less than a threshold value from each preset area as missing areas;

[0014] Add the MVS points at the corresponding positions of each missing area in the second transformed MVS point cloud to each missing area of the structured light scanned point cloud to obtain the complete point cloud of the three-dimensional model of the object to be modeled.

[0015] Preferably, the initial pose transformation includes an initial rotation matrix and an initial translation vector. The step of using the PCA algorithm to analyze the initial scale estimation and the initial pose transformation between the structured light scanned point cloud and the preprocessed MVS point cloud includes:

[0016] S301: Perform an averaging calculation on all the scanned points of the structured light scanned point cloud to obtain the first point cloud centroid corresponding to the structured light scanned point cloud, and perform an averaging calculation on all the MVS points of the preprocessed MVS point cloud to obtain the second point cloud centroid corresponding to the preprocessed MVS point cloud;

[0017] S302: Calculate the first covariance matrix corresponding to the structured light scanned point cloud based on the first point cloud centroid, calculate the second covariance matrix corresponding to the preprocessed MVS point cloud based on the second point cloud centroid, and solve for the first eigenvalue and the first left singular matrix of the first covariance matrix, and the second eigenvalue and the second left singular matrix of the second covariance matrix through the method of singular value decomposition;

[0018] S303: Determine the first vector axis according to the maximum value of the first eigenvalue in the eigenvector corresponding to the first left singular matrix, and determine the second vector axis according to the maximum value of the second eigenvalue in the eigenvector corresponding to the second left singular matrix;

[0019] S304: Obtain the first longest side of the oriented bounding box of the structured light scanned point cloud according to the difference between the maximum projection and the minimum projection of each of the scanned points on the first vector axis, and obtain the second longest side of the oriented bounding box of the preprocessed MVS point cloud according to the difference between the maximum projection and the minimum projection of each of the MVS points on the second vector axis;

[0020] S305: Calculate the ratio of the first longest side to the second longest side to obtain the initial scale estimate;

[0021] S306: Use the matrix inversion of the first left singular matrix and the second left singular matrix for matrix multiplication to obtain the initial rotation matrix,

[0022] S307: Use the centroid of the first point cloud minus the product of the initial scale estimate, the initial rotation matrix, and the centroid of the second point cloud to obtain the initial translation vector.

[0023] Preferably, after the step of using the centroid of the first point cloud minus the product of the initial scale estimate, the initial rotation matrix, and the centroid of the second point cloud to obtain the initial translation vector, it includes:

[0024] Taking the structured light scanned point cloud as the target point cloud, transform the preprocessed MVS point cloud to the coordinate system of the structured light scanned point cloud based on the initial scale estimate, the initial rotation matrix, and the initial translation vector to obtain the initial transformed MVS point cloud;

[0025] Calculate the point cloud average mean square error between the initial transformed MVS point cloud and the structured light scanned point cloud, and determine whether the point cloud average mean square error is greater than the error threshold;

[0026] If the point cloud average mean square error is greater than the error threshold, then reverse the column vectors of the second left singular matrix, and recalculate according to steps S303 - S307 to obtain several new groups of rotation matrices and translation vectors;

[0027] Select a rotation matrix and a translation vector with the minimum loss function from each of the rotation matrices and each of the translation vectors as the initial rotation matrix and the translation vector.

[0028] Preferably, the step of filtering the initial MVS point cloud to obtain the preprocessed MVS point cloud includes:

[0029] Obtain the point depths of each MVS point of the initial MVS point cloud, and remove the MVS points whose point depths exceed the depth threshold to obtain the primary MVS point cloud;

[0030] Assign color attributes to the primary MVS point cloud according to the colors of the pixel points corresponding to each MVS point in the primary MVS point cloud, and remove background points based on the similarity between the primary MVS point cloud with the color attributes and the background color to obtain a secondary MVS point cloud;

[0031] Filter out the outliers in the secondary MVS point cloud through a statistical filter to obtain the preprocessed MVS point cloud, where the outliers represent the MVS points in the secondary MVS point cloud whose distance from the nearest MVS point is greater than the distance threshold.

[0032] Preferably, the step of using a monocular camera to capture images of the object to be modeled from multiple viewpoints and calculating the initial MVS point cloud of the object to be modeled through the MVS algorithm includes:

[0033] Use the SFM algorithm to perform pose estimation on each of the viewpoint images to obtain the camera estimation parameters and sparse 3D point clouds respectively corresponding to each of the viewpoint images;

[0034] Input each of the camera estimation parameters and each of the sparse 3D point clouds into the MVSNet neural network for depth estimation to obtain the initial MVS point cloud.

[0035] The present invention also provides a device for generating a three-dimensional model point cloud of an object to be modeled, including:

[0036] An initial MVS point cloud generation module, configured to calculate the structured light scanning point cloud of the object to be modeled through the structured light scanning method, use a monocular camera to capture images of the object to be modeled from multiple viewpoints, and calculate the initial MVS point cloud of the object to be modeled through the MVS algorithm, where the structured light scanning point cloud and the initial MVS point cloud have different scales;

[0037] A preprocessed MVS point cloud generation module, configured to filter the initial MVS point cloud to obtain a preprocessed MVS point cloud;

[0038] A first transformed MVS point cloud generation module, configured to use the PCA algorithm to analyze the initial scale estimation and initial pose transformation between the structured light scanning point cloud and the preprocessed MVS point cloud, use the structured light scanning point cloud as the target point cloud, and transform the preprocessed MVS point cloud into the coordinate system of the structured light scanning point cloud based on the initial scale estimation and the initial pose transformation to obtain a first transformed MVS point cloud;

[0039] The second transformation MVS point cloud generation module is used to iteratively analyze the precise scale estimation and precise pose transformation between the structured light scanned point cloud and the first transformation MVS point cloud through the ICP algorithm based on scale estimation, and transform the first transformation MVS point cloud to the coordinate system of the structured light scanned point cloud based on the precise scale estimation and the precise pose transformation to obtain the second transformation MVS point cloud;

[0040] The preset area generation module is used to traverse all MVS points in the second transformation MVS point cloud, and respectively construct corresponding preset areas for each of the MVS points with a single MVS point as the center of the sphere within a preset radius;

[0041] The scanned point quantity generation module is used to calculate the total quantity of all scanned points of the structured light scanned point cloud in each of the preset areas respectively to obtain the scanned point quantity of each of the preset areas;

[0042] The missing area generation module is used to screen out several preset areas with the scanned point quantity less than the quantity threshold from each of the preset areas as missing areas;

[0043] The complete point cloud generation module is used to add the MVS points at the corresponding positions of each of the missing areas in the second transformation MVS point cloud to each of the missing areas of the structured light scanned point cloud to obtain the complete point cloud of the three-dimensional model of the object to be modeled.

[0044] Preferably, the initial pose transformation includes an initial rotation matrix and an initial translation vector. The use of the PCA algorithm to analyze the initial scale estimation and the initial pose transformation between the structured light scanned point cloud and the preprocessed MVS point cloud includes:

[0045] Perform an average calculation on all the scanned points of the structured light scanned point cloud to obtain the first point cloud centroid corresponding to the structured light scanned point cloud, and perform an average calculation on all the MVS points of the preprocessed MVS point cloud to obtain the second point cloud centroid corresponding to the preprocessed MVS point cloud;

[0046] Calculate the first covariance matrix corresponding to the structured light scanned point cloud based on the first point cloud centroid, calculate the second covariance matrix corresponding to the preprocessed MVS point cloud based on the second point cloud centroid, and solve for the first eigenvalue and the first left singular matrix of the first covariance matrix and the second eigenvalue and the second left singular matrix of the second covariance matrix through the method of singular value decomposition;

[0047] Determine the first vector axis according to the maximum value of the first eigenvalue in the eigenvector corresponding to the first left singular matrix, and determine the second vector axis according to the maximum value of the second eigenvalue in the eigenvector corresponding to the second left singular matrix;

[0048] Obtain the first longest side of the oriented bounding box of the structured light scanned point cloud based on the difference between the maximum projection and the minimum projection of each of the scanned points on the first vector axis, and obtain the second longest side of the oriented bounding box of the preprocessed MVS point cloud based on the difference between the maximum projection and the minimum projection of each of the MVS points on the second vector axis;

[0049] Calculate the ratio of the first longest side to the second longest side to obtain the initial scale estimate;

[0050] Use the matrix inversion of the first left singular matrix and the second left singular matrix to perform matrix multiplication to obtain the initial rotation matrix,

[0051] Use the centroid of the first point cloud to subtract the product of the initial scale estimate, the initial rotation matrix, and the centroid of the second point cloud to obtain the initial translation vector.

[0052] Preferably, the step of using the centroid of the first point cloud to subtract the product of the initial scale estimate, the initial rotation matrix, and the centroid of the second point cloud to obtain the initial translation vector includes:

[0053] Taking the structured light scanned point cloud as the target point cloud, transform the preprocessed MVS point cloud to the coordinate system of the structured light scanned point cloud based on the initial scale estimate, the initial rotation matrix, and the initial translation vector to obtain the initial transformed MVS point cloud;

[0054] Calculate the point cloud average mean square error between the initial transformed MVS point cloud and the structured light scanned point cloud, and determine whether the point cloud average mean square error is greater than the error threshold;

[0055] If the point cloud average mean square error is greater than the error threshold, then reverse the column vectors of the second left singular matrix, and recalculate to obtain a new set of rotation matrices and translation vectors;

[0056] Select a rotation matrix and a translation vector with the minimum loss function from each of the rotation matrices and each of the translation vectors as the initial rotation matrix and the translation vector.

[0057] Preferably, the step of filtering the initial MVS point cloud to obtain the preprocessed MVS point cloud includes:

[0058] Obtain the point depth of each MVS point of the initial MVS point cloud, and remove the MVS points whose point depth exceeds the depth threshold to obtain the primary MVS point cloud;

[0059] Color attributes are assigned to the primary MVS point cloud according to the colors of the pixel points corresponding to each MVS point in the primary MVS point cloud, and background points are removed based on the similarity between the primary MVS point cloud with the color attributes and the background color to obtain a secondary MVS point cloud;

[0060] Outliers in the secondary MVS point cloud are filtered out by a statistical filter to obtain the preprocessed MVS point cloud, where the outliers represent MVS points in the secondary MVS point cloud whose distance from the nearest MVS point is greater than a distance threshold.

[0061] Preferably, the method of using a monocular camera to capture images of multiple viewpoints of the object to be modeled and calculating the initial MVS point cloud of the object to be modeled by the MVS algorithm includes:

[0062] Using the SFM algorithm to perform pose estimation on each of the viewpoint images respectively to obtain the camera estimation parameters and sparse 3D point clouds corresponding to each of the viewpoint images;

[0063] Inputting each of the camera estimation parameters and each of the sparse 3D point clouds into the MVSNet neural network for depth estimation to obtain the initial MVS point cloud.

[0064] The present invention also provides a system for generating a three-dimensional model point cloud of an object to be modeled, which adopts the method for generating a three-dimensional model point cloud of an object to be modeled as described above, or includes the device for generating a three-dimensional model point cloud of an object to be modeled as described above.

[0065] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, it can implement the method as described above.

[0066]

Beneficial effects

[0067] Other beneficial effects of the present invention will be described in the specific embodiments through the introduction of specific technical features and technical solutions. Those skilled in the art should be able to understand the beneficial technical effects brought by the technical features and technical solutions through the introduction of these technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The following will describe the preferred embodiments of the method for generating the point cloud of the three-dimensional model of the object to be modeled according to the present invention with reference to the drawings. In the figures:

[0069] Figure 1 is a schematic flow chart of the method for generating the point cloud of the three-dimensional model of the object to be modeled according to a preferred embodiment of the present invention;

[0070] Figure 2 is a schematic flow chart of the initial scale estimation between the structured light scanning point cloud and the preprocessed MVS point cloud using the PCA algorithm according to a preferred embodiment of the present invention;

[0071] Figure 3 is a schematic scene diagram of the acquisition device for the structured light scanning point cloud according to a preferred embodiment of the present invention;

[0072] Figure 4 is a schematic scene diagram of the acquisition device for the MVS point cloud according to a preferred embodiment of the present invention;

[0073] Figure 5 is a schematic comparison diagram between the structured light scanning point cloud and the original image of the object to be modeled according to a preferred embodiment of the present invention;

[0074] Figure 6 is a schematic change diagram of the acquisition process of the first transformed MVS point cloud according to a preferred embodiment of the present invention;

[0075] Figure 7 is a schematic change diagram of the acquisition process of the second transformed MVS point cloud according to a preferred embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0076] The following describes the present invention based on embodiments, but the present invention is not limited to these embodiments. In the following detailed description of the present invention, some specific details are described in detail. In order to avoid obscuring the essence of the present invention, well-known methods, processes, procedures, and components are not described in detail.

[0077] In addition, those of ordinary skill in the art should understand that the drawings provided herein are for illustrative purposes only, and the drawings are not necessarily drawn to scale.

[0078] Unless the context clearly requires otherwise, the terms "including", "comprising" and similar words throughout the specification and claims shall be construed in an inclusive sense rather than an exclusive or exhaustive sense; that is, the meaning of "including but not limited to".

[0079] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "a plurality" is two or more.

[0080] Referring to Figure 1 , an embodiment of the present application provides a method for generating a point cloud of a three-dimensional model of an object to be modeled, including:

[0081] S1: Calculate the structured light scanning point cloud of the object to be modeled by the structured light scanning method, capture images of multiple viewpoints of the object to be modeled using a monocular camera, and calculate the initial MVS point cloud of the object to be modeled through the MVS algorithm. Among them, the structured light scanning point cloud and the initial MVS point cloud have different scales;

[0082] S2: Filter the initial MVS point cloud to obtain a preprocessed MVS point cloud;

[0083] S3: Use the PCA algorithm to analyze the initial scale estimation and initial pose transformation between the structured light scanning point cloud and the preprocessed MVS point cloud. Taking the structured light scanning point cloud as the target point cloud, transform the preprocessed MVS point cloud to the coordinate system of the structured light scanning point cloud based on the initial scale estimation and the initial pose transformation to obtain a first transformed MVS point cloud;

[0084] S4: Iteratively analyze the accurate scale estimation and accurate pose transformation between the structured light scanning point cloud and the first transformed MVS point cloud through the ICP algorithm based on scale estimation. Based on the accurate scale estimation and the accurate pose transformation, transform the first transformed MVS point cloud to the coordinate system of the structured light scanning point cloud to obtain a second transformed MVS point cloud;

[0085] S5: Traverse all the MVS points in the second transformed MVS point cloud, and respectively take a single MVS point as the center of a sphere to construct a preset area corresponding to each MVS point within a preset radius;

[0086] S6: Calculate the total number of all scanning points of the structured light scanning point cloud in each preset area respectively to obtain the scanning point number of each preset area;

[0087] S7: Screen out several preset areas with the scanning point number less than the number threshold from each preset area as missing areas;

[0088] S8: Add the MVS points at the corresponding positions of each missing area in the second transformed MVS point cloud to each missing area of the structured light scanned point cloud to obtain the complete point cloud of the three-dimensional model of the object to be modeled.

[0089] Refer to Figure 3 As shown, in this embodiment, the modeling system acquires the structured light scanned point cloud of the object to be modeled through a structured light device. Among them, the structured light device preferably includes 2 industrial cameras, 1 color texture camera, 1 structured light projection device, and 1 electric control rotary table. When acquiring the structured light scanned point cloud, after the object to be modeled is placed on the electric control rotary table, trigger the structured light projection device to project phase shift and complementary Gray code stripes on the object to be modeled, and control the color texture camera to synchronously acquire the encoded images; then calculate the absolute phase map through the N-step phase shift algorithm and complementary Gray code, so as to complete the encoding of the surface of the object to be modeled. Further, complete the corresponding point matching of the left and right industrial cameras and perform triangulation through the phase stereo matching algorithm, so as to obtain the structured light scanned point cloud of one surface of the object to be modeled. After obtaining the structured light scanned point cloud of one surface of the object to be modeled, the modeling system controls the electric control rotary table to rotate and acquires the structured light scanned point clouds of multiple surfaces of the object to be modeled according to the above method. Finally, the modeling system completes the global matching of all point cloud data (that is, the structured light scanned point clouds of each surface of the object to be modeled) through the non-linear ICP (Iterative Closest Points) algorithm, so as to obtain the final structured light scanned point cloud. As Figure 5 shown, Figure 5 Figure (a) in Figure 5 characterizes the structured light scanned point cloud of the object to be modeled scanned by the structured light device, while figure (b) is the original object image directly taken of the object to be modeled; it can be seen from

[0090] Refer to Figure 4 , when acquiring the initial MVS (Multiple View Stereo) point cloud of the object to be modeled, the modeling system uses a monocular camera (such as Figure 4The mobile phone camera in it) takes multiple viewpoint images by shooting around the object to be modeled, and then uses the SFM (Structure from Motion) algorithm to estimate the pose of each viewpoint image respectively, so as to obtain the camera estimation parameters corresponding to each viewpoint image and the sparse 3D point cloud of the scene. Then, the output of the SFM algorithm is used as the input of MVS, and the camera estimation parameters and the sparse 3D point cloud of the scene are input into the MVSNet neural network for depth estimation, so as to obtain the initial MVS point cloud of the object to be modeled.

[0091] Since when using a structured light device to scan an object, a modulation template is constructed through phase-shifted fringes, the object to be modeled and the scanning background can be well distinguished. And the acquisition of the MVS point cloud usually has more noise, so it is necessary to filter the initial MVS point cloud to remove the noise in the initial MVS point cloud. Specifically, the modeling system first removes the background noise from the initial MVS point cloud by restricting the depth, so as to obtain the primary MVS point cloud; then removes the background points in the primary MVS point cloud according to the similarity between the primary MVS point cloud and the background color to obtain the secondary MVS point cloud; finally, filters out the outliers from the secondary MVS point cloud through a statistical filter to obtain the preprocessed MVS point cloud after denoising.

[0092] Before performing the corresponding registration process using the PCA algorithm, the morphology between the structured light scanned point cloud and the preprocessed MVS point cloud is as Figure 6 shown in Figure A of Figure 6 where the black part of the image in Figure A represents the structured light scanned point cloud, and the gray part represents the preprocessed MVS point cloud (the corresponding relationship between the type of color and the type of point cloud in the following text is the same as above). The modeling system uses the PCA (Principal Components Analysis) algorithm to analyze and obtain the initial scale estimation and initial pose transformation between the structured light scanned point cloud and the preprocessed MVS point cloud, and uses the structured light scanned point cloud as the target point cloud. Based on this initial scale estimation and initial pose transformation, the preprocessed MVS point cloud is roughly registered, and the preprocessed MVS point cloud is transformed into the coordinate system of the structured light scanned point cloud to obtain the first transformed MVS point cloud, where Figure 6 Figure B in Figure 6Figure C in [reference] shows the state after the rough registration of the structured light scanned point cloud and the first transformed MVS point cloud by the PCA algorithm. The analysis process of the initial scale estimation and the initial pose transformation between the structured light scanned point cloud and the preprocessed MVS point cloud is as follows: First, calculate the centroids of the structured light scanned point cloud and the preprocessed MVS point cloud respectively, obtaining the first point cloud centroid corresponding to the structured light scanned point cloud and the second point cloud centroid corresponding to the preprocessed MVS point cloud. Then, calculate the first covariance matrix corresponding to the structured light scanned point cloud based on the first point cloud centroid, calculate the second covariance matrix corresponding to the preprocessed MVS point cloud based on the second point cloud centroid, and solve for the first eigenvalue and the first left singular matrix of the first covariance matrix, and the second eigenvalue and the second left singular matrix of the second covariance matrix through the method of singular value decomposition. The modeling system determines the first vector axis according to the maximum value of the first eigenvalue in the eigenvector corresponding to the first left singular matrix, and determines the second vector axis according to the maximum value of the second eigenvalue in the eigenvector corresponding to the second left singular matrix. Then, obtain the first longest side of the oriented bounding box of the structured light scanned point cloud according to the difference between the maximum projection and the minimum projection of each scanned point on the first vector axis, and obtain the second longest side of the oriented bounding box of the preprocessed MVS point cloud according to the difference between the maximum projection and the minimum projection of each MVS point on the second vector axis. The modeling system calculates the ratio of the first longest side to the second longest side to obtain the initial scale estimation. Then, use the matrix inversion of the first left singular matrix and multiply it by the second left singular matrix to obtain the initial rotation matrix. Finally, subtract the product of the initial scale estimation, the initial rotation matrix, and the second point cloud centroid from the first point cloud centroid to obtain the initial translation vector. The modeling system combines the initial rotation geometry and the initial translation vector to obtain the initial pose transformation.

[0093] As Figure 7 shown in Figure C in [reference], there is still a certain difference in the scales of the roughly registered structured light scanned point cloud and the first transformed MVS point cloud. Therefore, it is necessary to continue the fine registration of the roughly registered structured light scanned point cloud and the first transformed MVS point cloud. The modeling system iteratively analyzes the precise scale estimation and the precise pose transformation between the structured light scanned point cloud and the first transformed MVS point cloud through the ICP algorithm based on scale estimation, and then transforms the first transformed MVS point cloud to the coordinate system of the structured light scanned point cloud based on the precise scale estimation and the precise pose transformation, obtaining the second transformed MVS point cloud in gray shown in Figure D in Figure 7 [reference]. Specifically, the modeling system uses the ICP algorithm based on scale estimation to perform fine registration on the structured light scanned point cloud and the first transformed MVS point cloud. The corresponding calculation formula is: where, R * represents the rotation matrix of the current iteration in the fine registration process, t * represents the translation vector of the current iteration in the fine registration process, s *The scale factor for the current time is denoted as, the rotation matrix for the previous time is denoted as R, the translation vector for the previous time is denoted as t, and the scale factor for the previous time is denoted as s. Denotes the centroid of the first transformed MVS point cloud. Denotes the centroid of the structured light scanned point cloud, a i Denotes a certain MVS point in the first transformed MVS point cloud and a certain scanned point in the structured light scanned point cloud. C denotes the covariance matrix. The specific process of this algorithm is as follows: First, find the nearest point in the structured light scanned point cloud for each point in the preprocessed MVS point cloud. If the distance between two points (i.e., a certain point in the preprocessed MVS point cloud and the corresponding nearest point in the structured light scanned point cloud) is greater than the preset distance threshold, then these two points are not regarded as a matching pair; if the distance between these two points is less than the preset distance threshold, then these two points are added as a matching pair to the matching point set. The modeling system calculates the rotation matrix for the current time according to the first calculation formula, and the first calculation formula is: R * = UV T ; where, the centroid of the preprocessed MVS point cloud and the centroid of the structured light scanned point cloud are calculated according to the second calculation formula and the third calculation formula respectively, and then the centroid of the first transformed MVS point cloud and the centroid of the structured light scanned point cloud are substituted into the fourth calculation formula to calculate the value of UV T to obtain the rotation matrix for the current time; specifically, the second calculation formula, the third calculation formula, and the fourth calculation formula are respectively: And, the modeling system calculates according to the fifth calculation formula to solve for the scale factor s for the current time * , where the fifth calculation formula is specifically: Where, Furthermore, after calculating the scale factor and rotation matrix for the current time, the modeling system substitutes each known parameter into the sixth calculation formula to solve for the translation vector t for the current time * , and the sixth calculation formula is specifically: The modeling system performs a similarity transformation on the first transformed MVS point cloud according to the scale factor, rotation matrix, and translation vector obtained for the current time, so as to achieve fine registration of the roughly registered structured light scanned point cloud and the first transformed MVS point cloud, transform the first transformed MVS point cloud to the coordinate system of the structured light scanned point cloud, and obtain the second transformed MVS point cloud. Preferably, the fine registration process of the structured light scanned point cloud and the first transformed MVS point cloud is an iterative loop process, and the steps of a single fine registration are as described above; the modeling system iteratively loops through the above steps of fine registration until the ICP algorithm converges or reaches the maximum number of iterations to complete the entire fine registration process.

[0094] To ensure the accuracy of the three-dimensional model obtained by final modeling, the modeling system uses the structured light scanned point cloud as the fusion basis for the two types of point clouds. First, all missing regions are identified according to the preset number of scanned points in the corresponding regions of the structured light scanned point cloud for each MVS point in the second-transformed MVS point cloud. Then, the MVS points in each missing region of the second-transformed MVS point cloud are added to the corresponding missing regions of the structured light scanned point cloud, realizing the mutual fusion of the structured light scanned point cloud and the preprocessed MVS point cloud, so as to obtain the complete point cloud of the three-dimensional model of the object to be modeled. Specifically, the modeling system traverses all MVS points in the second-transformed MVS point cloud. Each time, a single MVS point is used as the center of a sphere, and a spatial sphere is constructed according to the preset radius. The region where the spatial sphere is located is the preset region. After the fine registration of the second-transformed MVS point cloud and the structured light scanned point cloud is completed, the modeling system can directly count the total number of all scanned points in the preset region of the structured light scanned point cloud, that is, the number of scanned points corresponding to the preset region. If the number of scanned points in the preset region is greater than the number threshold, it means that the structured light can well model the preset region of the object to be modeled, so there is no need to add the MVS points corresponding to the preset region to the structured light scanned point cloud. If the number of scanned points in the preset region is less than the number threshold, it means that the structured light cannot well model the preset region of the object to be modeled, so the preset region is determined as a missing region. During the traversal process, each time the modeling system identifies a missing region, the MVS points corresponding to the missing region (the missing region is spherical, and the MVS points to be added to the structured light scanned point cloud are the centers of the spherical missing region) are added to the corresponding region of the structured light scanned point cloud until all MVS points are traversed, realizing the supplementation of the missing regions in the structured light scanned point cloud using the second-transformed MVS point cloud, completing the fusion of the second-transformed MVS point cloud and the structured light scanned point cloud, and obtaining the complete point cloud of the three-dimensional model of the object to be modeled.

[0095] In this embodiment, the modeling system respectively acquires the structured light scanned point cloud and the MVS point cloud of the object to be modeled, and then uses the PCA algorithm and the ICP algorithm to perform rough registration and fine registration on the structured light scanned point cloud and the MVS point cloud in sequence, so that the MVS point cloud after rough registration and fine registration is accurately transformed into the coordinate system of the structured light scanned point cloud, effectively avoiding the drawback that the MVS point cloud obtained based on a monocular camera cannot reflect the true size of the object to be modeled and affects the fusion with the structured light scanned point cloud. Furthermore, based on the structured light scanned point cloud, the registered MVS point cloud is used to supplement the missing regions in the structured light scanned point cloud, thereby improving the integrity of the supplemented structured light scanned point cloud and avoiding errors or failures in the three-dimensional modeling of the object to be modeled.

[0096] Refer to Figure 2, further, the initial pose transformation includes an initial rotation matrix and an initial translation vector. The step of using the PCA algorithm to analyze the initial scale estimation and the initial pose transformation between the structured light scanned point cloud and the preprocessed MVS point cloud includes:

[0097] S301: Calculate the average of all the scanned points of the structured light scanned point cloud to obtain the first point cloud centroid corresponding to the structured light scanned point cloud, and calculate the average of all the MVS points of the preprocessed MVS point cloud to obtain the second point cloud centroid corresponding to the preprocessed MVS point cloud;

[0098] S302: Calculate the first covariance matrix corresponding to the structured light scanned point cloud based on the first point cloud centroid, calculate the second covariance matrix corresponding to the preprocessed MVS point cloud based on the second point cloud centroid, and solve to obtain the first eigenvalue and the first left singular matrix of the first covariance matrix, and the second eigenvalue and the second left singular matrix of the second covariance matrix through the method of singular value decomposition;

[0099] S303: Determine the first vector axis according to the eigenvector corresponding to the first left singular matrix at the maximum value of the first eigenvalue, and determine the second vector axis according to the eigenvector corresponding to the second left singular matrix at the maximum value of the second eigenvalue;

[0100] S304: Obtain the first longest side of the oriented bounding box of the structured light scanned point cloud according to the difference between the maximum projection and the minimum projection of each scanned point on the first vector axis, and obtain the second longest side of the oriented bounding box of the preprocessed MVS point cloud according to the difference between the maximum projection and the minimum projection of each MVS point on the second vector axis;

[0101] S305: Calculate the ratio of the first longest side to the second longest side to obtain the initial scale estimation;

[0102] S306: Multiply the matrix inverse of the first left singular matrix and the second left singular matrix to obtain the initial rotation matrix,

[0103] S307: Use the first point cloud centroid to subtract the product of the initial scale estimation, the initial rotation matrix and the second point cloud centroid to obtain the initial translation vector.

[0104] In this embodiment, since the initial MVS point cloud is obtained by a monocular camera and monocular 3D reconstruction cannot recover the scene scale, the object size obtained by MVS is generally much smaller than the size of the real object. Therefore, in order to provide a better initial scale estimate and initial pose transformation for subsequent fine registration using the ICP algorithm, it is necessary to first analyze the initial scale estimate and initial pose transformation between the structured light scanning point cloud and the preprocessed MVS point cloud through the PCA algorithm. Specifically, the modeling system first calculates the second centroid of the preprocessed MVS point cloud and the first centroid of the structured light scanning point cloud respectively through the seventh formula and the eighth formula and the first centroid of the structured light scanning point cloud The average value of all scanning points of the structured light scanning point cloud is calculated through the eighth formula to obtain the first centroid of the point cloud, and the average value of all MVS points of the preprocessed MVS point cloud is calculated through the seventh formula to obtain the second centroid of the point cloud; among them, the seventh formula and the eighth formula are respectively N represents the total number of MVS points in the preprocessed MVS point cloud, and M represents the total number of scanning points in the structured light scanning point cloud. Then, the modeling system calculates the first covariance matrix corresponding to the structured light scanning point cloud based on the first centroid of the point cloud, and calculates the second covariance matrix corresponding to the preprocessed MVS point cloud based on the second centroid of the point cloud; and, in order to reduce the computational cost, the modeling system uses the method of singular value decomposition to calculate the eigenvalues D and the first left singular matrix U corresponding to the first covariance matrix and the second covariance matrix respectively b and the second left singular matrix U a . Among them, the calculation formulas of the first covariance matrix and the second covariance matrix are respectively C a represents the second covariance matrix, D a represents the second eigenvalue and is also the singular value of the second covariance matrix, U a , V a respectively represent the singular matrices of the second covariance matrix, C b represents the first covariance matrix, D b represents the first eigenvalue and is also the singular value of the first covariance matrix, U b , V b represent the singular matrices of the first covariance matrix. U a , U b are both matrices composed of eigenvectors, U a , U bThe column vectors are the principal vectors of the corresponding point cloud; the modeling system determines the first vector axis based on the eigenvector corresponding to the maximum value of the first eigenvalue in the first left singular matrix, and determines the second vector axis based on the eigenvector corresponding to the maximum value of the second eigenvalue in the second left singular matrix. Then, the first transformed MVS point cloud and the structured light scanned point cloud are respectively projected onto their corresponding vector axes, so as to obtain the longest sides S a and S b of the OBB (Oriented Bounding Box) corresponding to the first MVS point cloud and the structured light scanned point cloud respectively. Specifically, the projections of each point of the two point clouds are respectively: where X ai represents the projection of the MVS point a of the first transformed MVS point cloud i on the U a (representing the vector corresponding to the maximum eigenvalue) axis, and X bi represents the projection of the scanned point b of the structured light scanned point cloud i on the U b axis. The modeling system obtains the longest side S ai of the OBB corresponding to the first transformed MVS point cloud according to the difference between the maximum X ai and the minimum X a , and obtains the longest side S bi of the OBB corresponding to the structured light scanned point cloud according to the difference between the maximum X bi and the minimum X b . Further, the modeling system obtains the initial scale estimate according to the ratio between the longest side S b and the longest side S a ; that is, the initial scale estimate is: S = S b / S a a . The modeling system respectively solves for the initial rotation matrix and the initial translation vector between the preprocessed MVS point cloud and the structured light scanned point cloud according to the ninth calculation formula and the tenth calculation formula, and combines the initial rotation matrix and the initial translation vector to obtain the initial pose transformation; specifically, the ninth calculation formula and the tenth calculation formula are respectively as follows: where R0 is the initial rotation matrix and t0 is the initial translation vector; the ninth calculation formula represents using the matrix inverse of the first left singular matrix and multiplying it by the second left singular matrix to obtain the initial rotation matrix; the tenth calculation formula represents using the centroid of the first point cloud minus the product of the initial scale estimate, the initial rotation matrix, and the centroid of the second point cloud to obtain the initial translation vector. Thus far, the modeling system has solved for the initial scale estimate and the initial pose transformation required for the rough registration of the first transformed MVS point cloud and the structured light scanned point cloud, and then can transform the first transformed MVS point cloud to the coordinate system of the structured light scanned point cloud according to the initial scale estimate and the initial pose transformation to obtain the second transformed MVS point cloud.

[0105] Further, after the step of obtaining the initial translation vector by subtracting the product of the initial scale estimate, the initial rotation matrix, and the centroid of the second point cloud from the centroid of the first point cloud, the following steps are included:

[0106] S308: Using the structured light scanned point cloud as the target point cloud, transform the preprocessed MVS point cloud to the coordinate system of the structured light scanned point cloud based on the initial scale estimate, the initial rotation matrix, and the initial translation vector to obtain the initial transformed MVS point cloud;

[0107] S309: Calculate the point cloud average mean square error between the initial transformed MVS point cloud and the structured light scanned point cloud, and determine whether the point cloud average mean square error is greater than the error threshold;

[0108] S3010: If the point cloud average mean square error is greater than the error threshold, then take the negative of the column vectors of the second left singular matrix, and recalculate to obtain a new set of rotation matrices and translation vectors according to steps S303 - S307;

[0109] S3011: Select a rotation matrix and a translation vector with the minimum loss function from each of the rotation matrices and each of the translation vectors as the initial rotation matrix and the translation vector.

[0110] In this embodiment, the modeling system uses the structured light scanned point cloud as the target point cloud, and transforms the preprocessed MVS point cloud to the coordinate system of the structured light scanned point cloud based on the initial scale estimate, the initial rotation matrix, and the initial translation vector to obtain the initial transformed MVS point cloud. Then, the nearest scanned point of the MVS point of the initial transformed MVS point cloud in the structured light scanned point cloud is found by searching for the nearest point, and the distances between each group of MVS points and scanned points with the nearest distance are calculated and averaged to obtain the point cloud average mean square error between the initial transformed point cloud and the structured light scanned point cloud. The modeling system retrieves the error threshold and determines whether the point cloud average mean square error is greater than the error threshold. If the point cloud average mean square error is greater than the error threshold, it means that the registration of the preprocessed MVS point cloud based on the initial scale estimate, the initial rotation matrix, and the initial translation vector fails this time, and U needs to be considered separately aFor various cases where the column vectors are negative (specifically 7 cases), select the rotation matrix and translation vector that minimize the loss function. Specifically, the modeling system takes the negative of the column vectors of the second left singular matrix, and then recalculates the respective groups of rotation matrices and translation vectors corresponding to the forward and reverse directions according to steps S303 - S307. The modeling system calls the loss function to calculate each group of rotation matrices and translation vectors, and screens out a group of rotation matrices and translation vectors with the minimum loss function as the initial rotation matrix and initial translation vector required for the rough registration of the preprocessed MVS point cloud and the structured light scanned point cloud; among them, the loss function is specifically:

[0111] Further, the step of filtering the initial MVS point cloud to obtain the preprocessed MVS point cloud includes:

[0112] S201: Obtain the point depths of each MVS point in the initial MVS point cloud, and remove the MVS points whose point depths exceed the depth threshold to obtain a primary MVS point cloud;

[0113] S202: Assign color attributes to the primary MVS point cloud according to the colors of the pixel points corresponding to each MVS point in the primary MVS point cloud, and remove background points according to the similarity between the primary MVS point cloud with the color attributes and the background color to obtain a secondary MVS point cloud;

[0114] S203: Filter out the outlier points in the secondary MVS point cloud through a statistical filter to obtain the preprocessed MVS point cloud, where the outlier points represent the MVS points in the secondary MVS point cloud whose distances from the nearest MVS point are greater than the distance threshold.

[0115] In this embodiment, when the object to be modeled is photographed by a monocular camera and the object to be modeled is within a certain depth range, a depth range can be preset (the specific value of this depth range is set according to the actual scenario and will not be specifically limited here). The modeling system removes background noise from the initial MVS point cloud according to this depth range (that is, the MVS points outside this depth range are determined as background noise and directly removed), thereby obtaining a primary MVS point cloud. Moreover, since a good shooting background is pre-constructed when photographing the image of the object to be modeled, there are obvious regions between the color of the shooting background and the color of the object to be modeled. Therefore, the modeling system assigns color data to the primary MVS point cloud by traversing the colors of the pixel points corresponding to each MVS point in the primary MVS point cloud (that is, indicating the colors corresponding to each MVS point), and compares the color of the MVS point with the shooting background color, thereby removing the MVS points in the primary MVS point cloud whose color similarity to the shooting background color is greater than the threshold, and realizing the removal of a large number of background points in the primary MVS point cloud to obtain a secondary MVS point cloud. Further, the initial MVS point cloud in this embodiment is obtained by means of a neural network. Therefore, the initial MVS point cloud also includes some sparse outlier points. These outlier points represent the MVS points in the secondary MVS point cloud whose distance from the nearest MVS point is greater than the distance threshold. For example, for an MVS point, if there is no MVS point within 10 cm around it, then this MVS point has no practical significance and can be recognized as an outlier point. The modeling system filters out the outlier points in the secondary MVS point cloud through a statistical filter, thereby obtaining a preprocessed MVS point cloud with completely filtered noise.

[0116] Further, the step of photographing images of multiple viewpoints of the object to be modeled by using the monocular camera and calculating the initial MVS point cloud of the object to be modeled through the MVS algorithm includes:

[0117] S101: Use the SFM algorithm to perform pose estimation on each of the viewpoint images respectively, and obtain the camera estimation parameters and the sparse 3D point cloud corresponding to each of the viewpoint images respectively;

[0118] S102: Input each of the camera estimation parameters and each of the sparse 3D point clouds into the MVSNet neural network for depth estimation to obtain the initial MVS point cloud.

[0119] In this embodiment, the modeling system takes pictures of the object to be modeled by a monocular camera (such as a mobile phone camera or a professional single-lens reflex camera) surrounding the object to be modeled, and obtains multiple viewpoint images of the object to be modeled. Then, the incremental SFM (Structure From Motion) algorithm provided by the open-source system COLMAP is used to estimate the pose of each viewpoint image respectively, so as to obtain the camera estimation parameters corresponding to each viewpoint image and the sparse 3D point cloud of the scene (i.e., the scene geometric information). The modeling system uses the output of the SFM algorithm (i.e., the camera estimation parameters corresponding to each viewpoint image and the sparse 3D point cloud of the scene) as the input of MVS. In this embodiment, the MVSNet neural network is selected to perform depth estimation on the output of the SFM algorithm, so as to obtain the initial MVS point cloud of the object to be modeled. Preferably, in this embodiment, the Cascade MVSNet neural network (the Point MVSNet neural network can also be used) is selected to perform depth estimation, so as to improve the depth estimation accuracy of the weak texture area. The Cascade MVSNet neural network is an improved version of the MVSNet neural network. It adopts a cascaded structure to reduce the video memory consumption of the MVSNet neural network. By taking a reference image and multiple source images as inputs, it can obtain a depth map with higher resolution and accuracy corresponding to the reference image, and realize the acquisition of the MVS point cloud data (i.e., the initial MVS point cloud) on the surface of the object to be modeled through depth map fusion. Therefore, compared with the traditional PMVS algorithm, in this embodiment, dense reconstruction is performed through a neural network, and the depth estimation efficiency can be significantly improved.

[0120] The present invention also provides a device for generating a three-dimensional model point cloud of an object to be modeled, including:

[0121] An initial MVS point cloud generation module, configured to calculate the structured light scanning point cloud of the object to be modeled by a structured light scanning method, take pictures of multiple viewpoints of the object to be modeled by a monocular camera, and calculate the initial MVS point cloud of the object to be modeled through an MVS algorithm, wherein the structured light scanning point cloud and the initial MVS point cloud have different scales;

[0122] A preprocessing MVS point cloud generation module, configured to filter the initial MVS point cloud to obtain a preprocessing MVS point cloud;

[0123] A first transformed MVS point cloud generation module, configured to use the PCA algorithm to analyze the initial scale estimation and the initial pose transformation between the structured light scanning point cloud and the preprocessing MVS point cloud, take the structured light scanning point cloud as the target point cloud, and transform the preprocessing MVS point cloud to the coordinate system of the structured light scanning point cloud based on the initial scale estimation and the initial pose transformation to obtain a first transformed MVS point cloud;

[0124] The second transformation MVS point cloud generation module is used to iteratively analyze the accurate scale estimation and accurate pose transformation between the structured light scanned point cloud and the first transformation MVS point cloud through the ICP algorithm based on scale estimation, and transform the first transformation MVS point cloud into the coordinate system of the structured light scanned point cloud based on the accurate scale estimation and the accurate pose transformation to obtain the second transformation MVS point cloud;

[0125] The preset area generation module is used to traverse all MVS points in the second transformation MVS point cloud, and respectively construct preset areas corresponding to each MVS point with a single MVS point as the center of the sphere within a preset radius;

[0126] The scanned point quantity generation module is used to calculate the total quantity of all scanned points of the structured light scanned point cloud in each preset area respectively to obtain the scanned point quantity of each preset area;

[0127] The missing area generation module is used to screen out several preset areas with the scanned point quantity less than the quantity threshold from each preset area as the missing areas;

[0128] The complete point cloud generation module is used to add the MVS points at the corresponding positions of each missing area in the second transformation MVS point cloud into each missing area of the structured light scanned point cloud to obtain the complete point cloud of the three-dimensional model of the object to be modeled.

[0129] In addition, the present invention also provides a computer-readable storage medium for generating the point cloud of the three-dimensional model of the object to be modeled, such as a chip, an optical disc, etc. An execution program is stored on the computer-readable storage medium, and when the execution program is executed, the method described in any one of the above is implemented.

[0130] It should be noted that the computer-readable storage medium described in the embodiments of the present disclosure is not limited to the above-mentioned given embodiments. For example, it can also be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or combined with an instruction execution system, device, or component.

[0131] Those skilled in the art can understand that, on the premise of no conflict, the above-mentioned preferred solutions can be freely combined and superimposed. Among them, the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and this module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions. The numbering of the steps in this article is only for convenience of description and reference, and is not used to limit the order before and after. The specific execution order is determined by the technology itself, and those skilled in the art can determine various permitted and reasonable orders according to the technology itself.

[0132] It should be noted that the use of step numbers (letter or number numbers) to refer to certain specific method steps in the present invention is only for the purpose of description convenience and brevity, and by no means uses letters or numbers to limit the order of these method steps. Those skilled in the art can understand that the order of the relevant method steps should be determined by the technology itself and should not be unduly restricted due to the existence of step numbers. Those skilled in the art can determine various permitted and reasonable step orders according to the technology itself.

[0133] Those skilled in the art can understand that, on the premise of no conflict, the above-mentioned preferred solutions can be freely combined and superimposed.

[0134] It should be understood that the above-mentioned embodiments are only exemplary and not restrictive. Without departing from the basic principles of the present invention, various obvious or equivalent modifications or substitutions made by those skilled in the art to the above details will be included within the scope of the claims of the present invention.

Claims

1. A method for generating a point cloud of a three-dimensional model of an object to be modeled, characterized in that, Including the following steps: Calculating the structured light scanning point cloud of the object to be modeled by the structured light scanning method, taking images of multiple viewpoints of the object to be modeled by a monocular camera, and calculating the initial MVS point cloud of the object to be modeled by the MVS algorithm, wherein the structured light scanning point cloud and the initial MVS point cloud have different scales; Filtering the initial MVS point cloud to obtain a preprocessed MVS point cloud; Using the PCA algorithm to analyze the initial scale estimation and initial pose transformation between the structured light scanning point cloud and the preprocessed MVS point cloud, taking the structured light scanning point cloud as the target point cloud, and transforming the preprocessed MVS point cloud into the coordinate system of the structured light scanning point cloud based on the initial scale estimation and the initial pose transformation to obtain a first transformed MVS point cloud; Iteratively analyzing the accurate scale estimation and accurate pose transformation between the structured light scanning point cloud and the first transformed MVS point cloud by the ICP algorithm based on scale estimation, and transforming the first transformed MVS point cloud into the coordinate system of the structured light scanning point cloud based on the accurate scale estimation and the accurate pose transformation to obtain a second transformed MVS point cloud; Traversing all MVS points in the second transformed MVS point cloud, respectively taking a single MVS point as the center of a sphere, and constructing a corresponding preset area within a preset radius for each MVS point; Respectively calculating the total number of all scanning points of the structured light scanning point cloud in each preset area to obtain the scanning point number of each preset area; Selecting several preset areas with the scanning point number less than the number threshold from each preset area as missing areas; Adding the MVS points at the corresponding positions of each missing area in the second transformed MVS point cloud into each missing area of the structured light scanning point cloud to obtain the complete point cloud of the three-dimensional model of the object to be modeled.

2. The method for generating the point cloud of the three-dimensional model of the object to be modeled according to claim 1, wherein, The initial pose transformation includes an initial rotation matrix and an initial translation vector. The step of using the PCA algorithm to analyze the initial scale estimation and initial pose transformation between the structured light scanning point cloud and the preprocessed MVS point cloud includes: S301: Calculating the average of all scanning points of the structured light scanning point cloud to obtain the first point cloud centroid corresponding to the structured light scanning point cloud, and calculating the average of all MVS points of the preprocessed MVS point cloud to obtain the second point cloud centroid corresponding to the preprocessed MVS point cloud; S302: Calculating the first covariance matrix corresponding to the structured light scanning point cloud based on the first point cloud centroid, calculating the second covariance matrix corresponding to the preprocessed MVS point cloud based on the second point cloud centroid, and solving the first eigenvalue and the first left singular matrix of the first covariance matrix, and the second eigenvalue and the second left singular matrix of the second covariance matrix by the method of singular value decomposition; S303: Determining a first vector axis according to the maximum value of the first eigenvalue in the eigenvector corresponding to the first left singular matrix, and determining a second vector axis according to the maximum value of the second eigenvalue in the eigenvector corresponding to the second left singular matrix; S304: Obtain the first longest side of the oriented bounding box of the structured light scanned point cloud according to the difference between the maximum projection and the minimum projection of each of the scanned points on the first vector axis, and obtain the second longest side of the oriented bounding box of the preprocessed MVS point cloud according to the difference between the maximum projection and the minimum projection of each of the MVS points on the second vector axis; S305: Calculate the ratio of the first longest side to the second longest side to obtain the initial scale estimate; S306: Use the matrix inversion of the first left singular matrix and the second left singular matrix to perform matrix multiplication to obtain the initial rotation matrix; S307: Use the centroid of the first point cloud minus the product of the initial scale estimate, the initial rotation matrix, and the centroid of the second point cloud to obtain the initial translation vector.

3. The method for generating a point cloud of a three-dimensional model of an object to be modeled according to claim 2, wherein After the step of using the centroid of the first point cloud minus the product of the initial scale estimate, the initial rotation matrix, and the centroid of the second point cloud to obtain the initial translation vector, it includes: Taking the structured light scanned point cloud as the target point cloud, transform the preprocessed MVS point cloud to the coordinate system of the structured light scanned point cloud based on the initial scale estimate, the initial rotation matrix, and the initial translation vector to obtain the initial transformed MVS point cloud; Calculate the point cloud average mean square error between the initial transformed MVS point cloud and the structured light scanned point cloud, and determine whether the point cloud average mean square error is greater than the error threshold; If the point cloud average mean square error is greater than the error threshold, then reverse the column vectors of the second left singular matrix, and recalculate according to steps S303 - S307 to obtain a new set of rotation matrices and translation vectors; Select a rotation matrix and a translation vector with the minimum loss function from each of the rotation matrices and each of the translation vectors as the initial rotation matrix and the translation vector.

4. The method for generating the point cloud of the three-dimensional model of the object to be modeled according to claim 1, wherein The step of filtering the initial MVS point cloud to obtain the preprocessed MVS point cloud includes: Obtain the point depth of each MVS point of the initial MVS point cloud, and remove the MVS points whose point depth exceeds the depth threshold to obtain the primary MVS point cloud; Assign color attributes to the primary MVS point cloud according to the colors of the pixel points corresponding to each MVS point of the primary MVS point cloud, and remove background points according to the similarity between the primary MVS point cloud with the color attribute and the background color to obtain the secondary MVS point cloud; Filter out the outlier points of the secondary MVS point cloud through a statistical filter to obtain the preprocessed MVS point cloud, where the outlier points represent the MVS points in the secondary MVS point cloud whose distance from the nearest MVS point is greater than the distance threshold.

5. The method for generating the point cloud of the three-dimensional model of the object to be modeled according to claim 1, wherein The step of using a monocular camera to capture images of multiple viewpoints of the object to be modeled and calculating the initial MVS point cloud of the object to be modeled through the MVS algorithm includes: Use the SFM algorithm to perform pose estimation on each of the viewpoint images respectively to obtain the camera estimation parameters and sparse 3D point clouds corresponding to each of the viewpoint images; Input each of the camera estimation parameters and each of the sparse 3D point clouds into the MVSNet neural network for depth estimation to obtain the initial MVS point cloud.

6. A device for generating a point cloud of a three-dimensional model of an object to be modeled, characterized in that, It includes: An initial MVS point cloud generation module, which is used to calculate the structured light scanning point cloud of the object to be modeled by the structured light scanning method, capture images of multiple viewpoints of the object to be modeled using a monocular camera, and calculate the initial MVS point cloud of the object to be modeled through the MVS algorithm. Among them, the structured light scanning point cloud and the initial MVS point cloud have different scales; A preprocessing MVS point cloud generation module, which is used to filter the initial MVS point cloud to obtain a preprocessed MVS point cloud; A first transformed MVS point cloud generation module, which is used to use the PCA algorithm to analyze the initial scale estimation and initial pose transformation between the structured light scanning point cloud and the preprocessed MVS point cloud. Taking the structured light scanning point cloud as the target point cloud, based on the initial scale estimation and the initial pose transformation, transform the preprocessed MVS point cloud into the coordinate system of the structured light scanning point cloud to obtain a first transformed MVS point cloud; A second transformed MVS point cloud generation module, which is used to iteratively analyze the precise scale estimation and precise pose transformation between the structured light scanning point cloud and the first transformed MVS point cloud through the ICP algorithm based on scale estimation, and based on the precise scale estimation and the precise pose transformation, transform the first transformed MVS point cloud into the coordinate system of the structured light scanning point cloud to obtain a second transformed MVS point cloud; A preset region generation module, which is used to traverse all the MVS points in the second transformed MVS point cloud, and respectively use a single MVS point as the center of a sphere to construct corresponding preset regions within a preset radius for each of the MVS points; A scanned point number generation module, which is used to calculate the total number of all scanned points of the structured light scanning point cloud in each of the preset regions respectively to obtain the scanned point number of each of the preset regions; A missing region generation module, which is used to screen out several preset regions with the scanned point number less than the number threshold from each of the preset regions as missing regions; A complete point cloud generation module, which is used to add the MVS points at the corresponding positions of each of the missing regions in the second transformed MVS point cloud into each of the missing regions of the structured light scanning point cloud to obtain the complete point cloud of the three-dimensional model of the object to be modeled.

7. The generating device for the point cloud of the three-dimensional model of the object to be modeled according to claim 6, characterized in that, The initial pose transformation includes an initial rotation matrix and an initial translation vector. The use of the PCA algorithm to analyze the initial scale estimation and initial pose transformation between the structured light scanning point cloud and the preprocessed MVS point cloud includes: Perform an averaging calculation on all the scanned points of the structured light scanning point cloud to obtain the first point cloud centroid corresponding to the structured light scanning point cloud, and perform an averaging calculation on all the MVS points of the preprocessed MVS point cloud to obtain the second point cloud centroid corresponding to the preprocessed MVS point cloud; Calculate the first covariance matrix corresponding to the structured light scanned point cloud based on the centroid of the first point cloud, calculate the second covariance matrix corresponding to the preprocessed MVS point cloud based on the centroid of the second point cloud, and solve through singular value decomposition to obtain the first eigenvalue and the first left singular matrix of the first covariance matrix, and the second eigenvalue and the second left singular matrix of the second covariance matrix; Determine the first vector axis according to the maximum value of the first eigenvalue in the eigenvectors corresponding to the first left singular matrix, and determine the second vector axis according to the maximum value of the second eigenvalue in the eigenvectors corresponding to the second left singular matrix; Obtain the first longest side of the oriented bounding box of the structured light scanned point cloud according to the difference between the maximum projection and the minimum projection of each scanned point on the first vector axis, and obtain the second longest side of the oriented bounding box of the preprocessed MVS point cloud according to the difference between the maximum projection and the minimum projection of each MVS point on the second vector axis; Calculate the ratio of the first longest side to the second longest side to obtain the initial scale estimate; Use the matrix inverse of the first left singular matrix and the second left singular matrix to perform matrix multiplication to obtain the initial rotation matrix; Use the centroid of the first point cloud minus the product of the initial scale estimate, the initial rotation matrix, and the centroid of the second point cloud to obtain the initial translation vector.

8. The generating device for the point cloud of the three-dimensional model of the object to be modeled according to claim 7, characterized in that, The step of using the centroid of the first point cloud minus the product of the initial scale estimate, the initial rotation matrix, and the centroid of the second point cloud to obtain the initial translation vector includes: Taking the structured light scanned point cloud as the target point cloud, transform the preprocessed MVS point cloud to the coordinate system of the structured light scanned point cloud based on the initial scale estimate, the initial rotation matrix, and the initial translation vector to obtain the initial transformed MVS point cloud; Calculate the point cloud average mean square error between the initial transformed MVS point cloud and the structured light scanned point cloud, and determine whether the point cloud average mean square error is greater than the error threshold; If the point cloud average mean square error is greater than the error threshold, then reverse the column vectors of the second left singular matrix, and recalculate to obtain several new groups of rotation matrices and translation vectors; Select the rotation matrix and translation vector with the minimum loss function from each rotation matrix and each translation vector as the initial rotation matrix and the translation vector.

9. The generating device for the point cloud of the three-dimensional model of the object to be modeled according to claim 6, characterized in that The step of filtering the initial MVS point cloud to obtain the preprocessed MVS point cloud includes: Obtain the point depths of each MVS point in the initial MVS point cloud, and remove the MVS points whose point depths exceed the depth threshold to obtain the first MVS point cloud; Assign color attributes to the first MVS point cloud according to the colors of the pixel points corresponding to each MVS point in the first MVS point cloud, and remove background points according to the similarity between the first MVS point cloud with the color attributes and the background color to obtain the second MVS point cloud; Outliers in the secondary MVS point cloud are filtered out by a statistical filter to obtain the preprocessed MVS point cloud, where the outliers represent MVS points in the secondary MVS point cloud whose distance from the nearest MVS point is greater than a distance threshold.

10. The generating device for the point cloud of the three-dimensional model of the object to be modeled according to claim 6, wherein The method of using a monocular camera to capture images of multiple viewpoints of the object to be modeled and calculating an initial MVS point cloud of the object to be modeled by an MVS algorithm includes: Performing pose estimation on each of the viewpoint images using an SFM algorithm to obtain camera estimation parameters and sparse 3D point clouds respectively corresponding to each of the viewpoint images; Inputting each of the camera estimation parameters and each of the sparse 3D point clouds into an MVSNet neural network for depth estimation to obtain the initial MVS point cloud.

11. A system for generating a point cloud of a three-dimensional model of an object to be modeled, characterized in that, Adopting the method for generating a three-dimensional model point cloud of an object to be modeled according to any one of claims 1-5, or including the device for generating a three-dimensional model point cloud of an object to be modeled according to any one of claims 6-10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it can implement the method according to any one of claims 1-5.

Images