A three-dimensional reconstruction method, device and medium

Abstract

The present invention discloses a three-dimensional reconstruction method, device, and medium. The method includes: calculating the positional relationship between the end flange of an articulated robot and a 3D camera using a calibration pointer and a checkerboard calibration plate; placing an object to be scanned at the end flange of the articulated robot, and controlling the position change of the object to be scanned based on the positional relationship; scanning the object to be scanned in each position to obtain a point cloud of the object to be scanned in each position; and splicing the point clouds in each position to obtain a reconstructed three-dimensional model of the object to be scanned. The present invention can ensure that all details of the object to be scanned are within the field of view and depth of field of the 3D camera, and can accurately and comprehensively restore the texture information of the surface of the object to be scanned, with high applicability, high practicality, and high efficiency.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, equipment and medium. Background Art

[0002] In actual factory production processes, items on the production line often need to be manually inspected for appearance defects, which is time-consuming, slow, inefficient, and costly. This is especially true for precision-machined objects, which are difficult to detect with the human eye. Traditional methods also use a single ordinary camera for inspection and recognition, but because a single ordinary camera has harsh environmental requirements and cannot perceive depth and three-dimensional information, many inspection items are limited. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a three-dimensional reconstruction method, device, and medium that can automatically change the position and posture of a 3D camera and an articulated robot to obtain an accurate reconstructed model and three-dimensional information.

[0004] The present invention also provides an electronic device and a computer-readable storage medium for executing the above three-dimensional reconstruction method.

[0005] According to a first aspect of the present invention, a three-dimensional reconstruction method is used in a three-dimensional reconstruction system, the three-dimensional reconstruction system comprising a 3D camera, an articulated robot, a calibration pointer, and a checkerboard calibration plate; the 3D camera is mounted on a gantry frame on a stage, the articulated robot is used to adjust the position of an object to be scanned, the articulated robot is mounted on a stage base, the calibration pointer is mounted on an end flange of the articulated robot, and the checkerboard calibration plate is mounted on the stage base; the three-dimensional reconstruction method comprises:

[0006] Calculating the positional relationship between the end flange of the articulated robot and the 3D camera using the calibration pointer and the checkerboard calibration plate;

[0007] Placing the object to be scanned on the end flange of the articulated robot, and controlling the posture of the object to be scanned to change according to the position relationship;

[0008] Scanning the object to be scanned in each of the postures to obtain a point cloud of the object to be scanned in each of the postures;

[0009] The point clouds under each of the postures are spliced ​​together to obtain a reconstructed three-dimensional model of the object to be scanned.

[0010] The 3D reconstruction method according to the embodiment of the present invention has at least the following beneficial effects:

[0011] This embodiment first calibrates the positional relationship between the end flange of the articulated robot and the 3D camera through a calibration pointer. The accurate conversion relationship can be obtained through the positional relationship. The posture transformation of the object to be scanned is controlled according to the accurate conversion relationship to obtain a point cloud at each posture, thereby obtaining high-precision, complete and comprehensive point cloud data information of the three-dimensional morphology of the surface of the object to be scanned. The point cloud at each posture is spliced ​​to obtain a reconstructed three-dimensional model. Since all details of the object to be scanned can be within the field of view and depth of field of the 3D camera, the reconstructed three-dimensional model can accurately and comprehensively restore the texture information of the surface of the object to be scanned, and has high applicability, high practicality and high efficiency.

[0012] According to some embodiments of the present invention, the calculating the positional relationship between the end flange of the articulated robot and the 3D camera by using the calibration pointer and the checkerboard calibration plate includes:

[0013] Set the calibration pointer coordinate system, the joint robot body coordinate system and the 3D camera coordinate system;

[0014] Acquire an image of the checkerboard calibration plate through a 3D camera, and convert corner points in the image of the checkerboard calibration plate into corner point coordinates in the 3D camera coordinate system;

[0015] Obtaining the calibration pointer coordinates of the end of the calibration pointer in the coordinate system of the joint robot body;

[0016] Calculating a transformation matrix between the 3D camera coordinate system and the articulated robot body coordinate system by using the corner point coordinates in the 3D camera coordinate system and the calibrated pointer coordinates in the articulated robot body coordinate system;

[0017] The conversion matrix is ​​optimized by using the calibrated pointer coordinates and the corner point coordinates in at least three of the 3D camera coordinate systems, and the optimized conversion matrix is ​​used as the positional relationship between the end flange of the articulated robot and the 3D camera.

[0018] According to some embodiments of the present invention, the calculating the transformation matrix between the 3D camera coordinate system and the articulated robot body coordinate system by using the corner point coordinates in the 3D camera coordinate system and the calibrated pointer coordinates in the articulated robot body coordinate system includes:

[0019] P BASE =T TOOL2BASE *T CAM2TOOL *P CAM

[0020] T CAM2BASE =T TOOL2BASE *T CAM2TOOL =P BASE *(P CAM )-1

[0021] Among them, P BASE represents the point coordinates in the joint robot body coordinate system, P CAM Represents the point coordinates in the 3D camera coordinate system, T TOOL2BASE The conversion matrix T represents the conversion of the calibration pointer point coordinates in the 3D camera coordinate system to the point coordinates in the joint robot body coordinate system. CAM2TOOL Represents the transformation matrix that converts the point coordinates in the 3D camera coordinate system to the point coordinates in the articulated robot body coordinate system.

[0022] According to some embodiments of the present invention, optimizing the transformation matrix by using the calibrated pointer coordinates and the coordinates of at least three corner points in the 3D camera coordinate system, and using the optimized transformation matrix as the positional relationship between the end flange of the articulated robot and the 3D camera, includes:

[0023] Get the corner point cloud set P in the 3D camera coordinate system = {p1, p2, p3, ... p n} and the point cloud set Q of the calibration pointer coordinates in the coordinate system of the joint robot body = {q1, q2, q3, ...q n}; wherein, the corresponding relationship between the corner point cloud set P and the point cloud set Q is:

[0024] q i =R*p i +t

[0025] Where R represents a 3x3 rotation matrix and t represents a 3x1 translation matrix;

[0026] The centroid of the corner point cloud set P and the point cloud set Q is solved:

[0027]

[0028]

[0029] Wherein, p represents the centroid solution result corresponding to the point cloud set P, and q represents the centroid solution result corresponding to the point cloud set Q;

[0030] Perform a centroid operation on the centroid corresponding to the point cloud set P and the centroid corresponding to the point cloud set Q:

[0031] p′ i =p i -p

[0032] q′ i =q i -q

[0033] Among them, p′ i Represents the centroid removal result of the centroid solution corresponding to the point cloud set P, q′ i The centroid removal result of the centroid solution corresponding to the point cloud set Q is represented;

[0034] The rotation matrix and translation matrix with the minimum error are calculated as follows:

[0035] Compute the minimum error between the rotation matrix and the translation matrix:

[0036]

[0037] Among them, min R，t E represents the minimum error;

[0038] Calculate the optimal rotation matrix and optimal translation matrix corresponding to the minimum error:

[0039]

[0040] t * =p-Rq

[0041]

[0042] tr(RH)≥tr(RH)=tr(BR*H)

[0043] H=U∑V T

[0044] R * =VU T

[0045] Among them, R * represents the optimal rotation matrix, t * represents the optimal translation matrix, H represents the intermediate algebra, represents p′ i The transpose of , U represents a positive definite matrix, ∑ represents a diagonal matrix, V T represents the transpose of a positive definite matrix;

[0046] Parameters of the transformation matrix are modified according to the optimal rotation matrix and the optimal translation matrix.

[0047] According to some embodiments of the present invention, controlling the object to be scanned to change its posture according to the positional relationship includes:

[0048] Acquire a point cloud of the object to be scanned in the 3D camera coordinate system, and represent a first point position by combining the point cloud of the object to be scanned with a quaternion;

[0049] Converting the first point position into a second point position in the coordinate system of the articulated robot body according to the optimized conversion matrix of the positional relationship;

[0050] Calculating a corresponding rotation and translation matrix for transforming a preset point to be scanned to the second point;

[0051] The position and posture of the object to be scanned is changed according to the rotation and translation matrix.

[0052] According to some embodiments of the present invention, the 3D reconstruction method further includes:

[0053] The redundant cloud points of the reconstructed three-dimensional model are filtered out to obtain the final reconstructed three-dimensional model.

[0054] According to some embodiments of the present invention, filtering out redundant cloud points of the reconstructed three-dimensional model to obtain a final reconstructed three-dimensional model includes:

[0055] Setting a three-dimensional point cloud threshold according to the spatial position of the reconstructed three-dimensional model;

[0056] The point cloud of the reconstructed three-dimensional model is traversed, and point clouds outside the point cloud threshold are removed to obtain the final reconstructed three-dimensional model.

[0057] According to some embodiments of the present invention, the 3D reconstruction method further includes:

[0058] Calculating the deviation between the point cloud of the final reconstructed three-dimensional model and the original model; the calculation method includes:

[0059] Projecting the point cloud of the final reconstructed three-dimensional model onto the triangular facets of the original model;

[0060] The projection of the point cloud on the surface triangle ΔABC corresponding to the triangular patch is calculated by the following formula:

[0061]

[0062]

[0063] in, Represents the vector from the origin of space to the projection point, Represents the vector from the origin of space to the point cloud, is the normal vector of the triangle plane, l is the vector With normal vector The dot product of

[0064] Calculate the deviation distance of the point cloud of the second reconstructed three-dimensional model, and the calculation formula includes:

[0065]

[0066] Where d represents the deviation distance.

[0067] In a second aspect, an embodiment of the present invention provides an electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the three-dimensional reconstruction method as described in the first aspect.

[0068] In a third aspect, an embodiment of the present invention provides a computer storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the three-dimensional reconstruction method as described in the first aspect.

[0069] It should be noted that the beneficial effects of the second and third aspects of the present invention over the prior art are the same as the beneficial effects of the three-dimensional reconstruction method of the first aspect, and will not be described in detail here.

[0070] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0072] Figure 1 is a schematic diagram of a three-dimensional reconstruction method according to an embodiment of the present invention;

[0073] Figure 2 Schematic diagram of calculating positional relationship using a calibration pointer and a checkerboard calibration plate according to an embodiment of the present invention;

[0074] Figure 3 is a schematic diagram of scanning an object to be scanned in each posture using a 3D camera according to an embodiment of the present invention;

[0075] Figure 4 is a flow chart of a three-dimensional reconstruction method according to an embodiment of the present invention;

[0076] Figure 5 This is a flow chart of solving positional relationships according to an embodiment of the present invention;

[0077] Figure 6 This is a flow chart of controlling the change of the posture of an object to be scanned according to a position relationship according to an embodiment of the present invention;

[0078] Figure 7This is a flow chart of filtering out redundant cloud points in a reconstructed three-dimensional model according to an embodiment of the present invention;

[0079] Figure 8 is a flow chart of a filtering method according to an embodiment of the present invention;

[0080] Figure 9 is a flow chart for calculating the deviation between the final reconstructed three-dimensional model and the original digital model according to one embodiment of the present invention;

[0081] Figure 10 is a flow chart of a deviation calculation method according to an embodiment of the present invention;

[0082] Figure 11 FIG. 1 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0084] In the description of the present invention, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0085] In the description of the present invention, it should be understood that descriptions involving orientation, such as the orientation or positional relationship indicated by up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0086] In the description of the present invention, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0087] Traditional methods use a single ordinary camera for detection and recognition. However, since a single ordinary camera has harsh environmental requirements and cannot perceive depth and stereo information, many detection projects are limited.

[0088] Based on this, the three-dimensional reconstruction method provided by the embodiment of the present invention combines an articulated robot with a 3D camera, so that all details of the object to be scanned can be within the field of view and depth of field of the 3D camera, and can accurately and comprehensively restore the texture information of the surface of the object to be scanned, with high applicability, high practicality and high efficiency.

[0089] The technical solutions of the present invention will be described clearly and completely below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of the present invention, not all embodiments.

[0090] Reference Figure 4 FIG. 1 is a flow chart of a three-dimensional reconstruction method provided by an embodiment of the present invention. The three-dimensional reconstruction method is used in a three-dimensional reconstruction system, wherein the three-dimensional reconstruction system includes a 3D camera, an articulated robot, a calibration pointer, and a checkerboard calibration plate. The 3D camera is set on a stage through a gantry, the articulated robot is used to adjust the posture of the object to be scanned, the articulated robot is set on the stage base, the calibration pointer is set on the end flange of the articulated robot, and the checkerboard calibration plate is set on the stage base. The three-dimensional reconstruction method includes but is not limited to the following steps:

[0091] Step S100: Calculate the positional relationship between the end flange of the articulated robot and the 3D camera using the calibration pointer and the checkerboard calibration plate.

[0092] Step S200: Place the object to be scanned on the end flange of the articulated robot, and control the posture of the object to be scanned to change according to the position relationship.

[0093] Step S300: Scan the object to be scanned in each posture to obtain a point cloud of the object to be scanned in each posture.

[0094] Step S400: stitching the point clouds at each position to obtain a reconstructed three-dimensional model of the object to be scanned.

[0095] This embodiment first uses a calibration pointer and a checkerboard calibration plate to calibrate the positional relationship between the end flange of the articulated robot and the 3D camera through step S100. An accurate conversion relationship can be obtained through the positional relationship. Then, step S200 is used to control the posture transformation of the object to be scanned according to the accurate conversion relationship. Then, step S300 is used to scan the object to be scanned in each posture to obtain a point cloud in each posture, thereby obtaining high-precision, complete and comprehensive point cloud data information of the three-dimensional morphology of the surface of the object to be scanned. Finally, step S400 is used to obtain a reconstructed three-dimensional model from the point cloud in each posture. Since all details of the entire range of the object to be scanned can be within the field of view and depth of field of the 3D camera, the reconstructed three-dimensional model can accurately and comprehensively restore the texture information of the surface of the object to be scanned, and has high applicability, high practicality and high efficiency.

[0096] Reference Figure 5 In some embodiments of the present invention, calculating the positional relationship between the end flange of the articulated robot and the 3D camera by calibrating the pointer and the checkerboard calibration plate includes:

[0097] Step S110, setting the calibration pointer coordinate system, the articulated robot body coordinate system and the 3D camera coordinate system;

[0098] Step S120: Acquire an image of the checkerboard calibration plate through a 3D camera, and convert the corner points in the image of the checkerboard calibration plate into the coordinates of the corner points in the 3D camera coordinate system;

[0099] Step S130, obtaining the calibration pointer coordinates of the end of the calibration pointer in the coordinate system of the articulated robot body;

[0100] Step S140, calculating the transformation matrix between the 3D camera coordinate system and the articulated robot body coordinate system by using the corner point coordinates in the 3D camera coordinate system and the calibrated pointer coordinates in the articulated robot body coordinate system;

[0101] Step S150 , optimizing the transformation matrix by calibrating the pointer coordinates and the corner point coordinates in at least three 3D camera coordinate systems, and using the optimized transformation matrix as the positional relationship between the end flange of the articulated robot and the 3D camera.

[0102] It should be noted that, depending on the different ways of fixing the 3D camera, the transformation matrix is ​​solved in two cases: eye-in-hand and eye-outside-hand. Since the 3D camera is installed on the gantry and its relative position to the articulated robot is fixed, the eye-outside-hand method is adopted.

[0103] The transformation matrix between the 3D camera coordinate system and the articulated robot body coordinate system is calculated by calibrating the pointer coordinates and the corner point coordinates. Since the calibration pointer is more accurate than directly using the object calibration and the actual operation is simpler, the checkerboard calibration plate can more accurately calculate the transformation matrix between the 3D camera coordinate system and the articulated robot body coordinate system together with the calibration pointer coordinates; then the transformation matrix is ​​optimized by fixing the corner point coordinates and calibration pointer coordinates in at least three 3D camera coordinate systems, making the transformation more accurate and reducing the error value.

[0104] In some embodiments of the present invention, the transformation matrix between the 3D camera coordinate system and the articulated robot body coordinate system is calculated using the corner point coordinates in the 3D camera coordinate system and the calibrated pointer coordinates in the articulated robot body coordinate system, including:

[0105] P BASE =T TOOL2BASE *TC AM2TOOL *P CAM

[0106] T CAM2BASE =T TOOL2BASE *TC AM2TOOL =P BASE *(P CAM ) -1

[0107] Among them, P BASE represents the point coordinates in the joint robot body coordinate system, P CAM Represents the point coordinates in the 3D camera coordinate system, T TOOL2BASE The conversion matrix T represents the conversion of the calibration pointer point coordinates in the 3D camera coordinate system to the point coordinates in the joint robot body coordinate system. CAM2TOOL Represents the transformation matrix that converts the point coordinates in the 3D camera coordinate system to the point coordinates in the articulated robot body coordinate system.

[0108] The coordinate conversion formula is calculated by the corner point coordinates in the 3D camera coordinate system and the calibrated pointer coordinates in the joint robot body coordinate system. The conversion matrix is ​​obtained by solving the pseudo-inverse matrix, and an accurate conversion matrix can be obtained.

[0109] Reference Figure 6 In some embodiments of the present invention, the transformation matrix is ​​optimized by calibrating the pointer coordinates and the corner point coordinates in at least three 3D camera coordinate systems, and the optimized transformation matrix is ​​used as the positional relationship between the end flange of the articulated robot and the 3D camera, including:

[0110] Step S151: Obtain the corner point cloud set P={p1, p2, p3, ...p n} and the point cloud set Q of the calibrated pointer coordinates in the coordinate system of the joint robot body = {q1, q2, q3, ...q n}; Among them, the corresponding relationship between the corner point cloud set P and the point cloud set Q is:

[0111] q i =R*p i +t

[0112] Where R represents a 3x3 rotation matrix and t represents a 3x1 translation matrix.

[0113] Step S152: Calculate the centroid of the diagonal point cloud set P and the point cloud set Q:

[0114]

[0115]

[0116] Among them, p represents the centroid solution result corresponding to the point cloud set P, and q represents the centroid solution result corresponding to the point cloud set Q.

[0117] Step S153: Perform a centroid removal operation on the centroid corresponding to the point cloud set P and the centroid corresponding to the point cloud set Q:

[0118] p′ i =p i -p

[0119] q′ i =q i -q

[0120] Among them, p′ i Represents the centroid solution of the point cloud set P, q′ i Represents the centroid removal result of the centroid solution corresponding to the point cloud set Q.

[0121] Step S154: Calculate the rotation matrix and translation matrix with the minimum error by the following method:

[0122] Compute the minimum error between the rotation matrix and the translation matrix:

[0123]

[0124] Among them, min R，t E represents the minimum error.

[0125] Step S155: Calculate the optimal rotation matrix and optimal translation matrix corresponding to the minimum error:

[0126]

[0127] t * =p-Rq

[0128]

[0129] tr(RH)≥tr(RH)=tr(BR*H)

[0130] H=U∑V T

[0131] R * =VU T

[0132] Among them, R * represents the optimal rotation matrix, t * represents the optimal translation matrix, H represents the intermediate algebra, represents p′ i The transpose of , U represents a positive definite matrix, ∑ represents a diagonal matrix, V T represents the transpose of a positive definite matrix.

[0133] Step S156: Modify the parameters of the conversion matrix according to the optimal rotation matrix and the optimal translation matrix.

[0134] The essence of solving the transformation matrix is ​​an optimization problem. By minimizing the transformation error, the optimal solution of the transformation matrix can be obtained. Therefore, the expression of the optimal rotation matrix and the optimal translation matrix is ​​first obtained through the calculation formula of error minimization. Then, the optimal rotation matrix and the optimal translation matrix under the optimal solution are calculated through the algorithm. The transformation matrix with smaller error can be obtained, making the transformation matrix more accurate.

[0135] Reference Figure 7 In some embodiments of the present invention, controlling the position change of the object to be scanned according to the position relationship includes:

[0136] Step S201: Acquire a point cloud of the object to be scanned in a 3D camera coordinate system, and combine the point cloud of the object to be scanned with a quaternion to represent a first point position.

[0137] Step S202: convert the first point position into a second point position in the articulated robot body coordinate system according to the optimized conversion matrix of the positional relationship.

[0138] Step S203: Calculate the corresponding rotation and translation matrix for transforming the preset point to be scanned to the second point.

[0139] Step S204: changing the position and posture of the object to be scanned according to the rotation and translation matrix.

[0140] It should be noted that quaternions have small storage space, high computational efficiency, and no universal lock problem. Therefore, point coordinates plus quaternions are used to express the current position of the robot.

[0141] This embodiment calculates the corresponding rotation and translation matrix of the point to be scanned, and each point to be scanned can be transformed to a second point. It can be scanned in all directions without blind spots. The point clouds after scanning different points are in the same posture, and finally the complete three-dimensional morphology of the overall model is obtained.

[0142] Reference Figure 8 In some embodiments of the present invention, the three-dimensional reconstruction method further includes:

[0143] Step S500: filtering out redundant cloud points of the reconstructed 3D model to obtain a final reconstructed 3D model.

[0144] It should be noted that the posture of the second point is known, so the spatial position and direction of the support rod are also known, so it is necessary to filter out cloud points that are not related to the original model in the reconstructed 3D model.

[0145] This embodiment can obtain a more accurate final reconstructed 3D model by filtering out redundant cloud points of the reconstructed 3D model, and the final reconstructed 3D model will not be affected by components of the 3D reconstruction device, thereby improving applicability.

[0146] Reference Figure 9 In some embodiments of the present invention, filtering out redundant cloud points of the reconstructed three-dimensional model to obtain the final reconstructed three-dimensional model includes:

[0147] Step S501: Set a three-dimensional point cloud threshold by reconstructing the spatial position of the three-dimensional model.

[0148] Step S502: traverse the point cloud of the reconstructed three-dimensional model and remove the point cloud outside the point cloud threshold to obtain the final reconstructed three-dimensional model.

[0149] It should be noted that this embodiment uses an outer bounding box to algorithmically filter out the object support rod and irrelevant point clouds around the support rod. The specific method is to filter the point cloud in spatial position, set a threshold of the corresponding dimension, and then traverse all points in the point cloud to remove points that are not within the threshold range under the corresponding dimension.

[0150] This embodiment performs filtering by setting a point cloud threshold of a dimension, which is simple to implement and has a good filtering effect. Therefore, it is highly practical and suitable for filtering out the object support rod and irrelevant point clouds around the support rod.

[0151] In some embodiments of the present invention, the 3D reconstruction method further includes:

[0152] Reference Figure 10 Step S600: Calculate the deviation between the point cloud of the final reconstructed 3D model and the original model. The calculation method includes:

[0153] Project the point cloud of the final reconstructed 3D model onto the triangular facets of the original model;

[0154] The projection of the point cloud on the surface triangle ΔABC corresponding to the triangular patch is calculated using the following formula:

[0155]

[0156]

[0157] in, Represents the vector from the origin of space to the projection point, Represents the vector from the origin of space to the point cloud, is the normal vector of the triangle plane, l is the vector With normal vector The dot product of

[0158] Calculate the deviation distance of the point cloud of the second reconstructed 3D model. The calculation formula includes:

[0159]

[0160] Where d represents the deviation distance.

[0161] It should be noted that the final reconstructed 3D model after filtering only contains the complete shape of the object itself. The original digital model for comparison can be provided by the automation manufacturer. The overall point cloud to be compared is matched with the original digital model using the Iterative Closest Point algorithm (an algorithm that accurately aligns point clouds). Then, the provided original digital model is used for comparison. The degree of difference is mapped into a color scale of the point cloud, and the color of the point cloud is used to show the degree of difference. It can be used to detect bulges and depressions to be compared, locate defect positions, and model size differences. The original digital model used for comparison is mainly composed of triangular facets. The expression method is simple and intuitive, and the traversal speed is fast and efficient. The geometric relationship between the point cloud of the final reconstructed 3D model and the triangular facets in the original model can be used to calculate the deviation between the point cloud of the final reconstructed 3D model and the original model. The point cloud of the final reconstructed 3D model is projected onto the corresponding triangular facet of the original model to obtain the correspondence between the point cloud and the facet, thereby calculating the deviation value of the point cloud relative to the original model.

[0162] This embodiment compares the final reconstructed three-dimensional model with the original digital model to obtain the deviation value between the reconstructed three-dimensional model and the original digital model, which can facilitate the detection of protrusions and depressions to be compared, locate defect positions, and model size differences, thereby reducing the burden of detection work.

[0163] Reference Figures 1 to 3 After the above description, this embodiment further uses a specific example to illustrate that the three-dimensional reconstruction method specifically includes the following steps:

[0164] In the first step, the calibration pointer and the checkerboard calibration plate are used to calibrate the positional relationship between the robot end flange and the 3D camera.

[0165] The calibration of the articulated robot end flange and the 3D camera involves four coordinate systems, namely the articulated robot body coordinate system (BASE), the calibration pointer coordinate system (TOOL), the 3D camera coordinate system (CAM), and the checkerboard calibration board coordinate system (BOARD).

[0166] Before calibrating the positional relationship between the end flange of the articulated robot and the 3D camera, it is necessary to first perform tool calibration of the calibration pointer. The focus of tool calibration is to obtain the position of the tool center point for use in the next step of hand-eye calibration. The tool calibration tool that comes with the robot uses the six-point teaching method for calibration. After calibration is completed, select the calibration pointer on the robot controller. The quaternion displayed on the controller can be converted into a transformation matrix (T TOOL2BASE ), which is the position of the end of the calibration pointer in the coordinate system of the joint robot body.

[0167] Because the 3D camera is mounted on a gantry and its relative position to the articulated robot is fixed, an eye-out-of-hand approach is employed. A fixed 3D camera is used to photograph a calibration plate within the depth of field. The corner points on the two-dimensional image (the checkerboard calibration plate image) captured by the 3D camera are identified. The camera's intrinsic parameters are then used to convert the calibration plate corner points into spatial points in the 3D camera coordinate system. The articulated robot then moves to the corresponding corner points on the calibration plate, recording the xyz coordinates of the corresponding points on the calibration plate under the 3D camera and the end-point pointer on the articulated robot's flange.

[0168] For several sets of corresponding fixed points, the positions of point P in the base coordinate system of the joint robot body and the 3D camera coordinate system are P BASE and P CAM , the coordinate conversion formula is as follows:

[0169] P BASE =T TOOL2BASE *T CAM2TOOL *P CAM

[0170] Among them, T CAM2BASE It can be solved by pseudo-inverse matrix.

[0171] T CAM2BASE =T TOOL2BASE *T CAM2TOOL =P BASE *(P CAM ) -1

[0172] Assume that the corner point cloud set obtained in the camera coordinate system is P = {p1, p2, p3, ... p n}, obtain the corresponding teaching point cloud set Q = {q1, q2, q3, ...q n There exists a pairwise correspondence between the two point sets. There exist rotation and translation matrices R and t such that the following holds.

[0173] q i =R*p i +t

[0174] Solve the centroid corresponding to point sets P and Q:

[0175]

[0176]

[0177] Decentroiding operation:

[0178] p′ i =p i -p

[0179] q′ i =q i -q

[0180] Solving the transformation matrix is ​​essentially an optimization problem, which is to minimize the transformation error. The formula for minimizing the error is:

[0181]

[0182] Calculate R * Rotation matrix and t * Translation matrix:

[0183]

[0184] t * =p-Rq

[0185] Finally, by using the algorithm to calculate the hypothetical optimal solution R *

[0186]

[0187] tr(RH)≥tr(RH)=tr(BR*H)

[0188] Perform SVD decomposition on H:

[0189] H=U∑V T

[0190] R * =VU T

[0191] R * is the rotation matrix corresponding to the optimal solution, t * is the corresponding translation matrix.

[0192] The second step is to install the object to be scanned on the end flange of the articulated robot, change different postures and move it to the scanning depth of field of the 3D camera for scanning, and use quaternion posture solution to control the robot.

[0193] The object to be scanned replaces the calibration pointer and is mounted on the end flange of the six-axis articulated robot. During scanning, the robot can adjust the posture of the object to move it within the scanning depth of the 3D camera, ensuring low noise and accurate data every time.

[0194] Since the fixed scanned object and the robot end flange are in a rigid transformation relationship, other points can be converted to the initial point to be scanned (the second point mentioned above) through the coordinates of the articulated robot's own end flange (END) (or the loaded tool pointer end coordinate TOOL).

[0195] The robot's current position is expressed using point coordinates plus quaternions. Q = (x, y, z, (q0, q1, q2, q3)), and a point on the point cloud of the scanned object is P = (x, y, z). Assume that in the 3D camera visual coordinate system, the initial scan point and quaternion are And a point on the point cloud of the robot body base coordinate system after transformation is in Can be converted into the corresponding rotation and translation matrix The other point transformed from the point to be scanned to the corresponding point cloud of the robot body base coordinate system is The corresponding robot position and quaternion are Can be converted to At this time, the target point transformed from other points to be scanned to the initial point to be scanned is

[0196]

[0197]

[0198] …

[0199]

[0200] Among them, the transformation relationship is:

[0201]

[0202]

[0203] …

[0204]

[0205] Therefore, the point clouds of other points to be scanned Change to the initial point to be scanned The transformation matrix is:

[0206]

[0207]

[0208] …

[0209]

[0210] From this we can see that is the transformation matrix from the other points to be scanned to the initial point to be scanned. After all-around scanning without blind spots, the point clouds after scanning different points are in the same position, and finally a reconstructed 3D model is obtained.

[0211] The third step is to use a filtering algorithm to filter out the redundant point clouds of the reconstructed 3D model to obtain the final reconstructed 3D model. The final reconstructed 3D model is matched and compared with the original digital model, the difference after comparison is output, and the deviation value corresponding to the difference is calculated.

[0212] The initial posture of the point to be scanned Since the spatial position and orientation of the support rod are known, the outer bounding box is also known. The point cloud around the support rod and any irrelevant points in the surrounding area is filtered out using an algorithm. The method involves filtering the point cloud based on spatial position, setting a threshold for the corresponding dimension, and then traversing all points in the point cloud to remove any points that are outside the threshold range for the corresponding dimension. Since point clouds are points in space, filtering in three dimensions is required:

[0213]

[0214] After filtering, the final reconstructed 3D model contains only the complete shape of the object itself. The original digital model for comparison can be provided by the automation manufacturer. The overall point cloud to be compared is matched with the original digital model using the Iterative Closest Point algorithm (an algorithm that accurately aligns point clouds). The provided original digital model is then used for comparison. The degree of difference is mapped into a color scale of the point cloud. The color of the point cloud is used to indicate the degree of difference. This can be used to detect protrusions and depressions to be compared, locate defects, and model size differences.

[0215] The original digital model used for comparison is primarily composed of triangular facets, which are simple and intuitive to represent, and offer fast and efficient traversal. By utilizing the geometric relationship between the point cloud of the final reconstructed 3D model and the triangular facets in the original digital model, the deviation between the point cloud of the final reconstructed 3D model and the original digital model can be calculated. The point cloud of the final reconstructed 3D model is projected onto the corresponding triangular facets in the original digital model to obtain the correspondence between the point cloud and the facets. This is then used to calculate the deviation of the point cloud of the final reconstructed 3D model relative to the original digital model.

[0216] Calculate the projection point p′ of a point p on the triangle ΔABC on the surface of the original model:

[0217]

[0218]

[0219] in, is the normal vector of the triangle plane, l is the vector With normal vector After calculating the projection point p′, the distance from the point to the plane is calculated as the deviation distance d:

[0220]

[0221] Use the area method to determine whether the projection point p' is within the triangle:

[0222] S ΔABC =S ΔABP′ +S ΔBCP′ +S ΔACP′

[0223] Where S ΔABC is the area of ​​the original triangle, S ΔABP′ 、S ΔBCP′ and S ΔACP′ If the sum of the areas of the triangles formed by any two points and the projection point P′ is less than the area of ​​the original triangle, then the projection point P′ is within ΔABC, forming a point-to-plane correspondence; conversely, if the sum of the areas of the projected triangles is greater than the area of ​​the original triangle, then the projection point P′ is within ΔABC, and a point-to-plane correspondence cannot be formed.

[0224] Using the projections of all points in the final reconstructed 3D model's point cloud, we perform a spatial search on the corresponding triangular faces of the original digital model. We then determine the triangular faces of the original digital model that correspond to each point in the final reconstructed 3D model's point cloud, establishing correspondences. We then use the aforementioned method to determine point-to-face correspondences. If the geometric relationship matches, we calculate the distance between the point's projection and the triangle. This distance is the desired calculated deviation.

[0225] Reference Figure 11 Another embodiment of the present invention further provides an electronic device, which can be any type of smart terminal, such as a mobile phone, a tablet computer, a personal computer, etc.

[0226] Specifically, the electronic device 6000 includes: one or more control processors 6001 and a memory 6002, Figure 11 In the example, a control processor 6001 and a memory 6002 are used. The control processor 6001 and the memory 6002 can be connected via a bus or other means. Figure 11 The bus connection is taken as an example.

[0227] The memory 6002 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules corresponding to an electronic device in an embodiment of the present invention;

[0228] The control processor 6001 executes various functional applications and data processing of a three-dimensional reconstruction method by running the non-transitory software programs, instructions and modules stored in the memory 6002, that is, implements a three-dimensional reconstruction method of the above method embodiment.

[0229] The memory 6002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the use of a three-dimensional reconstruction method, etc. In addition, the memory 6002 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 6002 may optionally include a memory remotely located relative to the control processor 6001, and these remote memories may be connected to the electronic device 6000 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0230] When one or more modules are stored in the memory 6002 and are executed by the one or more control processors 6001, a three-dimensional reconstruction method in the above method embodiment is executed, for example, the above described Figures 4 to 10 method steps.

[0231] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0232] It should be noted that, since the electronic device in this embodiment and the above-mentioned three-dimensional reconstruction method are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the device embodiment and will not be described in detail here.

[0233] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute: the three-dimensional reconstruction method as described in the above embodiment.

[0234] It should be noted that, since a computer-readable storage medium in this embodiment and a three-dimensional reconstruction method described above are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the device embodiment and will not be described in detail here.

[0235] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0236] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the scope of the present invention.

Claims

1. A three-dimensional reconstruction method, characterized in that: Used in a three-dimensional reconstruction system, the three-dimensional reconstruction system includes a 3D camera, an articulated robot, a calibration pointer and a checkerboard calibration plate; the 3D camera is set on a stage through a gantry, the articulated robot is used to adjust the posture of the object to be scanned, the articulated robot is set on the stage base, the calibration pointer is set on the end flange of the articulated robot, and the checkerboard calibration plate is set on the stage base; The three-dimensional reconstruction method comprises: Calculating the positional relationship between the end flange of the articulated robot and the 3D camera by using the calibration pointer and the checkerboard calibration plate; calculating the positional relationship between the end flange of the articulated robot and the 3D camera by using the calibration pointer and the checkerboard calibration plate, comprising: Set the calibration pointer coordinate system, the joint robot body coordinate system and the 3D camera coordinate system; Acquire an image of the checkerboard calibration plate through a 3D camera, and convert corner points in the image of the checkerboard calibration plate into corner point coordinates in the 3D camera coordinate system; Obtaining the calibration pointer coordinates of the end of the calibration pointer in the coordinate system of the joint robot body; Calculating a transformation matrix between the 3D camera coordinate system and the articulated robot body coordinate system by using the corner point coordinates in the 3D camera coordinate system and the calibrated pointer coordinates in the articulated robot body coordinate system; Optimizing the conversion matrix by using the calibrated pointer coordinates and the coordinates of at least three corner points in the 3D camera coordinate system, and using the optimized conversion matrix as the positional relationship between the end flange of the articulated robot and the 3D camera; Placing the object to be scanned on the end flange of the articulated robot, and controlling the posture of the object to be scanned to change according to the position relationship; Scanning the object to be scanned in each of the postures to obtain a point cloud of the object to be scanned in each of the postures; The point clouds under each of the postures are spliced ​​together to obtain a reconstructed three-dimensional model of the object to be scanned.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The calculation of the transformation matrix between the 3D camera coordinate system and the articulated robot body coordinate system by using the corner point coordinates in the 3D camera coordinate system and the calibrated pointer coordinates in the articulated robot body coordinate system includes: in, represents the point coordinates in the joint robot body coordinate system, Represents the point coordinates in the 3D camera coordinate system, Represents the conversion matrix from the calibration pointer point coordinates in the 3D camera coordinate system to the point coordinates in the joint robot body coordinate system, Represents the transformation matrix that converts the point coordinates in the 3D camera coordinate system into the point coordinates in the joint robot body coordinate system. Represents the transformation matrix between the 3D camera coordinate system and the joint robot body coordinate system.

3. The three-dimensional reconstruction method according to claim 1, wherein: The step of controlling the object to be scanned to change its posture according to the positional relationship includes: Acquire a point cloud of the object to be scanned in the 3D camera coordinate system, and represent a first point position by combining the point cloud of the object to be scanned with a quaternion; Converting the first point position into a second point position in the coordinate system of the articulated robot body according to the optimized conversion matrix of the positional relationship; Calculating a corresponding rotation and translation matrix for transforming a preset point to be scanned to the second point; The position and posture of the object to be scanned is changed according to the rotation and translation matrix.

4. The three-dimensional reconstruction method according to claim 1, characterized in that: The three-dimensional reconstruction method further includes: The redundant cloud points of the reconstructed three-dimensional model are filtered out to obtain the final reconstructed three-dimensional model.

5. The three-dimensional reconstruction method according to claim 4, characterized in that: The filtering out redundant cloud points of the reconstructed three-dimensional model to obtain a final reconstructed three-dimensional model includes: Setting a three-dimensional point cloud threshold according to the spatial position of the reconstructed three-dimensional model; The point cloud of the reconstructed three-dimensional model is traversed, and point clouds outside the point cloud threshold are removed to obtain the final reconstructed three-dimensional model.

6. The three-dimensional reconstruction method according to claim 5, characterized in that: The three-dimensional reconstruction method further includes: Calculating the deviation between the point cloud of the final reconstructed three-dimensional model and the original model; the calculation method includes: Projecting the point cloud of the final reconstructed three-dimensional model onto the triangular facets of the original model; The surface triangles corresponding to the point cloud on the triangular patch are calculated by the following formula Projection on: in, Represents the vector from the origin of space to the projection point, Represents the vector from the origin of space to the point cloud, is the normal vector of the triangle plane, is a vector With normal vector The dot product of Calculate the deviation distance of the point cloud of the second reconstructed three-dimensional model, and the calculation formula includes: in, Indicates the deviation distance.

7. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the three-dimensional reconstruction method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the three-dimensional reconstruction method according to any one of claims 1 to 6.

Images