A method for monitoring the assembly process of precast concrete components based on stereo vision

Abstract

The present invention discloses a method for monitoring the assembly process of precast concrete components based on stereo vision. A binocular camera is applied to the hoisting construction process of precast components, and a method for target classification and center positioning based on ellipse detection is proposed. Then, a reference coordinate system and a component coordinate system are established according to the set targets, and corresponding control points are selected to monitor the alignment process of the sleeve - steel bar, and the attitude of the component is adjusted. When the errors of all control points are small enough, it indicates that the component can fall smoothly. Precise adjustment starts when the component falls to the target height, and at the same time, the errors of multiple control points are controlled. When the errors are reduced to the allowable range, the structural assembly is completed. Compared with the traditional method, the present invention reduces the labor cost and equipment investment, and can clearly provide the deviation amount. In addition, the proposed circle center detection based on deep learning can improve the detection rate and accuracy; the proposed structural hoisting monitoring based on coordinate system transformation can effectively measure the three processes of structural hoisting.

Description

Technical Field

[0001] The present invention relates to a method for monitoring the assembly process of precast concrete components based on stereo vision, and belongs to the technical field of structural construction monitoring. Background Technique

[0002] In recent decades, with the continuous development of building industrialization, prefabricated structures have promoted the popularization and use of such technologies around the world due to the advantages of their production and construction models in reducing construction time and costs, saving labor, and reducing pollution. At present, many scholars have applied new technologies to the construction process of prefabricated structures. However, in view of the current situation of the civil engineering industry, these emerging technologies are mostly used for the acceptance and management of precast components. The displacement measurement during the hoisting process of prefabricated structures still uses tools such as level gauges, theodolites, and total stations. Specialized technical personnel are required during the use of these devices, and they have low automation and high usage costs. Currently, the world is facing the problems of labor shortage and increasing labor costs, and the construction industry is also facing such a dilemma. According to relevant research, the civil engineering industry is currently extremely dependent on skilled labor, and surveyors, as skilled labor, are particularly in short supply. Applying emerging technologies to develop automated measurement methods can effectively reduce the impact of labor shortage on the construction industry and is also an inevitable trend for future development.

[0003] With the continuous development of technology, many new and high-precision displacement measurement technologies have been used in the measurement practice of civil engineering, which can be divided into two types: contact measurement and non-contact measurement. For contact sensors, there are types such as dial gauges, cable displacement sensors, linear variable differential pressure sensors, magnetostrictive displacement sensors, and liquid level gauges. Such sensors, such as LVDT (linear variable differential transformer), have the characteristics of high precision and can work in harsh environments. However, they need to set up reference points near the structure and have complex wiring, making it difficult to apply such methods in practice. Non-contact sensors include LDV (Laser Doppler vibrometer), microwave radar, feature tracking algorithms in computer image technology, particle image velocimetry (PIV), and template matching methods. These non-contact measurement methods have achieved good results in actual use, but they are limited to two-dimensional plane displacement measurement, while the measurement structure during the hoisting process of prefabricated structures undergoes three-dimensional spatial displacement. Even if the plane displacements in their respective directions are measured simultaneously from two directions, the out-of-plane displacement will render the plane displacement measurement method ineffective.

[0004] There are also some effective methods for the measurement of three-dimensional displacement, including GPS (Global Positioning System), TLS (three-dimensional laser scanning), photogrammetry, etc. GPS is also widely used in three-dimensional displacement measurement, but its error can usually reach the centimeter level and receivers need to be installed at each measuring point. Neither the accuracy nor the usage method is suitable for the measurement during the hoisting process. LiDAR can directly obtain the three-dimensional point cloud data of the overall structure through laser scanning and has received extensive attention in the field of civil engineering in recent years. However, the error is relatively large, and the cost of TLS equipment is expensive and cannot be used in large quantities. Photogrammetry is a measurement method with lower cost and better convenience. A single image can accurately calculate the planar displacement, but there are often large errors in estimating the depth.

[0005] At present, the method of binocular stereo vision has received more and more attention due to its advantages of high accuracy, multi-point measurement, and fast measurement speed. There have been many studies in the field of civil engineering to prove the feasibility of this method. Binocular cameras can monitor the three-dimensional displacement of each point during the vibration of the structure, and the standard error of the measurement results compared with LVDT and LDV sensors is less than 0.5 mm. At the same time, based on the known deformation amount, the dynamic strain prediction of the whole field can be obtained through simulation, and it has the ability to predict dynamic strain. To sum up, the binocular stereo vision scheme has the advantages of sufficient accuracy, multi-point measurement, low cost, and fast measurement speed, and is also applicable to a large field of view, meeting the requirements for constructing hoisting measurement.

[0006] Therefore, compared with the defects of traditional methods, it is necessary to propose a method with the advantages of full-process monitoring, high automation, and saving labor and equipment costs. Summary of the Invention

[0007] The technical problem to be solved by the present invention is: to provide a method for monitoring the assembly process of precast concrete components based on stereo vision, aiming at measuring the aerial attitude of components on-site during the hoisting process to assist in accurate assembly, reducing labor costs and equipment investment, and clearly providing the deviation amount.

[0008] The present invention adopts the following technical solutions to solve the above technical problems:

[0009] A method for monitoring the assembly process of precast concrete components based on stereo vision, comprising the following steps:

[0010] Step 1, paste 3 different types of targets on the existing structure and the precast component respectively, install binocular cameras and adjust the camera angles so that the fields of view of the left and right cameras can cover all the targets;

[0011] Step 2: Fix the camera pose and calibrate it using a calibration board to obtain the internal and external parameters of the binocular camera;

[0012] Step 3: Use the binocular camera to collect images of the precast component with a target and the existing structure, and use the pre-trained Yolov4-tiny network model to classify and identify the targets in the images to obtain the initial positions and types of the targets;

[0013] Step 4: According to the initial positions of the targets, use the ellipse center detection method to obtain the pixel coordinates of the targets, and then obtain the three-dimensional coordinates of the targets through the binocular stereo vision model;

[0014] Step 5: Establish a reference coordinate system based on the three-dimensional coordinates of the targets on the existing structure, establish a component coordinate system based on the three-dimensional coordinates of the targets on the precast component, and solve the transformation relationship between the reference coordinate system and the component coordinate system;

[0015] Step 6: Set the control points and control plane of the precast component. According to the transformation relationship solved in Step 5, convert the coordinates of the control points in the component coordinate system to the reference coordinate system to obtain the coordinates of the control points in the reference coordinate system;

[0016] Step 7: Compare the coordinates of the control points in the reference coordinate system with the corresponding ideal coordinates. When the errors between the coordinates of all control points in the reference coordinate system and the corresponding ideal coordinates are all less than the preset threshold, it is determined that the hoisting of the precast component is completed.

[0017] As a preferred solution of the present invention, the internal and external parameters of the binocular camera described in Step 2 are obtained through the following mathematical model:

[0018]

[0019] Among them, when solving the internal and external parameters of the left camera, (u v 1) T is the pixel coordinate of point X in the left camera, (x w y w z w 1) T is the world coordinate of point X, point X is any point on the calibration board, f α 、f β is the equivalent focal length of the left camera, u0 and v0 are the pixel coordinates of the principal point of the left camera's image plane, (x c y c z c ) T is the coordinate of point X in the left camera coordinate system, z cis the projection of the distance from the object point in the left camera to the optical center in the optical axis direction, [R T] is the external parameter matrix of the left camera composed of the rotation matrix R and the translation vector T, K is the internal parameter matrix of the left camera, and P is the 3×4 left camera projection matrix.

[0020] As a preferred embodiment of the present invention, the training process of the pre-trained Yolov4-tiny network model in step 3 is as follows:

[0021] Six types of targets are designed and manufactured. 200 pictures and 30 pictures of these six types of targets in different backgrounds are respectively collected as the training set and the validation set. The Mosaic data augmentation method is used to augment the training set. The augmented training set is used to train the Yolov4-tiny network model, and the validation set is used for verification to obtain the pre-trained Yolov4-tiny network model.

[0022] As a preferred embodiment of the present invention, the ellipse center detection method in step 4 is specifically as follows:

[0023] For the initial position of the target, the edge of the target is detected by the gray moment edge detection method to obtain arc segments. A threshold is set to remove the straight arcs and short arcs in the arc segments. The remaining arc segments are divided into the first to fourth quadrants by using the gradient of the edge pixels and the concavity and convexity of the arcs. One arc segment is selected from any three quadrants, and it is judged whether the three selected arc segments belong to the same ellipse. If so, the center of the ellipse is calculated. The selection is repeated several times to obtain 6 centers of the ellipse, and the average value of the 6 centers is taken as the final center of the ellipse.

[0024] As a preferred embodiment of the present invention, obtaining the three-dimensional coordinates of the target through the binocular stereo vision model in step 4 is specifically as follows:

[0025] According to the projection equations of the left and right cameras, a constraint equation is jointly established, and the constraint equation is solved by the least squares method to obtain the three-dimensional coordinates of the target; the constraint equation is as follows:

[0026]

[0027] Among them, (u l v l ), (u r v r ) are the pixel coordinates of point X′ in the left and right cameras respectively, P 1l , P 2l , P 3l respectively represent the first, second, and third rows of the left camera projection matrix, and P 1r , P 2r , P 3rThey respectively represent the first, second, and third rows of the right camera projection matrix, and the point X′ is the three-dimensional coordinate of any point on the target.

[0028] As a preferred embodiment of the present invention, the specific process of step 5 is as follows:

[0029] Establish a reference coordinate system o-xyz based on the target on the existing structure, and establish a component coordinate system o′-x′y′z′ based on the target on the precast component. The coordinate of any control point on the precast component in the component coordinate system is P′ = (x′ y′ z′), and the coordinate of this control point transformed to the reference coordinate system is P = (x y z). The relationship between P′ and P is as follows:

[0030] P = RP′ + T

[0031] Wherein, R is the rotation matrix between the reference coordinate system and the component coordinate system, and T is the translation vector from the component coordinate system to the reference coordinate system in the reference coordinate system. The rotation matrix is equal to the direction cosine matrix and is obtained by solving the directions of the known reference coordinate system and the component coordinate system:

[0032]

[0033] Wherein, i, j, and k are the unit vectors of each axis of the reference coordinate system respectively, and i′, j′, and k′ are the unit vectors of each axis of the component coordinate system respectively;

[0034] Let O1 be the origin of the reference coordinate system and O2 be the origin of the component coordinate system. The three-dimensional coordinates of the two in the world coordinate system are O1 = (x1 y1 z1) and O2 = (x2 y2 z2) respectively. The translation matrix in the reference coordinate system is obtained by the following formula:

[0035] T = R′T′ = R′(O2 - O1) = R′(x1 - x2 y1 - y2 z1 - z2)

[0036] Wherein, R′ is the rotation matrix between the camera coordinate system and the reference coordinate system.

[0037] As a preferred embodiment of the present invention, there are 5 control points of the precast component, specifically the center of the precast component and the centers of four sleeves located on the edges of the precast component; the control plane is parallel to the bottom surface of the precast component and is 1.5 m away from the bottom surface.

[0038] Compared with the prior art, the present invention adopts the above technical solutions and has the following technical effects:

[0039] 1. The present invention deeply innovates and organically integrates binocular camera measurement technology, computer vision technology, and target detection technology, which can effectively promote the automated and precise installation process of precast components. It is easy to operate and has fast detection speed, greatly reducing labor costs and equipment costs, and is especially suitable for the installation of precast components in a large number of infrastructure projects in China.

[0040] 2. The traditional total station measurement method requires at least two theodolites and two groups of surveyors to cooperate to complete. However, using the stereovision monitoring method of the present invention can greatly reduce labor and equipment costs.

[0041] 3. The traditional method can only measure the final precise adjustment stage, and manual visual observation is still required during the process of aligning the steel bar - sleeve. When the observation angle is poor, such as during the hoisting of pier columns, workers need to lie on the ground to observe, with old - fashioned means and difficult observation methods. The stereovision solution of the present invention can provide hoisting monitoring for three processes, including steel bar - sleeve alignment, component lowering, and precise adjustment.

[0042] 4. Compared with the traditional method that requires operating a total station to measure multiple target points one by one to measure the structural state once, the stereovision system of the present invention can achieve multi - point acquisition. Once the acquisition is completed, the deviation amount of the structural hoisting can be calculated, which can speed up the measurement speed.

[0043] 5. The traditional total station can only provide qualitative data, such as "the left platform is high, the right is low, etc.", and often requires many back - and - forth operations to basically adjust in place. The stereovision method of the present invention can provide accurate adjustment values, clarify the adjustment direction and quantity, and also provide the possibility for the research and development of automated adjustment equipment.

[0044] 6. The traditional monitoring method has certain requirements for the erection of the total station. It not only requires the total station to be on the central axis of the component, but also the total station cannot be erected too close to the observation position, as too large an elevation angle will lead to unobservable situations. In contrast, the equipment erection of the stereovision of the present invention only requires that the target is within the field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is the method framework diagram for monitoring the assembly process of precast concrete components based on stereovision of the present invention.

[0046] Figure 2 is the network structure diagram of Yolov4 - tiny of the present invention.

[0047] Figure 3 is the schematic diagram of the ellipse center detection method of the present invention, where (a) is the classification of ellipse arc segments, and (b) is the positioning of the ellipse center.

[0048] Figure 4 is the schematic diagram of the stereovision model of the present invention.

[0049] Figure 5 It is a schematic diagram of the component hoisting process of the present invention.

[0050] Figure 6 It is a schematic diagram of the hoisting structure of the present invention.

[0051] Figures 7(a)-7(f) It is a schematic diagram of the pier column hoisting result of the present invention. Detailed implementation manners

[0052] The following details the implementation manners of the present invention. Examples of the implementation manners are shown in the drawings. The implementation manners described below with reference to the drawings are exemplary and are only used to explain the present invention, and should not be construed as a limitation to the present invention.

[0053] As Figure 1 shown, it is a method framework diagram for monitoring the precast concrete component assembly process based on stereo vision of the present invention. This method includes binocular stereo vision three-dimensional displacement measurement, ellipse detection based on deep learning, and structural hoisting monitoring based on coordinate system transformation. According to the returned coordinate information, the deviation amount can be clearly obtained, and the control point coordinates can be adjusted to fall to the ideal elevation. Finally, the hydraulic press is used to precisely adjust the plane coordinates of the control points. After multiple adjustments, the control point coordinates are made to be as close as possible to the ideal values, and the hoisting is completed after obtaining the optimal result. The specific steps are as follows:

[0054] (1) First, different types of target marks are pasted on the control points of the object to be monitored, and the target marks on both sides of the component form a control plane.

[0055] (2) A binocular camera is used to collect images of the component with target marks arranged in the previous step. The Yolov4-tiny network is used to identify the target marks in the images, and the gray moment edge detection method is used to perform sub-pixel level segmentation on the boundaries to improve the detection accuracy and detection effect.

[0056] (3) For the images collected in the previous step, using the stereo vision model and the parallax principle, the three-dimensional coordinates of the space points are determined by two cameras through the binocular stereo vision model. The process is that the internal and external parameters of the two cameras can be obtained through camera calibration. According to the projection equations of the left and right cameras, a constraint equation is jointly established and solved using the least squares method.

[0057] (4) According to the measured values of the control points in the camera coordinate system obtained from the images collected by the binocular camera in the previous step, the space coordinates of the control points are converted by establishing a space rectangular coordinate system and a coordinate system transformation method, and then the spatial position of the component is judged by comparing the control point coordinates with the design values.

[0058] Specifically, the ellipse detection method based on deep learning mainly detects the center of the ellipse through target recognition and classification based on the Yolov network. A lightweight Yolov target detection network is used to train a target recognition and classification model with high confidence and light weight. While classifying the target, it can also remove the invalid background in the image and reduce the amount of calculation. Secondly, due to the inherent characteristics of the perspective projection transformation, the center target of the circle will often be deformed into an ellipse due to geometric distortion. Therefore, the ellipse detection method is used to locate the center of the target in the recognition frame, and the ellipse that appears in the image as a large eccentricity due to geometric distortion can also be effectively detected. The target classification and center positioning method not only realizes the classification and detection speed of the target, effectively avoids the situation of false detection and missed detection under a large field of view, but also solves the problem that the circle detection is sensitive to geometric distortion.

[0059] Specifically, the target recognition and classification method based on the Yolov network is designed and manufactured by 6 types of targets. The target is composed of several circles. The large circle in the middle is the object to be detected by the ellipse detection, and the number of small circles around it is used as the basis for distinguishing different landmarks. 200 and 30 pictures of the 6 types of targets under different backgrounds are collected as training sets and verification sets respectively. These data are then put into the constructed deep learning network for training, and the trained model is saved. In order to train a model with a high recognition rate in the case of a small number of data sets, the training data is subjected to the Mosaic data enhancement training technique during the training process, that is, 4 training images are randomly read, and after operations such as cropping and scaling, they are combined into one picture according to a certain ratio, which increases the number of batches in the training process in disguise and reduces the requirements for hardware. The target detection model is trained using the prepared data set and the Yolov4-tiny network. The pictures collected by the binocular camera are input into the trained Yolov4-tiny model, and the model recognizes the approximate position of the target and classifies it, outputs the pixel coordinates of each recognition frame and the type of recognition, and is used for ellipse detection of the target.

[0060] like Figure 2As shown in the figure, the Yolov4-tiny network structure can be divided into three parts: CSPDarkNet53-tiny, FPN, and YoloHead. The backbone network of Yolov4-tiny uses CSPDarkNet53-tiny to replace CSPDarkNet53 of Yolov4. The CSPDarkNet53-tiny network still uses the CSPBlock module. Compared with the Resblock module of Yolov3-tiny, the CSPBlock module can improve the learning ability of the convolutional network. Although the amount of computation increases, its detection accuracy is also improved. In terms of improving the detection speed, Yolov4-tiny uses the Leaceryru function to replace the MISH activation function as the activation function in the CSPDarkNet53-Tiny network. At the same time, Yolov4-tiny uses the Feature Pyramid Network (FPN) to replace the Spatial Pyramid Pooling and Path Aggregation Network (PANet) to further reduce the detection time. Finally, Yolov4-tiny uses two different scale feature maps, namely 13×13 and 26×26, to predict the detection results.

[0061] Specifically, as Figure 3 shown, the ellipse center detection method is to perform sub-pixel level segmentation on the boundary through the gray moment edge detection method to improve the detection accuracy and detection effect. The basic principle is to assume that the actual edge distribution in the actual image is consistent with the gray moment of the ideal step edge model and the moment invariance principle, and determine the position of the actual edge through this relationship. In 1D edge extraction, let the sequence g j (j = 1, 2,..., n) be the gray values of the actual edge points, then the first three-order gray moments of this sequence are:

[0062]

[0063] Its sub-pixel edge is:

[0064]

[0065] In the formula

[0066] When processing a 2D image, the 2D sub-pixel edge can be determined by applying the above 1D edge detection along the gradient direction at each point along the initial edge.

[0067] After obtaining the corresponding boundary using the edge detection algorithm, there is still noise other than the ellipse boundary. Thresholds need to be set separately to remove straight arcs with very small curvatures and short arcs with few pixels to reduce the subsequent amount of computation. To reduce the number of arcs combined with each other, the ellipse is divided into four parts using the gradient of the edge pixels and the concavity and convexity of the arcs, as Figure 3(a). First, divide the areas of the two regions of the smallest bounding rectangle where it is located by arc segments, as the condition for the first classification of the arc segments. Define δ as the difference between the number of pixels above and below the arc segment. When the area above the arc segment is greater than the area below, δ is positive; otherwise, it is negative. When δ < 0, this arc is convex upward and located in the first or second quadrant; conversely, this arc is convex downward and located in the third or fourth quadrant. Secondly, according to the gradient direction θ of each edge point i , conduct the second classification of each arc segment. When the arc is convex upward and θ i < 0, then it belongs to the first quadrant; conversely, this arc is located in the second quadrant; when the arc is convex downward and θ i < 0, then it belongs to the third quadrant; otherwise, it is located in the fourth quadrant.

[0068] Such as Figure 3 (b), after completing the classification of the elliptical arc segments, select three arcs that meet the quadrant position constraints as a group. Estimate the ellipse center based on the property that the midpoint connection of a group of parallel chords of the ellipse must pass through the ellipse center, and use the sandwich method to find a group of parallel chords and obtain the midline of this group of parallel chords. Similarly, find the center lines of different arc segments to determine an ellipse center. Using the above method, the centers of the ellipses formed by pairwise arc pairs can be calculated. If the distance between the obtained center points is less than the threshold, it is considered that these arcs may belong to the same ellipse. The equations of the midlines of the 4 groups of parallel chords that can be calculated for the three arc segments, theoretically, these 4 groups of midline equations intersect at the same center point. Due to errors, at most 6 center points will be generated. Take the average value of the 6 groups of points as the ellipse center.

[0069] Then use the least squares method to fit the elliptical equation for each candidate point on the arc to obtain other parameters of the ellipse. Let several discrete detection points on the ellipse be represented as P i (x i , y i ), where i = 1, 2,..., N. Let the objective function be:

[0070]

[0071] When the value of the objective function is minimized, determine the values of each coefficient. According to the extreme value principle, there are the following equalities:

[0072]

[0073] The values of each coefficient solved can further derive the values of the major axis, minor axis, and inclination angle of the major axis of the ellipse. According to the center point, major axis, minor axis, and inclination angle of the ellipse, group the ellipses with close numerical values into one category, and then score the ellipses based on the distance from each boundary point to the fitted ellipse. Then, use the ellipse with the highest score to represent this category.

[0074] Specifically, binocular stereo vision three-dimensional displacement measurement reconstructs the three-dimensional coordinates of spatial points through multiple images based on the parallax principle. Generally, the pinhole camera model is used to describe the camera imaging process. The conversion relationship between the two-dimensional information and three-dimensional information of the measurement point X follows the pinhole camera model, and its mathematical model is expressed as:

[0075] The mathematical model transformed into image pixel coordinates through the camera model can be expressed for the left camera as:

[0076]

[0077] where x = (u v 1) T is the pixel coordinate of point X in the left camera, X = (x w y w z w 1) T is the world coordinate of point X, f α 、f β are the equivalent focal lengths of the camera, u0 and v0 are the pixel coordinates of the principal point of the camera's image plane, z c is the projection of the distance from the object point to the optical center in the camera along the optical axis direction. [R T] is the external parameter matrix of the left camera composed of the rotation matrix R and the translation vector T, K is the internal parameter matrix of the camera, and P is the camera projection matrix with 3 rows and 4 columns.

[0078] In dynamic measurement, two cameras are used to determine the three-dimensional coordinates of spatial points through the binocular stereo vision model, and the stereo vision model is as Figure 4 shown. At this time, it is assumed that the world coordinate coincides with the left camera coordinate system. Then, the rotation matrix and translation vector of the left camera are both 0, while the rotation matrix and translation vector of the right camera are the rotation and translation transformation relationships of the right camera coordinate system relative to the left camera coordinate system, which can be solved through camera calibration. The process is to obtain the internal and external parameters of the two cameras through camera calibration, and jointly establish constraint equations for solution according to the projection equations of the left and right cameras. The mathematical relationship is expressed as follows:

[0079]

[0080] where (u l v l ), (u r v r ) are the pixel coordinates of point X' in the left and right cameras respectively, P il 、P ir respectively represent the i-th (i = 1, 2, 3) rows of the left and right camera projection matrices. There are 4 sets of independent equations in the above formula to solve 3 unknowns, and the least squares method can be used for solution.

[0081] Specifically, the structural hoisting monitoring method based on coordinate system transformation measures and transforms the coordinates of the control points of the points with obvious features through a binocular camera, and then compares them with the design values to obtain the adjustment deviation. Since the points measured by the binocular camera are limited to the object surface, it is difficult to measure all the control points within the field of view of a single binocular camera. Therefore, the present invention adopts the method of establishing a space rectangular coordinate system and coordinate system transformation to calculate the space coordinates of the control points, and then judges the spatial position of the component by comparing the control point coordinates with the design values.

[0082] Specifically, the transformation of the coordinate system is to establish a reference coordinate system o-xyz and a component coordinate system o'-x'y'z' by setting targets on both the existing structure and the hoisting component. In the component coordinate system, the three-dimensional space coordinates of the control points are solved through the known component design information. Then, the control point coordinates are transformed into the reference coordinate system through coordinate system transformation, and the coordinates of the control points can be compared with the design values. Let point P be any control point on the component, and its coordinates in the component coordinate system are obtained according to the component design information as P'=(x' y' z'); the coordinates of point P transformed into the reference coordinate system are P=(x y z). The transformation of the three-dimensional coordinates in different space coordinates is divided into two steps: rotation transformation and translation transformation, which can be represented by a rotation matrix and a translation vector, and their relationship is shown in the following formula:

[0083] P = RP'+T

[0084] Among them, R is the rotation matrix of the reference coordinate system and the component coordinate system, and T is the translation vector from the component coordinate system to the reference coordinate system in the reference coordinate system. Among them, the rotation matrix is equal to the direction cosine matrix, which can be obtained by solving the directions of the known reference coordinate system and the component coordinate system:

[0085]

[0086] Among them, i, j, k, i', j', k' are the unit vectors of each axis of the reference coordinate system and the component coordinate system respectively.

[0087] The solution of the translation vector can be obtained by subtracting the origins of the reference coordinate system and the component coordinate system, but it still needs to be transformed from the camera coordinate system to the reference coordinate system. Let O1 be the origin of the reference coordinate system, and O2 be the origin of the component coordinate system. The three-dimensional coordinates of the two in the measured world coordinate system are O1=(x1 y1 z1), O2=(x2 y2 z2) respectively. The translation matrix in the reference coordinate system can be obtained by the following formula:

[0088] T = R'T' = R'(O2 - O1) = R'(x1 - x2 y1 - y2 z1 - z2)

[0089] where R′ is the rotation matrix between the camera coordinate system and the reference coordinate system, and the solution method is as described above. The direction vectors of the three principal axes of the x-axis, y-axis, and z-axis in the camera coordinate system are (1 0 0), (0 1 0), and (0 0 1), respectively. Figure 5 is the process of component hoisting.

[0090] The following uses a typical column hoisting embodiment to illustrate the specific implementation steps of the method for monitoring the assembly process of precast concrete components based on stereo vision invented.

[0091] Step 1: Work preparation. First, set up targets on the pile cap and the column respectively, install the binocular cameras and adjust the camera angles so that the fields of view of the two cameras can cover the targets simultaneously. Fix the camera poses and calibrate them with a calibration board to obtain the internal and external parameters of the binocular cameras, and obtain the three-dimensional coordinates of the targets through the stereo vision model. Establish the reference coordinate system and the component coordinate system respectively according to the three-dimensional coordinates of the targets, and solve the conversion relationship between the two coordinate systems. Convert the coordinates of the component coordinate system to the reference coordinate system so that the measured values and the design values can be directly compared.

[0092] Step 2: Control point setting. Set the control plane parallel to the bottom of the column, 1.5 m away from the bottom plane. The control points are the center of the column and the centers of the four sleeves at the corners. Assume that the control plane coincides with the O-XY plane of the component coordinate system, and the origin O is at the center of the column. Then the ideal coordinates of the five control points when the hoisting is completed are (0, 0, 0), (910, 910, 0), (910, -910, 0), (-910, 910, 0), (-910, -910, 0). When all the control points coincide with the ideal coordinates, it can be regarded as the completion of hoisting.

[0093] Step 3: Monitoring the process of aligning the steel bars and sleeves. Lift the column above the pile cap, adjust the structural pose so that the design values of the control point coordinates in the X-axis direction and the Y-axis direction are close. When the planar coordinates of the five control points are close to the design values, the sleeves are aligned with the steel bars, and the column can continue to descend. At this time, the steel bars on the pile cap can enter the sleeves at the bottom of the column, and then adjust the coordinates of the control points on the column and descend to the ideal elevation.

[0094] Step 4: Precise control process. Based on the hoisting position at the end of the previous step, start to finely adjust the pose of the column, and use a hydraulic press to precisely adjust the planar coordinates of the control points. After multiple adjustments, make the control point coordinates as close to the ideal values as possible. Finally, when the errors of the five control points reach small values simultaneously, it can be judged that the column hoisting is completed. Figure 6 is the schematic diagram of the hoisting structure. Figure 7(a) shows the changes in the planar coordinates of each control point during the fine adjustment process of the column. Figures 7(b)-7(f) shows the dynamic change process of the last 10 control points.

[0095] The above method of using a binocular camera to monitor the hoisting process of prefabricated structures reduces labor costs and equipment investment compared to traditional methods, and can clearly provide the deviation amount. After establishing the reference coordinate system and the component coordinate system according to the set target, and selecting the corresponding control points according to the corresponding structure, first monitor the casing-rebar alignment process, adjust the component posture, and when the control point errors are small enough, the component can fall smoothly. When the component falls to the target height, precise adjustment can be started. During the fine-tuning process, the errors of the control points are controlled at the same time. When the errors of the two are reduced to the allowable range, the structural assembly is completed. Comprehensive evaluation shows that the method of the present invention has good accuracy, practicality and advancement, and has a wide range of application prospects for the precise hoisting of prefabricated components.

[0096] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for monitoring the assembly process of precast concrete components based on stereo vision, characterized in that, It includes the following steps: Step 1: Paste 3 targets of different types on the existing structure and the precast component respectively. Install a binocular camera and adjust the camera angle so that the fields of view of the left and right cameras can cover all the targets; Step 2: Fix the camera pose and calibrate it with a calibration board to obtain the internal and external parameters of the binocular camera; The internal and external parameters of the binocular camera are obtained through the following mathematical model: Among them, when solving the internal and external parameters of the left camera, (u v 1) T is the pixel coordinate of point X in the left camera, (x w y w z w 1) T is the world coordinate of point X, where point X is any point on the calibration board, f α 、f β are the equivalent focal lengths of the left camera, u0 and v0 are the pixel coordinates of the principal point of the left camera's image plane, (x c y c z c ) T is the coordinate of point X in the left camera coordinate system, z c is the projection of the distance from the object point to the optical center in the left camera in the direction of the optical axis, [R T] is the external parameter matrix of the left camera composed of the rotation matrix R and the translation vector T, K is the internal parameter matrix of the left camera, and P is the 3×4 left camera projection matrix; Step 3: Use the binocular camera to collect images of the precast component and the existing structure with targets, and use the pre-trained Yolov4-tiny network model to classify and identify the targets in the images to obtain the initial positions and types of the targets; Step 4: According to the initial positions of the targets, use the ellipse center detection method to obtain the pixel coordinates of the targets, and then obtain the three-dimensional coordinates of the targets through the binocular stereo vision model; Step 5: Establish a reference coordinate system based on the three-dimensional coordinates of the targets on the existing structure, establish a component coordinate system based on the three-dimensional coordinates of the targets on the precast component, and solve the transformation relationship between the reference coordinate system and the component coordinate system; The specific process is as follows: Establish a reference coordinate system o-xyz based on the targets on the existing structure, establish a component coordinate system o′-x′y′z′ based on the targets on the precast component. The coordinates of any control point on the precast component in the component coordinate system are P′=(x′ y′ z′), and the coordinates of this control point in the reference coordinate system are P=(x y z). The relationship between P′ and P is as follows: P = RP′ + T where, R is the rotation matrix between the reference coordinate system and the component coordinate system, T is the translation vector from the component coordinate system to the reference coordinate system in the reference coordinate system. The rotation matrix is equal to the direction cosine matrix and is obtained by solving the directions of the known reference coordinate system and the component coordinate system: where i, j, and k are the unit vectors of the respective axes of the reference coordinate system, and i ′ , j ′ , k ′ are the unit vectors of the respective axes of the component coordinate system; Let O1 be the origin of the reference coordinate system and O2 be the origin of the component coordinate system. The three-dimensional coordinates of the two in the world coordinate system are O1=(x1 y1 z1) and O2=(x2 y2 z2) respectively. The translation matrix in the reference coordinate system is obtained by the following formula: T = R ′ T ′ = R ′ (O2 - O1) = R ′ (x1 - x2 y1 - y2 z1 - z2) where R ′ is the rotation matrix between the camera coordinate system and the reference coordinate system; Step 6: Set the control points and control planes of the precast component. According to the transformation relationship solved in Step 5, convert the coordinates of the control points in the component coordinate system to the reference coordinate system to obtain the coordinates of the control points in the reference coordinate system; Step 7: Compare the coordinates of the control points in the reference coordinate system with the corresponding ideal coordinates. When the errors between the coordinates of all control points in the reference coordinate system and the corresponding ideal coordinates are less than the preset threshold, it is judged that the hoisting of the precast component is completed.

2. The method for monitoring the assembly process of precast concrete components based on stereo vision according to claim 1, characterized in that, The training process of the pre-trained Yolov4-tiny network model described in Step 3 is as follows: Design and produce 6 types of targets, collect 200 images and 30 images of these 6 types of targets in different backgrounds as the training set and the validation set respectively. Use the Mosaic data augmentation method to expand the training set, use the expanded training set to train the Yolov4-tiny network model, and use the validation set for validation to obtain the pre-trained Yolov4-tiny network model.

3. The method for monitoring the assembly process of precast concrete components based on stereo vision according to claim 1, characterized in that, The ellipse center detection method described in Step 4 is specifically as follows: For the initial position of the target, detect the edge of the target by the gray moment edge detection method to obtain arc segments. Set a threshold to remove the straight arcs and short arcs in the arc segments. Use the gradient of the edge pixels and the concavity and convexity of the arcs to divide the remaining arc segments into the first to fourth quadrants. Select one arc segment from any three quadrants and determine whether the three selected arc segments belong to the same ellipse. If so, calculate the center of the ellipse. Repeat the selection several times to obtain 6 centers of the ellipse, and take the average value of the 6 centers as the final center of the ellipse.

4. The method for monitoring the assembly process of precast concrete components based on stereo vision according to claim 1, characterized in that, The three-dimensional coordinates of the target obtained through the binocular stereo vision model described in step 4 are as follows: According to the projection equations of the left and right cameras, jointly establish a constraint equation, and solve the constraint equation by the least squares method to obtain the three-dimensional coordinates of the target; the constraint equation is as follows: Among them, (u l v l ), (u r v r ) are the pixel coordinates of point X ′ in the left and right cameras respectively. P 1l , P 2l , P 3l represent the 1st, 2nd, and 3rd rows of the left camera projection matrix respectively. P 1r , P 2r , P 3r represent the 1st, 2nd, and 3rd rows of the right camera projection matrix respectively. Point X ′ is the three-dimensional coordinate of any point on the target.

5. The method for monitoring the assembly process of precast concrete components based on stereo vision according to claim 1, characterized in that, There are 5 control points for the precast member described in step 6, specifically the center of the precast member and the centers of the four sleeves located on the edges of the precast member; the control plane is parallel to the bottom surface of the precast member and is 1.5 m away from the bottom surface.

Images