Segment automatic plastering visual guidance method and system

Abstract

The invention provides a duct piece automatic plastering visual guidance method and system, and relates to the technical field of duct piece automation. The method comprises the following steps: collecting an image and point cloud data of a standard duct piece, obtaining geometric features of the standard duct piece through the image, and recording a motion path of a spatula to be defined as a standard path; then, the image of the segment to be plastered is analyzed to carry out preliminary rough pairing, and then accurate pairing is carried out through point cloud data; calculating the center of the to-be-plastered segment point cloud according to the registered point cloud, generating cleaning tracks, and connecting the cleaning tracks in series to form an actual cleaning path; analyzing the height information of the standard segment point cloud and the height information of the actual segment to be plastered, and optimizing the contact surface to minimize the height distance error; and finally, counting uneven points through image processing, comprehensively analyzing the path smoothness, the distance error and the proportion of the uneven points, and generating a qualification index to evaluate the plastering quality in real time. The full-automatic pipe piece plastering process is achieved, the construction efficiency and quality are remarkably improved, the cost is reduced, and the full-automatic pipe piece plastering machine has wide application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of segment automation, and specifically provides a visual guidance method and system for segment automatic plastering. Background Technique

[0002] In the field of segment plastering, the existing technology has significant defects. Traditional plastering tools are mostly 304 stainless steel or high manganese steel trowels, whose shapes are difficult to adapt to the complex surfaces of segments. Multiple operations are required for plastering, and it highly depends on manual skills. A slight mistake will lead to uneven plastering and defects, and subsequent repeated corrections are needed. Using a steel ruler to assist in plastering is also not ideal. It is laborious to operate, has low efficiency, low automation level, highly depends on manual operation, and performs poorly in precision control, unable to meet the high-standard requirements for segment plastering.

[0003] Currently, the intelligent application in the field of segment plastering is seriously insufficient, and it is still mainly labor-intensive production. Manual plastering has low efficiency and high costs. At least two workers are required at one plastering station to maintain the production line rhythm. Workers are prone to fatigue due to long-term repetitive operations, which not only affects the plastering quality and increases subsequent repair work, but also, as a technical job, plastering workers have high employment costs. With the rising raw material costs, the manufacturing cost of segments remains high, and the competitiveness of enterprises is weak. Due to the lack of intelligent equipment, the plastering process cannot be monitored in real time, and the quality of segments is difficult to guarantee, seriously restricting the development of the industry.

[0005] The existing technology has the following deficiencies:

[0006] Traditional segment plastering processes mostly rely on manual operation, with problems such as low efficiency, unstable quality, and high labor costs. During the manual plastering process, it is often difficult for workers to keep the movement path of the trowel smooth, resulting in uneven plastering effects and even unevenness, affecting subsequent construction and the overall performance of segments. In addition, manual operation has limitations in height control and geometric feature recognition, and it is difficult to achieve high-precision construction. Especially in complex construction environments, errors and defects are likely to occur.

[0007] The above information disclosed in the background technique section is only used to enhance the understanding of the background of the present disclosure, so it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0008] The purpose of the present invention is to provide a visual guidance method and system for segment automatic plastering to solve the problems raised in the above background technique.

[0009] To achieve the above purpose, the present invention provides the following technical solutions:

[0010] A visual guidance method for segment automatic plastering, the specific steps include:

[0011] Step 1: Collect the images and point cloud data of the standard segment. The images are used to obtain the geometric features of the standard segment. By manually operating the robot, record the movement path of the trowel and calculate the smoothness of the path.

[0012] Step 2: Collect the images and point cloud data of the segment to be plastered. Analyze the images of the segment to be plastered to determine the feature points, and perform a preliminary rough matching of the segment to be plastered with the standard segment through the feature points, and then perform precise matching through the point cloud data.

[0013] Step 3: Calculate the center of the point cloud of the segment to be plastered according to the registered point cloud, obtain the cleaning trajectory through the transformation of adjacent point cloud centers, and then concatenate the cleaning trajectories to obtain the actual cleaning path.

[0014] Step 4: Analyze the point cloud data of the standard segment to obtain the height value of each point, then obtain the height value of the trowel at each point, obtain the total distance error between the standard segment and the trowel path, and minimize the contact surface through an optimization algorithm.

[0015] Step 5: Collect the images after the construction of the segment to be plastered in real time, convert the images into grayscale images, count the uneven points in the grayscale images, and comprehensively analyze the smoothness of the path, the total distance error, and the proportion of uneven points to obtain the qualification index of the current plastering and judge whether the current plastering is qualified.

[0016] Further, the specific collection of the relevant data of the standard segment includes:

[0017] The geometric features include the coordinate information of the edges and corners of the standard segment. Use the edge detection algorithm to obtain the edge information of the standard segment, and use the corner detection algorithm to determine the corners of the standard segment.

[0018] Further, the specific calculation of the smoothness of the path includes:

[0019] By manually operating the robot, record the movement path of the trowel, store the path coordinates in the form of a point set, and the format is , indicating the position of the first pixel point on the axis and the axis, is the total number of pixel points. The formula for calculating the smoothness of the standard path is:

[0020]

[0021] Among them, is the smoothness of the standard path, is the abscissa of the th pixel point in the standard path, The vertical coordinate of the th pixel point in the standard path, is the horizontal coordinate of the th pixel point in the standard path, is the vertical coordinate of the th pixel point in the standard path, and .

[0022] Furthermore, the preliminary rough pairing of the segment to be plastered specifically includes:

[0023] Using the edge detection algorithm and the corner detection algorithm to obtain the feature points of the segment to be plastered, that is, the points on the edge and the corner points are used as feature points. For each feature point , calculate the feature vector , is the dimension of the descriptor. Use the Euclidean distance between descriptors to match the feature points, calculate the matched feature point pairs. Let the feature of the segment to be plastered be , and the feature of the standard segment be . The distance formula between feature points is:

[0024]

[0025] where, is the distance between the th feature point of the segment to be plastered and the th feature point of the standard segment, is the feature vector of the th feature point of the segment to be plastered, is the feature vector of the th feature point of the standard segment;

[0026] For rough registration, use RANSAC (Random Sample Consensus algorithm) to select inliers and remove outliers, and calculate the rough registration transformation matrix based on the inliers. The formula is as follows:

[0027]

[0028] where, is the transformation matrix of rough registration, is the rotation matrix, is the translation vector;

[0029] Furthermore, the precise pairing through point cloud data specifically includes:

[0030] Collect the point cloud data of the segment to be plastered and record it as , is the total number of point cloud data of the segment to be plastered. The point cloud data of the standard segment is recorded as , Let \(N\) be the total number of point cloud data of standard segments. Taking the point cloud of standard segments, the point cloud of segments to be plastered, and the initial transformation matrix as inputs, for each point in the point cloud, find the nearest point pair, calculate the optimal transformation between the point pairs, and let the registered point cloud be , and optimize the transformation using the least squares method, which is expressed by the formula:

[0031]

[0032] where is the updated transformation matrix calculated based on the point cloud matching result after each iteration, is any point pair, is the current transformation matrix;

[0033] Apply the updated transformation matrix to obtain the registered point cloud to update the point cloud of the segments to be plastered, which is expressed as , and calculate the root mean square error of the current registration result. The formula is as follows:

[0034]

[0035] where is the root mean square error of the current registration result;

[0036] When , stop the iteration, is the preset threshold.

[0037] Furthermore, the specific steps to obtain the actual cleaning path include:

[0038] The formula for calculating the center of the point cloud of the segments to be plastered is as follows:

[0039]

[0040] where is the center of the point cloud of the segments to be plastered, is the -th point coordinate of the segments to be plastered, that is ;

[0041] Calculate the transformation between adjacent centers to obtain the cleaning trajectory. The formula is as follows:

[0042]

[0043] where is the cleaning trajectory, describing the transformation from the previous geometric center to the current geometric center, is the geometric center of the previous point cloud;

[0044] Generate the actual cleaning path according to the calculated cleaning trajectory to guide the cleaning robot, specifically including:

[0045]

[0046] Among them, is the actual cleaning path, represents the first cleaning trajectory, represents the second cleaning trajectory, represents the th cleaning trajectory, is the number of path points;

[0047] Furthermore, the total distance error of obtaining the standard segment and the trowel path specifically includes:

[0048] Analyze the point cloud data of the standard segment, calculate the height value of each point, and generate a height data set. The formula is as follows:

[0049]

[0050] Among them, is the height value of the th point of the standard segment, is the height data set;

[0051] According to the spatial range of the point cloud, divide it into regular grids. For each grid, count all the points in the grid, calculate the average height of these points, and obtain the height function of the standard segment at any point:

[0052]

[0053] Among them, is the average height of the grid, is the height value of the th point in the grid, is the number of points in the grid;

[0054] After completing the grid division, an interpolation method can be used to generate a continuous height function. The formula is

[0055]

[0056] Among them, is the radial basis function, is the weight coefficient;

[0057] Install a sensor on the trowel, and collect the height data of the trowel position in real time during the movement of the trowel. Compare the real-time monitored trowel height data with the height data of the standard segment to obtain the total height error:

[0058]

[0059] Among them, is the total distance error, The standard segment at the position height, The height at the position on the trowel path height, represents the coordinate information , is the number of points included in the error calculation;

[0060] Define the objective function as the total distance error, minimize the contact surface by optimizing the objective function, and use the least squares method for optimization. Set the height model as:

[0061]

[0062] where, are the parameters to be optimized;

[0063] Calculate the total distance error as the loss function:

[0064]

[0065] Calculate the partial derivatives of the loss function with respect to the parameters and , and update the parameters according to the gradient descent method:

[0066]

[0067]

[0068] where, is the learning rate.

[0069] Furthermore, the judgment of whether the current plastering is qualified specifically includes:

[0070] Real-time collect the image of the currently to-be-plastered segment after construction, convert the image to a grayscale image,

[0071] Then convert the grayscale image to a binary image, and the formula is as follows:

[0072]

[0073] where, is the pixel value of the binary image, is the set pixel threshold;

[0074] Count the number of uneven points in the image, and obtain the proportion of the uneven points in the entire to-be-plastered point cloud. The formula is as follows:

[0075]

[0076] where, is the proportion of uneven points, is the number of uneven points in the image;

[0077] The formula for obtaining the qualification index is:

[0078]

[0079] where, is the qualification index, is the smoothness of the path of the segment to be plastered, is the maximum allowable value of the total distance error, are the weight coefficients of each item respectively, , and ;

[0080] Compare the qualification index with the preset qualification threshold. If the qualification index is higher than the preset qualification threshold, it indicates that the construction of the current segment to be plastered is in a qualified state, and the construction can be stopped. If the qualification index is lower than the preset qualification threshold, it indicates that the construction of the current segment to be plastered is in an unqualified state, then continue plastering and feedback the result to optimize the point cloud pairing step.

[0081] The present invention further provides a visual guidance system for automated segment plastering. The visual guidance system for automated segment plastering is used to implement the above-mentioned visual guidance method for automated segment plastering, including:

[0082] A standard segment feature acquisition module, which is used to acquire the image and point cloud data of the standard segment. The image is used to obtain the geometric features of the standard segment. By manually operating the robot, record the movement path of the trowel and calculate the smoothness of the path;

[0083] A segment to be plastered analysis module, which is used to acquire the image and point cloud data of the segment to be plastered, analyze the image of the segment to be plastered to determine the feature points of the segment to be plastered, and perform a preliminary rough pairing of the segment to be plastered with the standard segment through the feature points, and then perform precise pairing through the point cloud data;

[0084] A cleaning trajectory generation module, which is used to calculate the center of the point cloud of the segment to be plastered based on the registered point cloud, obtain the cleaning trajectory through the transformation of adjacent point cloud centers, and then connect the cleaning trajectories in series to obtain the actual cleaning path;

[0085] A height error analysis module, which is used to analyze the point cloud data of the standard segment to obtain the height value of each point, then obtain the height value of the trowel at each point, obtain the total distance error between the standard segment and the trowel path, and minimize the contact surface through an optimization algorithm;

[0086] The plastering quality assessment module is used to collect the images of the segments after plastering in real time, convert the images into grayscale images, count the uneven points in the grayscale images, and comprehensively analyze the smoothness of the path, the total distance error, and the proportion of uneven points to obtain the qualified index of the current plastering and determine whether the current plastering is qualified.

[0087] In the above technical solution, the technical effects and advantages provided by the present invention are as follows:

[0088] Compared with the plastering technology using ordinary trowels and steel tapes, the present invention has the following advantages. First, the efficiency is greatly improved. The traditional plastering method with ordinary trowels and steel tapes highly relies on manual operation, with a cumbersome process and slow speed. However, the present invention enables more segment plastering tasks to be completed within the same time, greatly improving the production efficiency and meeting the requirements of large-scale production. Second, the plastering quality is significantly improved. Due to the limitations of its shape and operation method, ordinary trowels are extremely likely to cause uneven plastering and defects. When using a steel tape to assist in plastering, it is also difficult to guarantee the accuracy. The present invention can control the flatness error of the segment surface within an acceptable range, reduce subsequent repair work, and improve the reliability and stability of the product. In addition, the long-term operation cost is reduced. The traditional plastering method relies on a large number of skilled workers, resulting in high labor costs. Although there is a certain initial investment in the intelligent plastering system of the present invention, in the long-term operation, the machine replaces a large number of workers, reducing the labor cost expenditure. Moreover, the present invention avoids the safety risk that workers are prone to fatigue due to long-term repetitive labor, which may lead to operation errors. The present invention can timely feedback the construction status, ensure that the plastering meets the predetermined standards, reduce the need for manual intervention, and thus effectively reduce the construction cost and improve the overall construction efficiency. Description of the Drawings

[0089] Figure 1 It is a schematic diagram of the overall method flow of the present invention;

[0090] Figure 2 It is a schematic diagram of the system structure of the present invention. Detailed Embodiments

[0091] To make the objectives, technical solutions, and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to specific embodiments.

[0092] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should have the ordinary meanings understood by those of ordinary skill in the field to which the present invention belongs. The "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "comprising" or "including" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Upper", "lower", "left", "right", etc. are only used to represent relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0093] Embodiment:

[0094] Please refer to Figure 1 , the present invention provides a technical solution:

[0095] An automated visual guidance method for segment plastering, the specific steps include:

[0096] Step 1: Collect the image and point cloud data of the standard segment. The image is used to obtain the geometric features of the standard segment. By manually operating the robot, record the movement path of the trowel and calculate the smoothness of the path;

[0097] In this embodiment, the collection of the relevant data of the standard segment specifically includes:

[0098] The geometric features include the coordinate information of the edges and corners of the standard segment. Use the Canny edge detection algorithm to obtain the edge information of the standard segment, use the Harris corner detection algorithm to determine the corners of the standard segment, and use the edge information and corner information to help determine the coordinate information of the feature points on the standard segment later.

[0099] The edge detection algorithm specifically includes:

[0100] Calculate the gradients of the image in and directions to obtain the edge strength and direction. Suppress non-edge points in the gradient direction and retain edge points. Use double thresholds for edge classification. Finally, determine the final edge by connecting strong edges and weak edges. Represent the edge detection algorithm with the formula:

[0101]

[0102] Among them, is the edge strength value at the recognition position , which is the decision basis for whether this point is an edge. is an image of a standard segment, is the gradient of the image in the horizontal direction, representing the change in image brightness along the x-axis, is the gradient of the image in the vertical direction, representing the rate of change of image brightness along the y-axis. These gradient values are used to determine the edge strength and direction of the image. If the gradient value of a certain point is large, it indicates that this point is an edge. are the low threshold and high threshold. The low threshold is used to preliminarily judge whether a point may be an edge, and the high threshold is used to determine the edge strength. Only when the edge strength exceeds this value will it be determined as an edge; these two thresholds are used for edge connection and screening to reduce the influence of noise on edge detection. The edge detection process is divided into three stages: Strong edges: points with intensity values higher than Weak edges: points with intensity values between and Non-edges: points with intensity values lower than .

[0103] Calculating corner points specifically includes:

[0104] For each pixel point , calculate the gradient information of its surrounding area and construct a second-order matrix according to the following formula:

[0105]

[0106]

[0107] where, is the second-order matrix;

[0108] The formula for calculating the response function is:

[0109]

[0110] where, is the determinant of the matrix , representing the intensity of the corner point, is the trace of the matrix (the sum of the diagonal elements), representing the total change in the local area of the image, is an empirical parameter, usually taking values between 0.04 and 0.06, and is used to adjust the sensitivity of the response function.

[0111] Among them:

[0112]

[0113]

[0114] Perform the above calculations for each pixel point in the image to obtain the response function , determine the threshold value and only retain the points with values greater than the threshold value as corner points.

[0115] Step 2: Collect the image and point cloud data of the segment to be plastered, analyze the image of the segment to be plastered to determine the feature points of the segment to be plastered, perform a preliminary rough matching of the segment to be plastered and the standard segment through the feature points, and then perform a precise matching through the point cloud data;

[0116] In this embodiment, the smoothness of the calculated path specifically includes:

[0117] By manually operating the robot, record the movement path of the trowel, store the path coordinates in the form of a point set, and the format is , indicating the positions of the first pixel point on the axis and the axis, is the total number of pixel points, and the formula for calculating the smoothness of the standard path is:

[0118]

[0119] Among them, is the smoothness of the standard path, is the abscissa of the th pixel point in the standard path, the th pixel point in the standard path, is the abscissa of the th pixel point in the standard path, is the th pixel point in the standard path, and .

[0120] Step 3: Calculate the center of the point cloud of the segment to be plastered according to the registered point cloud, obtain the cleaning trajectory through the transformation of adjacent point cloud centers, and then connect the cleaning trajectories to obtain the actual cleaning path;

[0121] In this embodiment, the preliminary rough matching of the segment to be plastered specifically includes:

[0122] Use the edge detection algorithm and corner detection algorithm to obtain the feature points of the segment to be plastered, that is, the points on the edge and the corner points are used as feature points. For each feature point , calculate the feature vector , Let \(d\) be the dimension of the descriptor. Specific calculation methods for feature vectors include SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc. In this embodiment, the SIFT method is used for calculation. First, the Difference of Gaussians is utilized to find the feature points in the image to be plastered. For each feature point, according to the direction and magnitude of the local image gradient, the main direction of each feature point is calculated. A 16×16 pixel window is defined around the feature point and divided into 4×4 sub-regions. For each sub-region, the histogram of the gradient direction is calculated. Each histogram contains 8 directions. Thus, the feature descriptor of a key point can be expressed as:

[0123]

[0124] where, is the gradient histogram of each sub-region, and its dimension is 8 (number of directions) × 16 (number of sub-regions) = 128;

[0125] The Euclidean distance between descriptors is used to match the feature points, and the pairs of matching feature points are calculated. Let the feature of the segment to be plastered be , and the feature of the standard segment be . The formula for calculating the distance between feature points is:

[0126]

[0127] where, is the distance between the \(i\)-th feature point of the segment to be plastered and the \(j\)-th feature point of the standard segment, is the feature vector of the \(i\)-th feature point of the segment to be plastered, is the feature vector of the \(j\)-th feature point of the standard segment; is the \(i\)-th feature point of the segment to be plastered, is the \(j\)-th feature point of the standard segment;

[0128] For rough registration, RANSAC (Random Sample Consensus algorithm) is used to select inliers and remove outliers, and the rough registration transformation matrix is calculated based on the inliers. The formula is as follows:

[0129]

[0130] where, is the transformation matrix for rough registration, is the rotation matrix, is the translation vector;

[0131] Select For the matched feature points, calculate the centroids of the point cloud of the segment to be plastered and the point cloud of the standard segment. The formula for calculating the centroid of the point cloud of the segment to be plastered is:

[0132]

[0133] Among them, is the centroid of the point cloud of the segment to be plastered. The centroid is the average position of all matching feature points and represents the geometric center of the point cloud. is the th point cloud of the segment to be plastered. , is the total number of matching feature points, and , is the mean value of the pairs of matching feature points in the segment to be plastered at the position;

[0134] The formula for calculating the centroid of the standard segment point cloud is:

[0135]

[0136] Among them, is the centroid of the standard segment point cloud, is the th point cloud of the standard segment, , , is the mean value of the pairs of matching feature points in the standard segment at the position;

[0137] Decentralize the points in each point cloud (i.e., subtract the centroid) to obtain a new point cloud:

[0138]

[0139] Among them, is the new point cloud of the segment to be plastered after decentralization, is the new point cloud of the standard segment after decentralization;

[0140] The formula for constructing the covariance matrix is:

[0141]

[0142] Among them, is the covariance matrix, and the operator represents the outer product of vectors, is the th point cloud in the standard segment after decentralization;

[0143] The formula for performing singular value decomposition on the covariance matrix is:

[0144]

[0145] Among them, and is an orthogonal matrix, representing the rotation relationship of the point cloud, is a diagonal matrix containing singular values;

[0146] Here, it is necessary to first calculate the product of the transpose of matrix and , that is . By solving the eigenvalue problem, the eigenvalues and eigenvectors of matrix are found, that is, by solving the equation , is the identity matrix, is the eigenvalue. Arrange the eigenvectors of in columns to form matrix , ensuring they are arranged in descending order of eigenvalues; Similarly, calculate the product of matrix and the transpose of , that is . Perform the same eigenvalue decomposition on , that is . Here is the eigenvalue. Arrange the eigenvectors of in columns to form matrix , also arranged in descending order of eigenvalues; Calculate the singular values, which are the square roots of the eigenvalues of or . Arrange these singular values in descending order to form a diagonal matrix , and the elements on its diagonal are , is the eigenvalue.

[0147] Calculate the rotation matrix using the results of singular value decomposition. The formula is as follows:

[0148]

[0149] If the positive definiteness of the rotation matrix is ensured (to avoid non-rotation), it can be adjusted by the following formula:

[0150]

[0151] Calculate the translation vector by the difference between the centroids. The formula is as follows:

[0152]

[0153] Among them, the translation vector represents the translation amount required to move from the centroid of the to-be-plastered surface after rotation to the centroid of the standard segment.

[0154] In this embodiment, the precise pairing through point cloud data specifically includes:

[0155] Collect the point cloud data of the to-be-plastered segment and record it as , is the total number of point cloud data of the segment to be plastered, and the point cloud data of the standard segment is recorded as , is the total number of point cloud data of the standard segment. Taking the point cloud of the standard segment, the point cloud of the segment to be plastered, and the initial transformation matrix as inputs, for each point in the point cloud, find the nearest point pair, calculate the optimal transformation between the point pairs, and let the registered point cloud be . Use the least squares method to optimize the transformation, which is expressed by the formula:

[0156]

[0157] where is the updated transformation matrix calculated according to the point cloud matching result after each iteration, is any point pair, is the current transformation matrix;

[0158] Apply the updated transformation matrix to obtain the registered point cloud to update the point cloud of the segment to be plastered, which is expressed as . Calculate the root mean square error of the current registration result, and the formula is as follows:

[0159]

[0160] where is the root mean square error of the current registration result;

[0161] When , stop the iteration, is the preset threshold.

[0162] In this embodiment, the obtaining of the actual cleaning path specifically includes:

[0163] The formula for calculating the center of the point cloud of the segment to be plastered is as follows:

[0164]

[0165] where is the center of the point cloud of the segment to be plastered, is the th point coordinate of the segment to be plastered, that is ;

[0166] Calculate the transformation between adjacent centers to obtain the cleaning trajectory, and the formula is as follows:

[0167]

[0168] where is the cleaning trajectory, which describes the transformation from the previous geometric center to the current geometric center, is the geometric center of the previous point cloud;

[0169] Generate an actual cleaning path based on the calculated cleaning trajectory to guide the cleaning robot, specifically including:

[0170]

[0171] Among them, is the actual cleaning path, represents the first cleaning trajectory, represents the second cleaning trajectory, represents the th cleaning trajectory, is the number of path points;

[0172] Step 4: Analyze the point cloud data of the standard segment to obtain the height value of each point, then obtain the height value of the trowel at each point, obtain the total distance error between the standard segment and the trowel path, and minimize the contact surface through an optimization algorithm;

[0173] In this embodiment, the obtaining of the total distance error between the standard segment and the trowel path specifically includes:

[0174] Analyze the point cloud data of the standard segment, calculate the height value of each point, and generate a height data set. The formula is as follows:

[0175]

[0176] Among them, is the height value of the th point of the standard segment, is the height data set;

[0177] According to the spatial range of the point cloud, divide it into regular grids. For example, use uniformly and grids. For each grid, count all the points in the grid, calculate the average height of these points, and obtain the height function of the standard segment at any point:

[0178]

[0179] Among them, is the average height of the grid, is the height value of the th point in the grid, is the number of points in the grid;

[0180] After completing the grid division, an interpolation method can be used to generate a continuous height function. Commonly used interpolation methods include bilinear interpolation, spline interpolation, and radial basis function interpolation. Here, taking radial basis function interpolation as an example, the formula is

[0181]

[0182] Among them, is the radial basis function, and

[0183] Regarding the determination of the weight coefficient:

[0184] Suppose there are known data points , among which, is the height value at point . The expression of RBF interpolation can be written as:

[0185]

[0186] Among them, is the distance between point and the known point , is the number of known data points, is the radial basis function, and commonly used ones include Gaussian function, thin plate spline, etc.

[0187] Using the least squares method to determine the weight, first for each known point , evaluate the radial basis function value , is the distance from point to , and construct a matrix:

[0188]

[0189] Among them, represents the matrix;

[0190] Construct the target value vector:

[0191]

[0192] Among them, is the target value vector, containing all known height values;

[0193] Using the least squares method, solve the following equation:

[0194]

[0195] Among them, is the weight vector;

[0196] can be expressed as:

[0197]

[0198] Thereby, each weight coefficient is obtained.

[0199] Install a laser rangefinder sensor on the trowel. During the movement of the trowel, the height data of the trowel position is collected in real time by the sensor. The height data of the trowel monitored in real time is compared with the height data of the standard segment to obtain the total height error:

[0200]

[0201] Among them, is the total distance error, representing the deviation between the trowel path and the target height of the standard segment. The height of the standard segment at position . The height at position on the trowel path. represents the coordinate information . is the number of points included in the error calculation;

[0202] Define the objective function as the total distance error. By optimizing the objective function, the contact surface is minimized, and the least squares method is used for optimization. The height model is set as:

[0203]

[0204] Among them, are the parameters to be optimized;

[0205] Calculate the total distance error as the loss function:

[0206]

[0207] Calculate the partial derivatives of the loss function with respect to the parameters and , and update the parameters according to the gradient descent method:

[0208]

[0209]

[0210] Among them, is the learning rate; initially, the learning rate can be set to 0.01 or 0.001, and then adjusted according to the performance during training.

[0211] Step 5: Collect the image of the segment to be plastered in real time after construction, convert the image into a grayscale image, count the uneven points in the grayscale image, and comprehensively analyze the smoothness of the path, the total distance error, and the proportion of uneven points to obtain the qualification index of the current plastering and judge whether the current plastering is qualified.

[0212] In this embodiment, the judgment on whether the current plastering is qualified specifically includes:

[0213] Collect the image of the currently to-be-plastered segment after construction in real time, and convert the image into a grayscale image.

[0214] The color value of each pixel in the real-time to-be-plastered segment image is expressed as:

[0215]

[0216] where, is the color value of each pixel, represents the intensity of the red channel at the position of ; represents the intensity of the green channel at the position of ; represents the intensity of the blue channel at the position of ;

[0217] Convert the color image into a grayscale image through the grayscale formula. The grayscale formula is:

[0218]

[0219] where, is the grayscale value at the position of , and the value range is between (for 8-bit images);

[0220] Then convert the grayscale image into a binary image. The formula is as follows:

[0221]

[0222] where, is the pixel value of the binary image, 1 represents an uneven point, and 0 represents a flat point, is the set pixel threshold for distinguishing flat and uneven points;

[0223] Count the number of uneven points in the image, and obtain the proportion of uneven points in the entire to-be-plastered point cloud. The formula is as follows:

[0224]

[0225] where, is the proportion of uneven points, is the number of uneven points in the image;

[0226] The formula for obtaining the qualification index is:

[0227]

[0228] where, is the qualification index, is the smoothness of the path of the segment to be plastered, is the maximum value allowed for the total distance error, are the weight coefficients of each item, ,and ;

[0229] The qualified index is compared with the preset qualified threshold. If the qualified index is higher than the preset qualified threshold, it indicates that the construction of the current segment to be plastered is qualified, and the construction can be stopped. If the qualified index is lower than the preset qualified threshold, it indicates that the construction of the current segment to be plastered is unqualified, and the plastering will continue, and the result will be fed back to optimize the point cloud matching step.

[0230] The path smoothness difference term is calculated by comparing the smoothness of the segment to be plastered. and the smoothness of standard segments , reflects whether the path of the plastering process is smooth. The path smoothness directly affects the plastering effect. A smooth path can reduce local unevenness and defects. The value of this item is If , then this item is 1, indicating a perfect match; if If the distance error is large, the item is close to 0, reflecting the degree of unqualified. The distance error item is used to evaluate the distance error between the segment to be plastered and the standard segment. Smaller errors mean that the spatula is closer to the predetermined path during the plastering process, thereby improving the quality of the plastering. Through normalization, it is ensured that this item is also If the error is zero, this item is 1; if the error reaches the maximum value When , this item is 0. Different industries and types of projects may have different requirements for flatness and error. For concrete plastering work, the maximum distance error can be set to 5mm. If the project has higher requirements for flatness, you can also consider reducing the maximum value to 3mm or less. The uneven point ratio item evaluates the plastering quality by the ratio RR of uneven points in the entire point cloud to be plastered. The smaller the proportion of uneven points, the better the plastering quality, and vice versa. Therefore, this item is also If (there is no uneven point), then this item is 1; if The larger the value is, the closer the item is to 0, indicating a decrease in quality. The preset qualified threshold can be set to 0.8, or adjusted according to the requirements for the smoothness of the surface.

[0231] The quantity or proportion of uneven points directly determines the quality of plastering. The main purpose of plastering is to achieve a flat surface. If there are many uneven points, it indicates that the plastering effect is poor. Therefore, it is reasonable to assign the highest weight to this item. The existence of uneven points is the most intuitive feeling for users and is usually the most easily detected problem in quality inspection. The smoothness of the path has an important impact on the uniformity and overall quality of plastering, but compared with the quantity of uneven points, its impact may be relatively indirect. A smooth path can better control the trowel, but if there are many uneven points in the final result, the impact of smoothness will be weakened. Therefore, although the path smoothness is still important, its weight can be set as the second highest. The distance error reflects the closeness of the trowel to the standard path. Although it has an impact on the plastering effect, in practical applications, the impact of distance error may be smaller than that of flatness and uneven points. In actual operation, path error may not directly cause unevenness, but it still needs attention. Therefore, it is reasonable to set the weight of distance error as the lowest. Ensure that , it can make the qualified index value within , which is convenient for direct qualified judgment.

[0232] Please refer to Figure 2 , the present invention further provides a vision guidance system for automatic segment plastering. The vision guidance system for automatic segment plastering is used to implement the above-mentioned vision guidance method for automatic segment plastering, including:

[0233] A standard segment feature acquisition module, which is used to acquire the image and point cloud data of the standard segment. The image is used to obtain the geometric features of the standard segment. By manually operating the robot, the movement path of the trowel is recorded, and the smoothness of the path is calculated;

[0234] An analysis module for the segment to be plastered, which is used to acquire the image and point cloud data of the segment to be plastered, analyze the image of the segment to be plastered to determine the feature points of the segment to be plastered, and perform a preliminary rough matching of the segment to be plastered with the standard segment through the feature points, and then perform a precise matching through the point cloud data;

[0235] A cleaning trajectory generation module, which is used to calculate the center of the point cloud of the segment to be plastered based on the registered point cloud data, obtain the cleaning trajectory through the transformation of adjacent point cloud centers, and then connect the cleaning trajectories to obtain the actual cleaning path;

[0236] A height error analysis module, which is used to analyze the point cloud data of the standard segment to obtain the height value of each point, and then obtain the height value of the trowel at each point, obtain the total distance error between the standard segment and the trowel path, and minimize the contact surface through an optimization algorithm;

[0237] The plastering quality assessment module is used to collect the images of the segment to be plastered in real time after construction, convert the images into grayscale images, count the uneven points in the grayscale images, comprehensively analyze the smoothness of the path, the total distance error, and the proportion of uneven points, so as to obtain the qualified index of the current plastering and determine whether the current plastering is qualified.

[0238] The above formulas are all dimensionless and take their numerical values for calculation. The formulas are obtained by collecting a large amount of data for software simulation to get a formula closest to the actual situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0239] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0240] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. They can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0241] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all should be covered by the protection scope of this application.

Claims

1. An automated visual guidance method for segment plastering, characterized in that The specific steps include: Step 1: Collect the images and point cloud data of the standard segment. The images are used to obtain the geometric features of the standard segment. By manually operating the robot, record the movement path of the trowel and calculate the smoothness of the path. Step 2: Collect the images and point cloud data of the segment to be plastered. Analyze the images of the segment to be plastered to determine the feature points of the segment to be plastered. Make a preliminary rough pairing between the segment to be plastered and the standard segment through the feature points, and then make a precise pairing through the point cloud data. Step 3: Calculate the center of the point cloud of the segment to be plastered based on the registered point cloud. Obtain the cleaning trajectory through the transformation of adjacent point cloud centers, and then concatenate the cleaning trajectories to obtain the actual cleaning path. Step 4: Analyze the point cloud data of the standard segment to obtain the height value of each point, and then obtain the height value of the trowel at each point. Obtain the total distance error between the standard segment and the trowel path, and minimize the contact surface through an optimization algorithm. Step 5: Collect the images after the construction of the segment to be plastered in real time, convert the images into grayscale images, count the uneven points in the grayscale images, and comprehensively analyze the smoothness of the path, the total distance error, and the proportion of uneven points to obtain the qualification index of the current plastering and judge whether the current plastering is qualified.

2. The visual guidance method for automatic surface plastering of segment according to claim 1, characterized in that The collection of the relevant data of the standard segment specifically includes: The geometric features include the coordinate information of the edges and corners of the standard segment. Use the edge detection algorithm to obtain the edge information of the standard segment, and use the corner detection algorithm to determine the corners of the standard segment.

3. The visual guidance method for segment automatic plastering according to claim 1, wherein The calculation of the smoothness of the path specifically includes: By manually operating the robot, record the movement path of the trowel, and store the path coordinates in the form of a point set, with the format as , indicating the position of the first pixel point on the axis and the axis, being the total number of pixel points, and the formula for calculating the smoothness of the standard path is: ； Among them, is the smoothness of the standard path, is the abscissa of the th pixel point in the standard path, the ordinate of the th pixel point in the standard path, is the abscissa of the th pixel point in the standard path, is the ordinate of the th pixel point in the standard path, and .

4. A visual guidance method for automatic plastering of segment, according to claim 1, characterized in that The preliminary rough pairing of the segment to be plastered specifically includes: Use edge detection algorithm and corner detection algorithm to obtain the feature points of the segment to be plastered, that is, the points on the edge and the corner points as feature points. For each feature point , calculate the feature vector , is the dimension of the descriptor. Use the Euclidean distance between descriptors to match feature points, calculate the matched feature point pairs. Let the feature of the segment to be plastered be , and the feature of the standard segment be . The distance formula between feature points is: ； Among them, is the distance between the -th feature point of the segment to be plastered and the -th feature point of the standard segment, is the feature vector of the -th feature point of the segment to be plastered, is the feature vector of the -th feature point of the standard segment; For rough registration, use RANSAC (Random Sample Consensus algorithm) to select inliers and remove outliers. Calculate the rough registration transformation matrix based on the inliers. The formula is as follows: ； Among them, is the transformation matrix for rough registration, is the rotation matrix, is the translation vector.

5. The visual guidance method for automatic segment plastering according to claim 4, characterized in that, The precise pairing through the point cloud data specifically includes: Collect the point cloud data of the segment to be plastered and record it as , is the total number of point cloud data of the segment to be plastered, and the point cloud data of the standard segment is recorded as , is the total number of point cloud data of the standard segment. Taking the point cloud of the standard segment, the point cloud of the segment to be plastered, and the initial transformation matrix as inputs, for each point in the point cloud, find the closest point pair, calculate the optimal transformation between the point pairs, and let the registered point cloud be , and optimize the transformation using the least squares method. The formula is expressed as: ； Among them, is the updated transformation matrix calculated according to the point cloud matching result after each iteration, is an arbitrary point pair, is the current transformation matrix; Apply the updated transformation matrix to obtain the registered point cloud to update the point cloud of the segment to be plastered, expressed as , and calculate the root mean square error of the current registration result. The formula is as follows: ； Among them, is the root mean square error of the current registration result; When the iteration is stopped, being a preset threshold value.

6. The visual guidance method for automatic segment plastering according to claim 5, characterized in that, The obtaining of the actual cleaning path specifically includes: The formula for calculating the center of the point cloud of the segment to be plastered is as follows: ； Among them, is the center of the point cloud of the segment to be plastered, is the coordinate of the th point of the segment to be plastered, that is, ; Calculate the transformation between adjacent centers to obtain the cleaning trajectory. The formula is as follows: ； Among them, is the cleaning trajectory, which describes the transformation from the previous geometric center to the current geometric center, is the geometric center of the previous point cloud; Generate the actual cleaning path according to the calculated cleaning trajectory to guide the cleaning robot, specifically including: ； Among them, is the actual cleaning path, represents the first cleaning trajectory, represents the second cleaning trajectory, represents the th cleaning trajectory, is the number of path points.

7. A visual guidance method for automatic plastering of segment according to claim 1, characterized in that The obtaining of the total distance error between the standard segment and the trowel path specifically includes: Analyze the point cloud data of the standard segment, calculate the height value of each point, and generate a height data set. The formula is as follows: ； Among them, is the height value of the th point of the standard segment, is the height data set; According to the spatial range of the point cloud, divide it into regular grids. For each grid, count all the points in the grid and calculate the average height of these points to obtain the height function of the standard segment at any point: ； Among them, is the average height of the grid, is the height value of the th point in the grid, is the number of points in the grid; After completing the grid division, an interpolation method can be used to generate a continuous height function. The formula is ； Among them, is a radial basis function, is a weight coefficient; Install a sensor on the trowel, and collect the height data of the trowel position in real time during the movement of the trowel through the sensor. Compare the real-time monitored trowel height data with the height data of the standard segment to obtain the total height error: ； Among them, is the total distance error, the height of the standard segment at position , the height of the position on the trowel path at position , represents the coordinate information , is the number of points included in the error calculation; Define the objective function as the total distance error, minimize the contact surface by optimizing the objective function, and use the least squares method for optimization. Set the height model as: ； Among them, is the parameter to be optimized; Calculate the total distance error as the loss function: ； Calculate the partial derivatives of the loss function with respect to the parameters and , and update the parameters according to the gradient descent method: ； ； Among them, is the learning rate.

8. A visual guidance method for automatic plastering of segment according to claim 1, characterized in that, The judgment of whether the current plastering is qualified specifically includes: Collect the image of the currently to-be-plastered segment after construction in real time, convert the image to a grayscale image, and then convert the grayscale image to a binary image. The formula is as follows: ； Among them, is the pixel value of the binary image, is the set pixel threshold; Count the number of uneven points in the image and obtain the proportion of uneven points in the entire to-be-plastered point cloud. The formula is as follows: ； Among them, is the proportion of uneven points, is the number of uneven points in the image; The formula for obtaining the qualification index is: ； Among them, is the qualified index, is the smoothness of the path of the segment to be plastered, is the maximum value allowed for the total distance error, are the weight coefficients of each item respectively, , and ; Compare the qualification index with a preset qualification threshold. If the qualification index is higher than the preset qualification threshold, it indicates that the construction of the currently to-be-plastered segment is in a qualified state, and the construction can be stopped. If the qualification index is lower than the preset qualification threshold, it indicates that the construction of the currently to-be-plastered segment is in an unqualified state, and the plastering continues, and the result is fed back to optimize the point cloud pairing step.

9. An automated segment plastering vision guidance system, characterized in that, The above-mentioned segment automatic plastering vision guidance system is used to implement the segment automatic plastering vision guidance method described in any one of claims 1-8, including: A standard segment feature acquisition module, which is used to collect the image and point cloud data of the standard segment. The image is used to obtain the geometric features of the standard segment. By manually operating the robot, record the movement path of the trowel and calculate the smoothness of the path; A to-be-plastered segment analysis module, which is used to collect the image and point cloud data of the to-be-plastered segment, analyze the image of the to-be-plastered segment to determine the feature points of the to-be-plastered segment, and perform a preliminary rough pairing of the to-be-plastered segment and the standard segment through the feature points, and then perform precise pairing through the point cloud data; A cleaning trajectory generation module, which is used to calculate the center of the to-be-plastered segment point cloud based on the registered point cloud, obtain the cleaning trajectory through the transformation of adjacent point cloud centers, and then concatenate the cleaning trajectories to obtain the actual cleaning path; A height error analysis module, which is used to analyze the point cloud data of the standard segment to obtain the height value of each point, then obtain the height value of the trowel at each point, obtain the total distance error between the standard segment and the trowel path, and minimize the contact surface through an optimization algorithm; A plastering quality evaluation module, which is used to collect the image of the to-be-plastered segment after construction in real time, convert the image to a grayscale image, count the uneven points in the grayscale image, comprehensively analyze the smoothness of the path, the total distance error, and the proportion of uneven points to obtain the qualification index of the current plastering and judge whether the current plastering is qualified.

Images