Surface condition detection and polishing robot path planning method for surface paint layer of special equipment

Abstract

The application discloses a kind of paint surface condition detection and polishing robot path planning methods for special equipment surface paint layer, the point cloud data obtained by three-dimensional structured light camera is processed, the spatial position characteristics of workpiece are obtained, the machining path required for polishing operation is calculated by computer robot, and the path parameters are optimized by analyzing the cutting amount of polishing head, so that more uniform surface removal rate is obtained;The method of deep learning is used to detect and analyze the result of robot polishing operation, and based on the visual characteristics of the workpiece before and after grinding, a hybrid layer thickness estimation algorithm is designed, and the grinding and polishing paint surface dataset is labeled based on the algorithm, and the path is optimized based on this benchmark;The path planning algorithm obtains the feedback of the change of paint surface, replans the path, and the robot performs grinding processing based on the path, to achieve the closed loop of the whole system.

Description

Technical Field

[0001] This invention relates to the field of paint coating repair technology, and in particular to a method for detecting the paint condition of the surface paint layer of special equipment and planning the path of a polishing robot. Background Technology

[0002] With the development of materials science, coating technology has penetrated into every corner of human society. To ensure the function and service life of the coating, high-end equipment often uses multiple layers of coating materials to achieve better protection. Repairing the coating before it completely fails can greatly extend the service life of the workpiece. The mainstream paint removal methods on the market include mechanical grinding, chemical solvents, shot blasting, and laser paint removal.

[0003] For the repair of multi-layer coatings on high-end equipment, due to the complexity of the process, experienced polishing workers are often relied upon to judge the polishing allowance based on experience, removing the coating layer by layer in small amounts multiple times to ensure the integrity of the normal paint layer and substrate. Automated processing is difficult, polishing efficiency is low, and manual polishing is physically demanding and prone to health problems. Current automation methods cannot completely replace manual labor, mainly because the uncertainty of the paint layer thickness on the workpiece surface caused by the spraying process means that offline trajectory planning methods based on polishing removal models can only obtain the amount of paint removed by polishing but cannot predict the amount of paint residue. Three-dimensional reconstruction methods are limited by reconstruction accuracy and cannot reflect changes in the paint layer; the trajectory can only reflect the shape of the workpiece but not the distribution of paint layer thickness. For the detection of paint residue, there is a lack of large-scale, real-time methods to meet the needs of trajectory planning.

[0004] In recent years, point cloud-based detection methods for detecting paint surface defects have emerged. These methods, based on the existence of a theoretical CAD model of the workpiece, are mainly divided into two types. One type reconstructs a CAD model from segmented point clouds using methods such as registration, and compares it with the theoretical CAD model to determine the location of defects such as burrs and dents. For example, point clouds are generated from multiple preset workpiece CAD models, and each is matched with the point cloud obtained from the sensor using an iterative nearest-point matching method. The most suitable model is selected, and subsequent trajectory planning is performed based on the matching results. The other approach defines common defect point cloud morphological features and directly identifies the defect point clouds. For example, network architectures that directly extract features from point cloud data can perform point cloud classification and segmentation tasks; and self-attention mechanisms are introduced into point cloud deep learning networks to improve network performance. The advantage of the point cloud method is that it can make full use of the point cloud information acquired by the 3D camera without the need to add additional sensors; however, the disadvantage is that the method is limited by the accuracy of the input point cloud itself. Currently, the accuracy of the area array structured light camera on the market is generally above 100μm. If further improvement of accuracy is required, a specially customized camera is needed and the camera's field of view is sacrificed, which makes it difficult to meet the needs of large-scale and high-precision coating defect detection. Summary of the Invention

[0005] This invention aims to at least partially solve one of the technical problems in related technologies. Therefore, one objective of this invention is to propose a method for detecting the paint condition of the surface paint layer on special equipment and for planning the path of a polishing robot. This method acquires the surface morphology features of the workpiece and, based on these features, plans the end-effector processing path and trajectory of the robot, thereby minimizing damage to the substrate while achieving the required paint removal rate on the workpiece surface.

[0006] According to the present invention, a method for detecting the paint condition of the surface paint layer of special equipment and planning the path of a polishing robot is proposed. The method steps are as follows:

[0007] S1: The point cloud and image of the workpiece are acquired by a 3D structured light camera. The point cloud is subjected to voxel downsampling and outlier filtering. The environmental points and test bench plane points are removed in sequence. Then, the workpiece point cloud is extracted by region growing clustering.

[0008] S2: Perform principal component analysis on the workpiece point cloud to construct a local coordinate system. Transform the workpiece point cloud to the origin at the center coordinate using a pose transformation matrix. Align the three principal component directions with the x, y, and z axes respectively to establish an AABB-type bounding box. On the xy plane of the local coordinate system, take points in the second principal component direction with the first principal component direction as the unfolding direction and at a set interval to generate a raster-type candidate path template and perform path sampling.

[0009] S3: Perform neighborhood point search at each sampling position, and use the corresponding coordinates of the neighborhood points in the point cloud to backfill the z coordinate of the sampling position according to the inverse proportional difference. Connect each path point and transform the path points in the local coordinate system back to the camera coordinate system through the pose transformation matrix. Determine the x-axis orientation of the robot end effector by the projection of the direction vector from the current path point to the next path point on the normal vector tangent plane, and use the Rodriguez formula to obtain the end effector pose matrix in the robot coordinate system.

[0010] S4: The Preston equation is used to describe the cutting amount of the grinding and polishing process per unit time. The instantaneous processing amount is estimated and accumulated at the sampling points of the grating path to form the path processing amount field.

[0011] S5: Segment and thickness regression of camera images. A deep learning network is used to classify the input image into three categories: background, processed, and unprocessed, and to estimate the paint layer thickness. The thickness estimate is mapped to the workpiece point cloud through the mapping relationship between the image and the point cloud, and the remaining amount of paint layer after path execution is calculated.

[0012] S6: Based on the processing volume estimation in step S4, the grating path is combined and optimized: a series of long straight paths are generated at smaller intervals, and any number of trajectories are selected from them and iterated repeatedly through the optimization algorithm to make the overall removal rate of the workpiece surface reach the optimal level. The grating interval is adjusted according to the grinding and polishing removal model to meet the theoretical optimal interval.

[0013] S7: Perform processing according to the optimization results of step S6, collect the processed image and feed the paint layer thickness estimate back to the path planning, repeat the mapping and optimization steps, thereby realizing the closed loop of detection and planning.

[0014] Preferably, in step S1:

[0015] According to step distance Divide the space occupied by the point cloud into several cube-shaped voxels:

[0016]

[0017] in, This represents the set of points in each voxel. Represents the first point in the point cloud One point;

[0018] Calculate the point set in each voxel Its center of gravity position:

[0019]

[0020] in, It is a point set The number of points in it;

[0021] When there is only one point in a voxel, the position of that point remains unchanged; when there are multiple points in a voxel, the calculated centroid point replaces the point set. This reduces the number of points in the point cloud, achieving the purpose of downsampling.

[0022] Point cloud collection For any point Its neighborhood point set :

[0023]

[0024] in, The preset search radius;

[0025] For each point Statistical analysis of its neighborhood point set The number of midpoints, and compared with a preset threshold. Compare, if it is less than ,but These are outliers and need to be removed from the point cloud data;

[0026] Apply Gaussian filtering to the workpiece point cloud and calculate the weights of neighboring points relative to that point. :

[0027]

[0028]

[0029] in, This represents the standard deviation of a Gaussian distribution.

[0030] The camera is fixed in place, with the eye outside the hand, for any point within the working area. By using the installation position conditions of the polishing robot, the three-dimensional position conditions of the points in the working area are restricted, thereby filtering out points outside the working area. The test bench plane point cloud is filtered out using the plane matching method. The fitting plane parameters are calculated by using the sampling consistency method on the point cloud. For each point in the point cloud, the distance from the point to the reference plane is calculated. The point cloud within a certain distance is filtered out, thus removing the test bench plane.

[0031] Preferably, in step S2:

[0032] For workpiece point cloud Principal component analysis was performed to obtain the centroid and three principal component directions of the workpiece point cloud, and the pose transformation matrix was calculated. The point cloud is then transformed to a position where the centroid is at the origin and the directions of the three principal components are aligned with the x, y, and z axes, respectively, resulting in the point cloud. Constructing point clouds AABB-type bounding boxes are used to obtain their maximum and minimum coordinates. In the xy-plane of the local coordinate system of the point cloud, the x-axis, which is chosen as the first principal component of the point cloud, is selected as the unfolding direction of the first path. The total length is the length of the bounding box, and the unfolding is performed according to intervals. Take a series of points, spaced along the y-axis. Select points to obtain the raster path template.

[0033] Preferably, in step S3:

[0034] The path points are projected onto the point cloud, and points with x and y coordinates within a certain range are searched. The z coordinate of the path points is calculated using these neighboring points. The neighborhood search is performed by constructing a kd-tree, which is a temporary point cloud. By using point clouds The z-coordinate is set to zero and projected onto the xy plane to obtain the point cloud. Construct a kd-tree and perform a radius search using the kd-tree to find the range. Points within, due to and The points in the cloud are mapped one-to-one. For a neighboring point found by searching the path points, its representation in the point cloud is determined. The coordinates of the corresponding points are used to calculate the coordinates of any path point using the inverse proportional difference formula. In point cloud z-coordinate in:

[0035]

[0036] in, Indicates at the waypoint search radius Within, the total number of neighboring points found;

[0037] Connect the beginning and end of each trajectory point, and then use the pose transformation matrix. Transform the path points in the local coordinate system back to the camera coordinate system to obtain the workpiece grinding and polishing path in the camera coordinate system.

[0038] The direction of the minimum eigenvalue is calculated using principal component analysis, which is the normal vector direction of the local point cloud fitting plane. The x-axis of the polishing robot's end effector is aligned with the forward direction, the attitude matrix is ​​fixed, and the current path point is calculated. To the next path point The direction vector is projected onto the direction of the tangent plane of the normal vector:

[0039]

[0040] in, , Represents the normal vector;

[0041] Through unit vector Calculate the current x-axis orientation of the polishing robot's end effector. Calculated using the Rodriguez formula arrive rotation matrix This allows us to obtain the end effector pose matrix in the robot coordinate system. .

[0042] Preferably, in step S4:

[0043] The Preston equation, an empirical formula, is used to describe the amount of material removed during grinding and polishing per unit time.

[0044]

[0045] in, For pressure, For grinding speed, This is Preston's constant;

[0046] by With the center of the circle, For radius, Angular velocity, instantaneous linear velocity of the polishing disc of the polishing robot. for: ;

[0047] Therefore, instantaneous processing volume and distance Relationship:

[0048]

[0049] For the estimation of the processing amount of the trajectory formed between two adjacent path points, a series of sampling points are interpolated between the two adjacent path points, and the instantaneous processing amount is estimated for each sampling point. Then the total processing amount of the entire long straight grating path is approximately equivalent to the sum of the instantaneous processing amounts of all sampling points.

[0050] At a constant distance Interpolating between two adjacent path points, we have Where r is the equivalent radius of the grinding disc, for each sampling point In planar point clouds Using radius Sampling is performed to obtain a neighborhood point cloud; subsequently, the point cloud is... Find the corresponding point and obtain the neighborhood point cloud. Based on the normal vector at the sampling point Calculate each neighboring point The distance d from the sampling point in the tangential plane direction siDistance z in the direction of normal si

[0051]

[0052]

[0053] After calculating the instantaneous processing amount of each neighboring point, it is mapped onto the original point cloud. For each instantaneous processing quantity, divide by the number of sampling points within the period to ensure that the processing quantity estimate for a single long straight trajectory remains within the range of (0,1), i.e.

[0054]

[0055] in, This represents the total number of sampling points for a single trajectory. Let this be a constant used for normalization;

[0056] Definition of the first All sampling points in the grating path If we form a point set S, then any point in the point cloud... In the The equivalent processing amount after processing a grating path is:

[0057]

[0058] in, ;

[0059] Create a linked list to record all modified points in the corresponding trajectory, which will accelerate the subsequent calculation of loss values.

[0060] Preferably, in step S5:

[0061] For a single set of grinding data ,in It is the source image. It is single-channel thickness information, its initial state For the rough coating layer, final state As a substrate layer, Indicates the height of the image. Indicates the width of the image;

[0062] Since the illumination and workpiece position remain constant throughout the entire grinding process, assuming any... All are made by and The result of a linear combination, that is, for any pixel on All of them are:

[0063]

[0064] in, Characterizes the proportions of the three channels;

[0065] The method of projecting in RGB space is used to mix the information of the three channels for any pixel. In RGB space, establish and vector Then Projected to The projected distance obtained above is the required thickness estimate:

[0066]

[0067] To balance segmentation and paint layer thickness estimation, the deep learning network uses a comprehensive loss function. The overall loss is calculated, which consists of two core components: segmentation loss. and thickness prediction loss Segmentation loss Focus on the overall image and category edges; prediction loss Focusing solely on the overall surface loss of the workpiece, the following is an analysis:

[0068]

[0069] in, , These are hyperparameters for the two losses, used to adjust the ratio of the two weights;

[0070] For segmentation at different scales, a predicted value will be output separately; this is the segmentation loss. Further divided into three identical parts:

[0071]

[0072]

[0073] in, , , Three sets of hyperparameters for the loss, ranging from large to small scale, are used to adjust the model's attention to targets at different scales. To upsample the network mask output to its original size, One-hot encoded labels for images, Image size, For cross-entropy loss, represent The weights under the coordinates are used in the classification loss to strengthen the loss of edge regions;

[0074] In the paint layer thickness decoder, the mean square error loss is used as the thickness prediction loss:

[0075]

[0076] in, The output is the network thickness prediction value upsampled to the original size. For the thickness label input, represent The weights of the coordinate system are used to shield the influence of the background region on the thickness prediction loss function.

[0077] Preferably, in step S6:

[0078] Based on the prediction results of deep learning, a grayscale image representing the paint layer thickness estimate is obtained, where the grayscale value represents the paint layer thickness at that point. The thickness of the original paint layer and the rough-processed paint layer is 1, and the thickness of the non-workpiece area and the substrate layer is 0, which do not need to be processed. The remaining grayscale values ​​correspond to the paint layer thickness estimate of the mixed layer paint surface. By using the camera's intrinsic and extrinsic parameter matrices, the specific coordinates of any point in the point cloud in the pixel coordinate system are calculated, thereby mapping the grayscale image to the point cloud to obtain a point cloud with paint layer thickness.

[0079] In addition to the desired pose, an additional desired force is introduced for the path points. This force is adjusted based on the machining allowance in the local area of ​​the path points to control the cutting amount at different positions on the path. Furthermore, the estimated paint layer thickness is incorporated into the loss function calculation. By changing the calculation methods of the loss function and the desired force, different processing strategies can be implemented, thereby improving processing efficiency and reducing damage to the substrate.

[0080] Path point expectation power:

[0081] By adjusting the expected force at the path points, the amount of machining along the path is changed to adapt to the actual paint layer thickness variations at different locations on the workpiece surface, using a scaling factor. To represent the expected grinding force at the path point With the maximum expected grinding force in the path The relationship between them is ;

[0082] The applied grinding force changes linearly with the remaining paint thickness in the area traversed by the grinding disc, as shown by the point cloud within the area covered by the grinding disc at the sampling point. Calculate the average remaining paint layer thickness of the point cloud, and use it as a proportionality coefficient for the desired grinding force:

[0083]

[0084] in, For local point clouds The number of points in the middle, For point The paint layer thickness is estimated by statistically analyzing the minimum equivalent paint layer thickness in a local area, which is then used as a proportionality coefficient for the desired grinding force. ,Right now:

[0085]

[0086] Estimation of machining allowance along the path:

[0087] As the grinding force changes, the machining quantity per single path also changes accordingly:

[0088]

[0089] The estimated machining allowance at each point is expressed as follows:

[0090]

[0091] in, This is an empirical parameter that reflects the mapping relationship between the theoretical cutting amount and the estimated paint thickness.

[0092] Combinatorial optimization loss function:

[0093]

[0094]

[0095] in, For point clouds Mean value of the estimated intermediate coat allowance. It is a very small non-zero constant. For the number of paths, , , , , These are empirical parameters that reflect the mapping relationship between theoretical cutting amount and paint layer thickness estimation. When the paint surface is in its original state, we have... The expected force of each trajectory point is .

[0096] The beneficial effects of this invention are:

[0097] (1) Point cloud-based grating grinding and polishing path planning: Based on the point cloud acquired by the 3D structured light camera, the workpiece point cloud is obtained through downsampling, filtering, and clustering. A local point cloud extraction algorithm based on image mapping and local curvature is designed to generate a series of grating paths on the workpiece surface, and the pose information of the path points is extracted through the local point cloud normal vector. The instantaneous grinding model is introduced into the curved surface point cloud to calculate the influence of the grating path on the overall processing amount of the workpiece surface. The optimal path combination is selected from a series of fine grating paths to achieve the optimal overall grinding removal amount of the workpiece.

[0098] (2) Deep Learning-Based Paint Thickness Estimation Network: A hybrid layer thickness estimation algorithm was designed based on the visual features of the workpiece before and after grinding. A grinding and polishing paint surface dataset was created, and paint thickness estimation annotations were performed. Based on this, a synthetic dataset was created by simulating the grinding process using a rendering engine. The existing semantic segmentation network was improved to adapt to the paint thickness estimation task. Experiments verified the network's generalization effect on the test set. The synthetic dataset effectively improved the network's accuracy and generalization ability. Compared with the real paint thickness, the network can reflect the changing trend of the real paint thickness to a certain extent, verifying the feasibility of the method.

[0099] (3) Path planning based on paint thickness feedback: The paint thickness estimated by the deep learning network is mapped to the workpiece point cloud through the camera intrinsic parameter matrix. Based on the paint thickness estimation of the local point cloud in the path, the expected grinding force required for the path point is calculated. Based on the paint thickness estimation and the grating path processing amount estimation, the paint residue after the grating path is estimated. Based on this, the path combination optimization is performed to generate the processing path. The experiment verifies that this method can effectively plan the processing path and control the robot to process the area with thicker paint in a targeted manner, which significantly improves the processing efficiency. Attached Figure Description

[0100] Figure 1 This is a flowchart of a method for detecting the paint condition of special equipment surfaces and planning the path of a polishing robot, as proposed in this invention.

[0101] Figure 2 This is a flowchart of the point cloud polishing path planning proposed in this invention;

[0102] Figure 3 This is a comparison image of the environmental point cloud before and after removal proposed in this invention (wherein) Figure 3 'a' represents the state before removal. Figure 3 b represents the result after removal);

[0103] Figure 4 This is a flowchart of the region growing method proposed in this invention;

[0104] Figure 5This invention presents the workpiece surface point cloud and trajectory point normal vector map (wherein) Figure 5 'a' represents the point cloud on the workpiece surface, where Figure 5 b is the normal vector of the trajectory point;

[0105] Figure 6 This is a flowchart of the grating trajectory optimization proposed in this invention;

[0106] Figure 7 This is the overall flowchart of the path planning based on paint layer thickness estimation proposed in this invention;

[0107] Figure 8 This is the thickness estimation thermogram proposed in this invention;

[0108] Figure 9 This is a foreground information annotation diagram proposed in this invention. Detailed Implementation

[0109] Reference Figure 1 A method for detecting the paint condition of the surface paint layer of special equipment and for planning the path of a polishing robot, the method comprising the following steps:

[0110] S1: The point cloud and image of the workpiece are acquired by a 3D structured light camera. The point cloud is subjected to voxel downsampling and outlier filtering. The environmental points and test bench plane points are removed in sequence. Then, the workpiece point cloud is extracted by region growing clustering.

[0111] (1) Point cloud processing

[0112] The flowchart of point cloud polishing path planning is as follows: Figure 2 As shown, the point cloud needs to be processed before performing path planning to reduce the number of points in the point cloud and extract key point cloud information.

[0113] Voxel downsampling is used in the point cloud processing workflow. First, based on the step size... The space containing the point cloud is divided into several cube-shaped voxels, and the point cloud within the voxel range is extracted according to the following model.

[0114]

[0115] in, This represents the set of points in each voxel. Represents the first point in the point cloud One point;

[0116] Subsequently, the point set in each voxel Calculate its center of gravity The location is as follows:

[0117]

[0118] in, It is a point set The number of points in it.

[0119] When there is only one point in a voxel, the position of that point remains unchanged; when there are multiple points in a voxel, the newly calculated centroid point replaces the point set. This reduces the number of points in the point cloud, achieving the purpose of downsampling.

[0120] Outlier filtering is performed in the point cloud processing workflow using radius detection. The core idea is to identify and remove outliers through local point density analysis. Given a point cloud set... For any point The algorithm first calculates its neighborhood point set. :

[0121]

[0122] in, This is the preset search radius.

[0123] Then, for each point Statistical analysis of its neighborhood point set The number of midpoints, and compared with a preset threshold. Compare, if it is less than ,but These are outliers and need to be removed from the point cloud data.

[0124] In the point cloud processing workflow, Gaussian filtering is applied to the point cloud of the workpiece only after the workpiece is extracted. The weights of neighboring points relative to the current point are calculated, and a weighted average is performed to reduce the variation of the point cloud within a local range, thereby reducing the random error of the calculated normal vector.

[0125]

[0126]

[0127] in, This represents the standard deviation of a Gaussian distribution.

[0128] (2) Extraction from the processing area

[0129] Because the camera is fixed in a position where the eye is outside the hand, the surrounding environment of the workpiece can be removed using known environmental information. For any point within the working area... By using the location conditions of the robot installation, three-dimensional positional constraints can be imposed on points within the work area, thereby filtering out points outside the work area.

[0130] After removing the environmental background, the test bench plane point cloud is filtered out using a plane matching method. This method first calculates the fitting plane parameters by using the sampling consistency method on the point cloud. For each point in the point cloud, the distance from that point to the reference plane is calculated. The point cloud within a certain distance is then filtered out, thus removing the test bench plane.

[0131] The core idea of ​​the sample consensus algorithm is to randomly sample a subset from the point cloud and estimate the model parameters of that subset. For the case of a plane, three points are selected from the point cloud to calculate the plane equation.

[0132]

[0133] The algorithm then calculates the deviation of all samples in the subset from the model. If the deviation is less than a given threshold, the point is designated as an inlier. The number of inliers is counted for each sampling, and this process is repeated until all iterations are exhausted or the error rate reaches a given value. The algorithm ultimately outputs the model parameters with the highest number of inliers. After obtaining the plane equation, points within a certain thickness *d* of the plane are removed by comparing distances, resulting in the final point cloud. as follows:

[0134]

[0135] The before-and-after comparison of environmental point cloud removal is as follows: Figure 3 As shown.

[0136] After removing the environment and the worktable plane, only the workpiece and fixture remain in the camera's field of view, with the workpiece being significantly larger than the fixture. The remaining point cloud is then clustered using region growing. The point cloud with the largest number of elements in each cluster is selected to obtain the workpiece point cloud. The flowchart is as follows. Figure 4 As shown. The basic idea of ​​region growing is to perform a neighborhood search based on a seed point, and add similar neighboring points to the same point cloud group according to certain rules. The clustering rules are not limited to various attributes of neighboring points, including distance, color, feature descriptors, etc.

[0137] S2: Perform principal component analysis on the workpiece point cloud to construct a local coordinate system. Transform the workpiece point cloud to the origin at the center coordinate using a pose transformation matrix. Align the three principal component directions with the x, y, and z axes respectively to establish an AABB-type bounding box. On the xy plane of the local coordinate system, take points in the second principal component direction with the first principal component direction as the unfolding direction and at a set interval to generate a raster-type candidate path template and perform path sampling.

[0138] (3) Grating path

[0139] A raster-based path point generation method is used to generate the polishing path for the workpiece point cloud. Principal component analysis was performed to obtain the centroid and three principal component directions of the workpiece point cloud, and the pose transformation matrix was calculated. The point cloud is transformed to a position where the center coordinates are the origin and the three principal component directions are aligned with the x, y, and z axes, respectively, to obtain the point cloud. Subsequently, a point cloud was constructed. AABB-type bounding boxes are used to obtain their maximum and minimum coordinates. In the xy-plane of the point cloud's local coordinate system, to maximize the length of a single processing path, the x-axis, which is the first principal component of the point cloud, is chosen as the unfolding direction of the first path. The total length is the length of the bounding box, and the paths are unfolded according to intervals. Take a series of points. Arrange them at intervals along the y-axis. Select points to obtain the raster path template.

[0140] S3: Perform neighborhood point search at each sampling position, use the corresponding coordinates of the neighborhood points in the point cloud to backfill the z coordinate of the sampling position according to the inverse proportional difference, connect each path point and transform the path points in the local coordinate system back to the camera coordinate system through the pose transformation matrix; determine the x-axis orientation of the robot end effector by the projection of the direction vector from the current path point to the next path point on the normal vector tangent plane, and use the Rodriguez formula to obtain the end effector pose matrix in the robot coordinate system;

[0141] Projecting path points onto the point cloud involves searching for points in the point cloud whose x and y coordinates fall within a certain range, and then calculating the z-coordinate of the path point using these neighboring points. Neighborhood search is often performed by constructing a kd-tree, whose data structure divides the point cloud space into several subspaces according to the x, y, and z coordinates, improving search efficiency through binary search. Here, we construct a temporary point cloud. The point cloud is obtained by... The z-coordinate is set to zero and projected onto the xy plane to obtain the point cloud. Construct a kd-tree and perform a radius search using the kd-tree to find the range. Points within.

[0142] because and The points in the cloud correspond one-to-one. For a neighboring point found by searching the path points, its representation in the point cloud can be found. The coordinates of the corresponding points are used to calculate the coordinates of any path point using the inverse proportional difference formula. In point cloud The z-coordinate in the image. Specifically:

[0143]

[0144] in, Indicates at the waypoint search radius Within, the total number of neighboring points found;

[0145] Then, connect the beginning and end of each trajectory point, and use the pose transformation matrix. By transforming the path points in the local coordinate system back to the camera coordinate system, the workpiece grinding and polishing path in the camera coordinate system can be obtained.

[0146] The direction of the minimum eigenvalue is calculated using principal component analysis, which is the direction of the normal vector of the local point cloud fitting plane. Figure 5 As shown, this method aligns the x-axis of the end effector to the forward direction using a fixed attitude matrix. Specifically, this involves calculating the current path point... To the next path point The direction vector is projected onto the direction of the tangent plane of the normal vector:

[0147]

[0148] in, , Represents the normal vector;

[0149] Through unit vector Calculate the current orientation of the robot's end effector along the x-axis. Calculated using the Rodriguez formula arrive rotation matrix This allows us to obtain the end effector pose matrix in the robot coordinate system. .

[0150] S4: The Preston equation is used to describe the cutting amount of the grinding and polishing process per unit time. The instantaneous processing amount is estimated and accumulated at the sampling points of the grating path to form the path processing amount field.

[0151] (4) Calculate the amount of machining generated during the machining of the grating path.

[0152] Based on the grinding principle of the grinding and polishing actuator, the grinding effect of the grating path is analyzed. By modifying the interval between the two sets of gratings, a smaller surface roughness can be achieved on the workpiece after grinding. For grinding, the empirical formula Preston equation is often used to describe the cutting amount per unit time in grinding and polishing:

[0153]

[0154] in, For pressure, For grinding speed, It is Preston's constant;

[0155] Due to the normal rotational speed of the actuator during grinding and polishing... Much greater than the movement speed of the robot's end effector When considering grinding, the robot's movement speed is negligible, and only the instantaneous speed on the grinding wheel is considered. With the center of the circle, For radius, Let be the angular velocity, and its instantaneous linear velocity. for: .

[0156] From this, we can obtain the instantaneous processing volume and distance. Relationship:

[0157]

[0158] For the grating path, the grating spacing is adjusted to meet the theoretically optimal spacing of the grinding and polishing removal model, so that the workpiece as a whole achieves a grinding and polishing effect with relatively uniform surface roughness. A grating trajectory optimization method based on optimization problem thinking is used, such as... Figure 6 As shown, first, a series of long straight paths are generated at small intervals; second, an arbitrary number of paths are selected from them; finally, the paint removal rate on the workpiece surface is calculated. Through repeated iterations of the optimization algorithm, a set of long straight grating paths can be obtained, so that the overall paint removal rate on the workpiece surface reaches the optimal level.

[0159] For the estimation of the processing amount of the trajectory formed between two adjacent path points, a series of sampling points are interpolated between the two adjacent path points. The instantaneous processing amount is estimated for each sampling point. Then, the total processing amount of the entire long straight grating path is approximately equivalent to the sum of the instantaneous processing amounts of all sampling points.

[0160] Specifically, at a constant distance Interpolating between two adjacent path points, we have , where r is the equivalent radius of the grinding disc. For each sampling point In planar point clouds Using radius Sampling is performed to obtain a neighborhood point cloud; subsequently, the point cloud is... Find the corresponding point and obtain the neighborhood point cloud. According to the normal vector Calculate each neighboring point The distance d from the sampling point in the tangential plane direction si Distance in the direction of normal .

[0161]

[0162]

[0163] After calculating the instantaneous processing amount of each neighboring point, it is mapped onto the original point cloud. For each instantaneous processing quantity, divide by the number of sampling points within the period to ensure that the processing quantity estimate for a single long straight trajectory remains within the range of (0,1), i.e.

[0164]

[0165] in It is the total number of sampling points for a single trajectory. It is a constant used for normalization.

[0166] Definition of the first All sampling points in the grating path If we form a point set S, then any point in the point cloud... In the The equivalent processing amount after processing a grating path is:

[0167]

[0168] in, ;

[0169] When the point cloud of the workpiece processing area and the size of the grinding disc differ significantly, a single long straight grating path will only cover a portion of the point cloud area. Therefore, after the calculation of a single grating path is completed, most areas in the array still retain their initial values. Thus, a linked list is established to record all modified points in the trajectory to accelerate the subsequent calculation of loss values.

[0170] S5: Segment and thickness regression of camera images. A deep learning network is used to classify the input image into three categories: background, processed, and unprocessed, and to estimate the paint layer thickness. The thickness estimate is mapped to the workpiece point cloud through the mapping relationship between the image and the point cloud, and the remaining amount of paint layer after path execution is calculated.

[0171] (5) Design of paint layer thickness estimation algorithm

[0172] Before further processing, it is necessary to inspect the paint surface condition before and after processing. The paint layer thickness is mapped onto the point cloud using the image-point cloud mapping relationship. A path planning algorithm based on paint layer thickness estimation feedback is designed to calculate the remaining paint layer after path execution. Based on this, path optimization is performed, ultimately achieving a closed loop between detection and planning. Figure 7 As shown.

[0173] This study focuses on the thickness of the mixed layer as the primary object of paint layer thickness estimation, and designs a paint layer thickness estimation algorithm and corresponding control strategy. Specifically, for a single set of grinding data... ,in It is the source image. It is single-channel thickness information, its initial state For the rough coating layer, final state As a substrate layer, Indicates the height of the image. Indicates the width of the image.

[0174] Since the illumination and workpiece position remain constant throughout the entire grinding process, let's assume that any... All are made by and The result of a linear combination, that is, for any pixel on All of them are:

[0175]

[0176] in It represents the ratio of the three channels.

[0177] The information from the three channels is blended using a method that projects in RGB space. For any pixel... In RGB space, establish and vector Then Projected to The projected distance obtained is the required thickness estimate.

[0178]

[0179] Based on this method, the paint film thickness estimate for each pixel on the workpiece surface can be obtained, and the paint film thickness estimate can be transformed into a heat map, such as... Figure 8 As shown;

[0180] (6) Deep learning network design

[0181] (5) proposes a method for estimating paint layer thickness, but this method cannot be directly used for grinding path planning during the processing because the overall visual characteristics of the substrate layer under the lighting conditions at that position cannot be predicted before the processing is completed. Therefore, a deep learning method is introduced. By learning the known visual characteristics of grinding processing, the algorithm can fit the paint layer thickness estimate of each position on the workpiece surface at the current moment based solely on the workpiece surface state at past moments.

[0182] This step involves designing a deep learning network that, based on the input image, distinguishes between the background, processed areas, and unprocessed areas on the workpiece, and estimates the paint layer thickness. Since both functions use the same image input, and the segmentation category information and paint layer thickness are coupled, a common encoder can be used to encode the image. For the decoding part, this is equivalent to designing a separate module based on the semantic segmentation task, enabling the calculation of an independent prediction score within a specific semantic classification, which serves as the paint layer thickness output. Specifically:

[0183] 1) Main modules of a semantic segmentation network:

[0184] The network structure will be improved based on SAM2 and SAM2UNet to enable it to perform the task of paint layer thickness detection.

[0185] ① Image Encoder: The image encoder in SAM2UNet consists of four encoder modules. Each encoder module is composed of an adapter module and a Hiera module connected in series. The Hiera replaces the complex component functions designed in networks such as the Swin Transformer by using vision and training methods.

[0186] ② Mask Decoder: SAM2UNet builds upon the UNet architecture to construct a mask decoder. SAM2UNet adds a receptive field (RFB) module to the skip connections, generating new features while aligning the feature map size. The decoder, corresponding to the encoder, consists of three upsampling decoding modules. The output of each decoding module serves not only as input to the next decoding module but also provides high-dimensional feature information through a lateral segmentation head composed of a single convolutional layer, offering additional information for loss calculation.

[0187] ③ Memory Mechanism: SAM2's processing of temporal information is divided into three parts: memory encoding, memory bank, and memory attention mechanism. After the mask decoder generates a mask image, the memory encoding mechanism encodes the mask image with image feature information to obtain a memory feature vector representing the features of a specific category in the current frame. The purpose of the memory bank is to store the memory encoding information of multiple frames. After memory encoding, the memory feature vector and the feature map of the image encoder are unloaded from the video memory and stored in memory for later use. The self-attention mechanism and cross-attention mechanism fuse the memory vectors of previous multiple frames with the image feature map of the current frame for the mask decoder to generate the mask.

[0188] 2) Design and performance analysis of semantic segmentation network based on temporal information fusion:

[0189] ① Design of SAM2UNet Network for Temporal Information Fusion: Since the paint layer only gradually transitions from the original paint layer state to the substrate layer state during the grinding process, theoretically, the visual characteristics of the paint layer can be determined by comparing previous images. Based on this idea, temporal information is fused into the feature map of the SAM2UNet network. Specifically, for the temporal information processing of the grinding process, the memory attention mechanism in SAM2 is used: the network extracts the segmentation mask representing the paint surface for each frame output, combines it with the feature map of that frame, encodes it using a memory encoder, and stores it in the memory bank; at the end of the image encoder, a memory attention operation is performed, fusing the feature map and the memory information encoded in the memory bank, and then sending it to the subsequent decoder branch.

[0190] ② Experiments on the semantic segmentation network performance based on real polishing datasets: The mean intersection-over-union ratio (IoU) and mean accuracy were used as evaluation metrics for the semantic segmentation task. The mean accuracy is the average pixel classification accuracy across all categories, and the mean IoU is the average IoU across all categories. Higher mean accuracy and mean IoU indicate more accurate segmentation results.

[0191] 3) Algorithm design for paint layer thickness estimation:

[0192] ① Based on SAM2UNet which incorporates temporal information, a thickness decoding branch is added, enabling the network to adapt to paint layer thickness estimation tasks. The overall network architecture is as follows: Figure 9 As shown.

[0193] To improve the accuracy of paint thickness prediction, a separate branch is used for decoding the feature map output by the image encoder, which also uses the UNet structure. From the perspective of input and output alone, the task of paint thickness estimation is similar to monocular depth estimation. Therefore, the network design is based on the approach used in depth estimation, replacing the final segmentation head of the decoder with a thickness estimation head, and using the hardtanh activation function to map the output values ​​to the range of (1, 0).

[0194] ② Design of loss function for deep learning networks:

[0195] To balance segmentation and paint layer thickness estimation, a comprehensive loss function is used in the network. To calculate the overall loss, it consists of two core parts: segmentation loss. and thickness prediction loss Segmentation loss Focusing on the overall image and category edges aims to improve the performance of the segmentation head and ensure higher accuracy in the network's segmentation results; prediction loss. Focusing solely on the workpiece surface aims to maximize the accuracy of paint layer prediction. The overall loss is as follows:

[0196]

[0197] in , These are hyperparameters for the two losses, used to adjust the ratio of the two weights.

[0198] In a mask decoder, a prediction value is output for each segmentation at different scales; therefore, the segmentation loss... It can be further divided into three identical parts, as shown below:

[0199]

[0200] in , , Three sets of hyperparameters for the loss, ranging from large to small scale, are used to adjust the model's attention to targets at different scales. The specific segmentation loss is as follows:

[0201]

[0202] in, To upsample the network mask output to its original size, One-hot encoded labels for images, Image size This represents the cross-entropy loss. represent The weights under the coordinates, which default to 1, are used in the classification loss to strengthen the loss of edge regions.

[0203] In the paint layer thickness decoder, the mean squared error loss is used as the thickness prediction loss, as shown below:

[0204]

[0205] in, The output is the network thickness prediction value upsampled to the original size. For the thickness label input, represent The weights of the coordinate system are used to shield the influence of the background region on the thickness prediction loss function.

[0206] ③ Model inference process:

[0207] When a grinding process is completed and the paint surface is inspected, the system should have saved at least two images, one of which is the original paint surface image. For a single inference, two images are input simultaneously. The network first performs inference on the first frame image. Since the memory is empty at this time, the feature map of the image encoder is directly input into the decoder. When inferring the current frame, the network has already saved the memory features of the original paint surface, and will fuse the memory features with the current feature map through a memory self-attention mechanism. In the post-processing of inference, the paint layer thickness information of the background needs to be filtered out. Here, the mask of the background of category 0 in the three-class segmentation information obtained from inference is used to set the background part of the grayscale image to 0.

[0208] (7) Dataset design

[0209] (6) A paint layer thickness estimation network was designed using deep learning. This method is data-driven, and the final effect is closely related to the quality of the training data. In this step, a dataset that meets the training requirements of the deep learning network will be designed. The dataset is based on two parts: images of the paint layer removal process of the workpiece during grinding, and the thickness estimation of the paint-substrate hybrid layer based on the above method; the dataset will be synthesized through a rendering engine to expand the dataset and enhance the generalization ability of the network.

[0210] 1) The design of the real polished paint surface dataset includes:

[0211] ① Training tags:

[0212] For semantic segmentation, this problem is defined as a pixel-by-pixel three-class classification problem, with semantic categories being: background, processed region, and unprocessed region. For paint thickness detection, this problem is defined as a pixel-by-pixel regression problem, using a normalized percentage to describe the relative thickness of the paint layer.

[0213] ② Timing information encoding:

[0214] The changes in the paint layer during grinding are unidirectional; therefore, obtaining image information from the early stages of grinding is helpful for image segmentation at the current moment. For the same set of grinding data, using the same number in the file name for encoding, and recording the timestamp information at the time of capture, allows retrieval of the same set of processing data from the dataset, and provides information on the workpiece state before and after processing.

[0215] ③ Data distribution:

[0216] To achieve substrates with various curvatures and colors, PLA material was used in the dataset for 3D printing to obtain different substrate colors and geometric features. In addition, machined aluminum alloy sheets were used as supplementary substrates to expand the range of substrate materials.

[0217] 2) The annotation of the polished paint surface dataset includes:

[0218] ① Paint layer thickness marking:

[0219] After collecting a complete set of processing data, the dataset is uniformly labeled. Therefore, a paint thickness estimation method based on image sequences can be used to calculate the thickness label for each pixel, as follows:

[0220]

[0221] ② Background semantic information annotation:

[0222] Because the workpiece is clearly distinguishable from the background, X-AnyLabeling software is used for semi-automatic annotation. During the annotation process, the workpiece is first selected using prompts, and the software uses a deep learning model to infer and automatically generate a workpiece mask. When the background color is similar to the substrate or paint finish, automatic annotation cannot effectively identify and segment the target; therefore, manual review of the annotation results is required, and the annotation information needs to be manually adjusted to the optimal state.

[0223] ③ Foreground semantic information annotation:

[0224] Due to the presence of the hybrid layer, even with manual annotation, it is difficult to determine the specific boundary between the paint layer and the substrate layer, which can easily lead to annotation errors. This makes it difficult for the model to learn effective boundary information, resulting in poor model fitting. Therefore, based on paint thickness estimation, the remaining semantic labels are annotated. Specifically, portions with a thickness value greater than 0.5 are classified as the unprocessed surface (label "1"), and portions less than 0.5 are classified as the processed surface (label "2"). The two types of labels generated are as follows: Figure 9 As shown.

[0225] 3) Creation of virtual datasets based on data synthesis:

[0226] This invention synthesizes a grinding processing dataset based on Blender's Cycles rendering engine. By synthesizing a large amount of paint surface data with random colors and lighting, it covers visual feature changes in unknown scenes and increases the model's generalization ability.

[0227] The specific method involves establishing a geometric model of the grinding test bench and the workpiece, then simulating the effect of tool processing on the surface morphology of the workpiece, modifying the surface material of the workpiece model to make it exhibit the visual effect after grinding, and finally rendering the image data.

[0228] S6: Based on the processing volume estimation in step S4, the grating path is combined and optimized: a series of long straight paths are generated at smaller intervals, and any number of trajectories are selected from them and iterated repeatedly through the optimization algorithm to make the overall removal rate of the workpiece surface reach the optimal level. The grating interval is adjusted according to the grinding and polishing removal model to meet the theoretical optimal interval.

[0229] (8) Path optimization design based on paint layer thickness estimation

[0230] (6) and (7) have already designed a deep learning network for estimating paint thickness. In this step, a path optimization method will be designed to feed the paint thickness estimate back into the path planning, thereby completing the closed loop of the detection and planning.

[0231] Based on the prediction results from deep learning, a grayscale image representing the estimated paint layer thickness can be obtained. The grayscale value represents the paint layer thickness at that point; the original paint layer and the rough-processed paint layer have a thickness of 1; non-workpiece areas and the substrate layer do not require processing and have a thickness of 0; the remaining grayscale values ​​correspond to the estimated paint layer thickness of the mixed-layer paint surface. Using the camera's intrinsic and extrinsic parameter matrices, the specific coordinates of any point in the point cloud in the pixel coordinate system can be calculated. By mapping the grayscale image to the point cloud, a point cloud containing the paint layer thickness can be obtained.

[0232] Specifically, the path planning method makes the following two improvements based on the above: First, in addition to the desired pose, an additional desired force is introduced for each path point. This force is adjusted based on the machining allowance within the local area of ​​each path point, thereby controlling the cutting amount at different locations along the path. Second, the estimated paint layer thickness is incorporated into the loss function calculation. By changing the calculation methods of the loss function and the desired force, different processing strategies can be implemented, thereby improving processing efficiency and reducing damage to the substrate.

[0233] 1) Path point expectation force:

[0234] By adjusting the expected force at the path points, the machining amount along the path is changed to adapt to the actual paint layer thickness variations at different locations on the workpiece surface. The changes in contact area caused by the deformation of the grinding disc due to the applied grinding force are ignored, and a scaling factor is used... To represent the expected grinding force at the path point With the maximum expected grinding force in the path The relationship between them is .

[0235] To improve efficiency, the applied grinding force should linearly change with the remaining paint thickness in the area traversed by the grinding disc. Here, this is achieved by using the point cloud of the area covered by the grinding disc at the sampling point. The average remaining paint layer thickness of the point cloud is calculated and used as a proportionality coefficient for the desired grinding force, i.e.:

[0236]

[0237] in, For local point clouds The number of points in the middle, For point The paint layer thickness is estimated. From the perspective of protecting the substrate, the amount of grinding caused by the grinding disc passing through a certain point should not exceed the remaining paint layer thickness at that point. Therefore, it is necessary to calculate the minimum equivalent paint layer thickness in the local area as a proportionality coefficient for the desired grinding force. ,Right now:

[0238]

[0239] 2) Path processing allowance estimation:

[0240] As the grinding force changes, the machining quantity per single path also changes accordingly:

[0241]

[0242] The estimated machining allowance at each point can be expressed as:

[0243]

[0244] in It is an empirical parameter that reflects the mapping relationship between the theoretical cutting amount and the paint layer thickness estimate. Its main function is to constrain the influence of the path coverage area on the overall uniformity of the machining allowance in the loss function. The machining allowance estimate is only used for the calculation of the loss function and cannot directly reflect the actual situation after machining.

[0245] 3) Combinatorial optimization loss function:

[0246] Building upon the previous approach, paint layer thickness estimation is introduced. Based on the polishing removal model and the estimated paint layer thickness, the remaining paint layer thickness after polishing can be roughly estimated. The loss function no longer uses the uniformity of the processing amount along the polishing trajectory as the evaluation metric, but instead focuses on the uniformity of the paint layer thickness after processing. The modified loss function is as follows:

[0247]

[0248]

[0249] in, For point clouds Mean value of the estimated intermediate coat allowance. It is a very small non-zero constant. For the number of paths, , , , , These are empirical parameters that reflect the mapping relationship between theoretical cutting amount and paint layer thickness estimation. When the paint surface is in its original state, we have... The expected force of each trajectory point is .

[0250] S7: Perform processing according to the optimization results of step S6, collect the processed image and feed the paint layer thickness estimate back to the path planning, repeat the mapping and optimization steps, thereby realizing the closed loop of detection and planning.

Claims

1. A method for detecting the paint condition of the surface paint layer of special equipment and for planning the path of a polishing robot, characterized in that, The method steps are as follows: S1: The point cloud and image of the workpiece are acquired by a 3D structured light camera. The point cloud is subjected to voxel downsampling and outlier filtering. The environmental points and test bench plane points are removed in sequence. Then, the workpiece point cloud is extracted by region growing clustering. S2: Perform principal component analysis on the workpiece point cloud to construct a local coordinate system. Transform the workpiece point cloud to the origin at the center coordinate using a pose transformation matrix. Align the three principal component directions with the x, y, and z axes respectively to establish an AABB-type bounding box. On the xy plane of the local coordinate system, take points in the second principal component direction with the first principal component direction as the unfolding direction and at a set interval to generate a raster-type candidate path template and perform path sampling. S3: Perform neighborhood point search at each sampling position, and use the corresponding coordinates of the neighborhood points in the point cloud to backfill the z coordinate of the sampling position according to the inverse proportional difference. Connect each path point and transform the path points in the local coordinate system back to the camera coordinate system through the pose transformation matrix. Determine the x-axis orientation of the robot end effector by the projection of the direction vector from the current path point to the next path point on the normal vector tangent plane, and use the Rodriguez formula to obtain the end effector pose matrix in the robot coordinate system. S4: The Preston equation is used to describe the cutting amount of the grinding and polishing process per unit time. The instantaneous processing amount is estimated and accumulated at the sampling points of the grating path to form the path processing amount field, and the grinding and polishing removal model is constructed. S5: Segment and thickness regression of camera images. Use a deep learning network to classify the input image into three categories: background, processed, and unprocessed. Estimate the paint layer thickness to obtain the current paint layer thickness estimate of the workpiece surface. The thickness estimation is mapped to the workpiece point cloud by mapping the image to the point cloud, and the amount of paint remaining after path execution is calculated. S6: Based on the processing volume estimation in step S4, the grating path is combined and optimized: a series of long straight paths are generated at smaller intervals, and any number of trajectories are selected from them and iterated repeatedly through the optimization algorithm to make the overall removal rate of the workpiece surface reach the optimal level. The grating interval is adjusted according to the grinding and polishing removal model to meet the theoretical optimal interval. S7: Perform processing according to the optimization results of step S6, collect the processed image and feed the paint layer thickness estimate back to the path planning, repeat the mapping and optimization steps, thereby realizing the closed loop of detection and planning.

2. The method for detecting the paint condition of special equipment surfaces and planning the path for a polishing robot, as described in claim 1, is characterized in that... In step S1: According to step distance Divide the space occupied by the point cloud into several cube-shaped voxels: in, This represents the set of points in each voxel. Represents the first point in the point cloud One point; Calculate the point set in each voxel Its center of gravity position: in, It is a point set The number of points in it; When there is only one point in a voxel, the position of that point remains unchanged; when there are multiple points in a voxel, the calculated centroid point replaces the point set. This reduces the number of points in the point cloud, achieving the purpose of downsampling. Point cloud collection For any point Its neighborhood point set : in, The preset search radius; For each point Statistical analysis of its neighborhood point set The number of midpoints, and compared with a preset threshold. Compare, if it is less than ,but These are outliers and need to be removed from the point cloud data; Apply Gaussian filtering to the workpiece point cloud and calculate the weights of neighboring points relative to that point. : in, This represents the standard deviation of a Gaussian distribution. The camera is fixed in place, with the eye outside the hand, for any point within the working area. By using the installation position conditions of the polishing robot, the three-dimensional position conditions of the points in the working area are restricted, thereby filtering out points outside the working area. The test bench plane point cloud is filtered out using the plane matching method. The fitting plane parameters are calculated by using the sampling consistency method on the point cloud. For each point in the point cloud, the distance from the point to the reference plane is calculated. The point cloud within a certain distance is filtered out, thus removing the test bench plane.

3. The method for detecting the paint condition of special equipment surfaces and planning the path for a polishing robot, as described in claim 2, is characterized in that... In step S2: For workpiece point cloud Principal component analysis was performed to obtain the centroid and three principal component directions of the workpiece point cloud, and the pose transformation matrix was calculated. The point cloud is transformed to a position where the centroid is at the origin and the three principal component directions are aligned with the x, y, and z axes, respectively, to obtain the point cloud. Constructing point clouds AABB-type bounding boxes are used to obtain their maximum and minimum coordinates. In the xy-plane of the local coordinate system of the point cloud, the x-axis, which is chosen as the first principal component of the point cloud, is selected as the unfolding direction of the first path. The total length is the length of the bounding box, and the unfolding is performed according to intervals. Take a series of points, spaced along the y-axis. Select points to obtain the raster path template.

4. The method for detecting the paint condition of special equipment surfaces and planning the path for a polishing robot, as described in claim 3, is characterized in that... In step S3: The path points are projected onto the point cloud, and points with x and y coordinates within a certain range are searched. The z coordinate of the path points is calculated using these neighboring points. The neighborhood search is performed by constructing a kd-tree, which is a temporary point cloud. By using point clouds The z-coordinate is set to zero and projected onto the xy plane to obtain the point cloud. Construct a kd-tree and perform a radius search using the kd-tree to find the range. Points within, due to and The points in the cloud are mapped one-to-one. For a neighboring point found by searching the path points, its representation in the point cloud is determined. The coordinates of the corresponding points are used to calculate the coordinates of any path point using the inverse proportional difference formula. In point cloud z-coordinate in: in, Indicates at the waypoint search radius Within, the total number of neighboring points found; Connect the beginning and end of each trajectory point, and then use the pose transformation matrix. Transform the path points in the local coordinate system back to the camera coordinate system to obtain the workpiece grinding and polishing path in the camera coordinate system. The direction of the minimum eigenvalue is calculated using principal component analysis, which is the normal vector direction of the local point cloud fitting plane. The x-axis of the polishing robot's end effector is aligned with the forward direction, the attitude matrix is ​​fixed, and the current path point is calculated. To the next path point The direction vector is projected onto the direction of the tangent plane of the normal vector: in, , Represents the normal vector; Through unit vector Calculate the current x-axis orientation of the polishing robot's end effector. Calculated using the Rodriguez formula arrive rotation matrix This allows us to obtain the end effector pose matrix in the robot coordinate system. .

5. The method for detecting the paint condition of special equipment surfaces and planning the path for a polishing robot, as described in claim 4, is characterized in that... In step S4: The empirical formula Preston equation is used to describe the amount of material removed during grinding and polishing per unit time: in, For pressure, For grinding speed, This is Preston's constant; by With the center of the circle, For radius, Angular velocity, instantaneous linear velocity of the polishing disc of the polishing robot. for: ; Therefore, instantaneous processing volume and distance Relationship: For the estimation of the processing amount of the trajectory formed between two adjacent path points, a series of sampling points are interpolated between the two adjacent path points, and the instantaneous processing amount is estimated for each sampling point. Then the total processing amount of the entire long straight grating path is approximately equivalent to the sum of the instantaneous processing amounts of all sampling points. At a constant distance Interpolating between two adjacent path points, we have Where r is the equivalent radius of the grinding disc, for each sampling point In planar point clouds Using radius Sampling is performed to obtain a neighborhood point cloud; subsequently, the point cloud is... Find the corresponding point and obtain the neighborhood point cloud. Based on the normal vector at the sampling point Calculate each neighboring point The distance d from the sampling point in the tangential plane direction si Distance z in the direction of normal si After calculating the instantaneous processing amount of each neighboring point, it is mapped onto the original point cloud. For each instantaneous processing quantity, divide by the number of sampling points within the period to ensure that the processing quantity estimate for a single long straight trajectory remains within the range of (0,1), i.e. in, This represents the total number of sampling points for a single trajectory. Let this be a constant used for normalization; Definition of the first All sampling points in the grating path If we form a point set S, then any point in the point cloud... In the The equivalent processing amount after processing a grating path is: in, ; Create a linked list to record all modified points in the corresponding trajectory, which will accelerate the subsequent calculation of loss values.

6. The method for detecting the paint condition of special equipment surfaces and planning the path for a polishing robot, as described in claim 5, is characterized in that... In step S5: For a single set of grinding data ,in It is the source image. It is single-channel thickness information, its initial state For the rough coating layer, final state As a substrate layer, Indicates the height of the image. Indicates the width of the image; Since the illumination and workpiece position remain constant throughout the entire grinding process, assuming any... All are made by and The result of a linear combination, that is, for any pixel on All of them are: in, Characterizes the proportions of the three channels; The method of projecting in RGB space is used to mix the information of the three channels for any pixel. In RGB space, establish and vector Then Projected to The projected distance obtained above is the required thickness estimate: To balance segmentation and paint layer thickness estimation, the deep learning network uses a comprehensive loss function. The overall loss is calculated, which consists of two core components: segmentation loss. and thickness prediction loss Segmentation loss Focus on the overall image and category edges; prediction loss Focusing solely on the overall surface loss of the workpiece, the following is an analysis: in, , These are hyperparameters for the two losses, used to adjust the ratio of the two weights; For segmentation at different scales, a predicted value will be output separately; this is the segmentation loss. Further divided into three identical parts: in, , , Three sets of hyperparameters for the loss, ranging from large to small scale, are used to adjust the model's attention to targets at different scales. To upsample the network mask output to its original size, One-hot encoded labels for images, Image size For cross-entropy loss, represent The weights under the coordinates are used in the classification loss to strengthen the loss of edge regions; In the paint layer thickness decoder, the mean square error loss is used as the thickness prediction loss: in, The output is the network thickness prediction value upsampled to the original size. For the thickness label input, represent The weights of the coordinate system are used to shield the influence of the background region on the thickness prediction loss function.

7. The method for detecting the paint condition of special equipment surfaces and planning the path for a polishing robot, as described in claim 6, is characterized in that... In step S6: Based on the prediction results of deep learning, a grayscale image representing the paint layer thickness estimate is obtained, where the grayscale value represents the paint layer thickness at that point. The thickness of the original paint layer and the rough-processed paint layer is 1, and the thickness of the non-workpiece area and the substrate layer is 0, which do not need to be processed. The remaining grayscale values ​​correspond to the paint layer thickness estimate of the mixed layer paint surface. By using the camera's intrinsic and extrinsic parameter matrices, the specific coordinates of any point in the point cloud in the pixel coordinate system are calculated, thereby mapping the grayscale image to the point cloud to obtain a point cloud with paint layer thickness. In addition to the desired pose, an additional desired force is introduced for the path points. This force is adjusted based on the machining allowance in the local area of ​​the path points to control the cutting amount at different positions on the path. Furthermore, the estimated paint layer thickness is incorporated into the loss function calculation. By changing the calculation methods of the loss function and the desired force, different processing strategies can be implemented, thereby improving processing efficiency and reducing damage to the substrate. Path point expectation power: By adjusting the expected force at the path points, the amount of machining along the path is changed to adapt to the actual paint layer thickness variations at different locations on the workpiece surface, using a scaling factor. To represent the expected grinding force at the path point With the maximum expected grinding force in the path The relationship between them is ; The applied grinding force changes linearly with the remaining paint thickness in the area traversed by the grinding disc, as shown by the point cloud within the area covered by the grinding disc at the sampling point. Calculate the average remaining paint layer thickness of the point cloud, and use it as a proportionality coefficient for the desired grinding force: in, Point clouds in the neighborhood The number of points in the middle, For point The paint layer thickness is estimated by statistically analyzing the minimum equivalent paint layer thickness in a local area, which is then used as a proportionality coefficient for the desired grinding force. ,Right now: Estimation of machining allowance along the path: As the grinding force changes, the machining quantity per single path also changes accordingly: The estimated machining allowance at each point is expressed as follows: in, This is an empirical parameter that reflects the mapping relationship between the theoretical cutting amount and the estimated paint thickness. Combinatorial optimization loss function: in, For point clouds Mean value of the estimated intermediate coat allowance. It is a very small non-zero constant. For the number of paths, , , , , These are empirical parameters that reflect the mapping relationship between theoretical cutting amount and paint layer thickness estimation. When the paint surface is in its original state, we have... The expected force of each trajectory point is .

Images