Automobile skid size measurement method and system based on three-dimensional reconstruction

Abstract

The invention discloses an automobile skid size measurement method and system based on three-dimensional reconstruction, and the method comprises the following steps: S1, controlling a mechanical arm to carry out the image shooting of an automobile skid according to a preset path and angle, and obtaining a plurality of processed images; s2, preprocessing the processed image, wherein preprocessing comprises data cleaning and point cloud feature extraction; s3, inputting the preprocessed processing image into the trained identification model to obtain part information including part classification and positioning; and S4, carrying out size parameter calculation on the part information to obtain the size of the automobile skid. And the sizes of different parts in the automobile skid can be accurately obtained, so that whether the production of the skid meets the specification requirement or not can be judged. The method can adapt to different automobile skid, and the universality of detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of dimensional measurement, and in particular to a method and system for measuring the dimensions of an automotive skid based on three-dimensional reconstruction. Background Art

[0002] In the field of automotive manufacturing, the painting skid plays a core role in body transportation and precise fixation. The accuracy of its dimensions is not only directly related to the smooth operation of the automotive production line but also has an inestimable impact on overall production safety. As a key component connecting various production links, the painting skid must ensure the stable and precise conveyance and positioning of the vehicle body on a high-speed and efficient automated production line. However, traditional dimensional measurement methods are inadequate when dealing with large-scale painting skids that need to move dynamically along the production line. These methods are often limited to two-dimensional measurement tools or traditional three-dimensional measurement instruments, which are not only cumbersome and complex to operate when dealing with large and dynamically changing objects but also difficult to achieve high-precision measurement results.

[0003] Specifically, traditional measurement methods encounter several challenges in practical applications: First, due to the large volume of the painting skid, it is difficult for traditional tools to quickly and comprehensively cover all measurement points, resulting in low measurement efficiency; second, the problem of error accumulation in a dynamic measurement environment is significant, affecting the measurement accuracy and making it difficult to meet the stringent accuracy requirements in automotive manufacturing; third, the high efficiency and continuity of assembly line operations require that the measurement process must be fast and not affect the production rhythm, while traditional methods often require pausing or decelerating the assembly line for adjustment, seriously reducing production efficiency; finally, traditional measurement means are insufficient in adapting to the dimensional changes of the painting skid in the moving state on the assembly line, and it is difficult to capture and process the dynamic changes of dimensional data in real time, thus limiting the further improvement of production quality.

[0004] Therefore, in view of the limitations of traditional dimensional measurement techniques in the application of painting skids, there is an urgent need in the industry for an innovative, efficient, and adaptable measurement method. Summary of the Invention

[0005] Aiming at the above-mentioned defects, the purpose of the present invention is to propose a method and system for measuring the dimensions of an automotive skid based on three-dimensional reconstruction to achieve high-precision and dynamic dimensional measurement of the painting skid, thereby ensuring the safety and efficiency of the automotive production line.

[0006] To achieve this purpose, the present invention adopts the following technical solutions: A method for measuring the dimensions of an automotive skid based on three-dimensional reconstruction, comprising the following steps:

[0007] Step S1: Control the robotic arm to capture images of the automotive skid according to a preset path and angle, and obtain a plurality of processed images;

[0008] Step S2: Preprocess the processed image, where the preprocessing includes data cleaning and point cloud feature extraction;

[0009] Step S3: Input the preprocessed processed image into the trained recognition model to obtain component information including component classification and positioning;

[0010] Step S4: Calculate the dimensional parameters of the component information to obtain the dimensions of the automotive skid.

[0011] Preferably, the steps of data cleaning in step S2 are as follows:

[0012] Step A1: Construct a three-dimensional point set P from the three-dimensional point cloud data captured by the three-dimensional scanning device, where P = {p1, p2,... pn}, and pn represents the nth point cloud data;

[0013] Step A2: Obtain any point cloud data pi in the three-dimensional point set P, and obtain all the point cloud data adjacent to the point cloud data pi in the three-dimensional point set P as the second point cloud data;

[0014] Calculate the average distance and standard deviation of all the second point cloud data from the point cloud data pi;

[0015] Calculate the selection distance through the average distance and standard deviation. If the distance between the point cloud data pi and the adjacent second point cloud data is greater than the selection distance, delete the second point cloud data. If the distance between the point cloud data pi and the adjacent second point cloud data is less than the selection distance, save the second point cloud data;

[0016] Step A3: Repeat step A2 until all the point cloud data in the three-dimensional point set P have been calculated;

[0017] Obtain the first point cloud data set after cleaning;

[0018] Step A4: Perform normalization processing on the first point cloud data set to obtain the second point cloud data set.

[0019] Preferably, the point cloud feature extraction in step S2 includes curvature feature extraction and normal vector feature extraction;

[0020] The curvature feature extraction steps are as follows:

[0021] For each point cloud data pi in the second point cloud data set, obtain all the point cloud data adjacent to the point cloud data pi in the original point cloud data as the third point cloud data;

[0022] Perform local plane or quadratic surface fitting by the least squares method, and then solve the curve equation according to multiple third point cloud data to obtain the curvature of the point cloud data pi;

[0023] The steps for extracting the normal vector features are as follows:

[0024] Perform singular value decomposition on the coordinate matrix X of the third point cloud data, and use the eigenvector corresponding to the smallest singular value as the normal vector of the point cloud data pi.

[0025] Preferably, the training steps of the recognition model include:

[0026] Step S31: Model construction:

[0027] Set the tensor for processing the image as: H×W×C, and the tensor of the convolutional kernel as: h×w×C, and set the output feature map F of the convolutional operation through the tensors of the processed image and the convolutional kernel;

[0028] Among them, the output feature map of the convolutional operation H and W are the height and width of the processed image respectively, h and w are the height and width of the convolutional kernel respectively, c is the number of channels, i and j are the coordinates of the output feature map of the convolutional operation respectively, the ranges of w′ and h′ are 0≤h′≤H - h + 1 and 0≤w′≤W - w + 1 respectively, I(w′ + m, h′ + n, c) represents the pixel value at the position (w′ + m, h′ + n) and channel c, and K(w′, h′, c) represents the pixel value of the processed image at the position (w′, h′) and channel c;

[0029] Step S32: Training of the recognition model:

[0030] Let the predicted probability distribution of the recognition model be The true label is e = (e 1 , e 2 … e k ), e i and respectively represent the true probability and predicted probability that the sample belongs to the i-th class, and its cross-entropy loss is:

[0031] Let the output of the recognition model be Q, and the predicted probability is obtained through the activation function σ

[0032] Obtain the gradient of the cross-entropy loss function with respect to the output Q:

[0033]

[0034] Then calculate the gradient of the cross-entropy loss function with respect to the model parameter λ through the chain rule. Let the output Q of the recognition model be obtained through the linear transformation Q = Wr + b and the activation function, where W is the weight matrix, r is the input, and b is the bias. Then the gradient for the weight matrix is:

[0035]

[0036] For the offset gradient:

[0037]

[0038] Step S33: Update of the offset and weights:

[0039] Update the model parameters using the Stochastic Gradient Descent (SGD) optimizer, where the update formulas for the weight matrix W and the bias b are respectively:

[0040]

[0041] η is the learning rate.

[0042] Preferably, the dimensional parameters in step S4 include the dimensional length of the component, the angle of the component, and the flatness of the component;

[0043] The formula for obtaining the dimensional length is as follows:

[0044]

[0045] where (x 1 , y 1 , z 1 ) and (x 2 , y 2 , z 2 ) are the coordinates of two points respectively;

[0046] The formula for obtaining the angle of the component is as follows:

[0047] where when the measurement target is a line segment, are the direction vectors of two line segments respectively, and when the measurement target is a plane, are the normal vectors of two planes respectively;

[0048] The steps for obtaining the flatness of the component are as follows:

[0049] Fit a reference plane, where the equation of the reference plane is as follows:

[0050] C = ax + by + o;

[0051] Solve for the plane parameters a, b, and o by minimizing the error function ;

[0052] Obtain the distance d from each point in the plane to the reference plane i ,

[0053] Take the standard deviation of all distances d i as the flatness of the plane.

[0054] An automotive skid size measurement system based on 3D reconstruction, using the above-mentioned automotive skid size measurement method based on 3D reconstruction, includes an image acquisition module, an image processing module, an identification module, and a calculation module;

[0055] The image acquisition module is used to control the robotic arm to take images of the automotive skid according to a preset path and angle, and obtain multiple processed images;

[0056] The image processing module is used to preprocess the processed images, where the preprocessing includes data cleaning and point cloud feature extraction;

[0057] The identification module is used to input the preprocessed processed images into a trained identification model to obtain component information including component classification and positioning;

[0058] The calculation module is used to calculate the size parameters of the component information to obtain the size of the automotive skid.

[0059] Preferably, the image processing module includes a first cleaning sub-module, a second cleaning sub-module, a third cleaning sub-module, and a fourth cleaning sub-module;

[0060] The first cleaning sub-module is used to construct a 3D point set P from the 3D point cloud data captured by the 3D scanning device;

[0061] The second cleaning sub-module is used to obtain any point cloud data pi in the 3D point set P, and obtain all the point cloud data adjacent to the point cloud data pi in the 3D point set P as the second point cloud data;

[0062] Calculate the selection distance through the average distance and standard deviation. If the distance between the point cloud data pi and the adjacent second point cloud data is greater than the selection distance, the second point cloud data is deleted. If the distance between the point cloud data pi and the adjacent second point cloud data is less than the selection distance, the second point cloud data is saved;

[0063] The third cleaning sub-module is used to replace the next point cloud data of the 3D point set P and re-invoke the second cleaning sub-module until all the point cloud data in the 3D point set P has been calculated to obtain the first cleaned point cloud data set;

[0064] The fourth cleaning sub-module is used to perform normalization processing on the first point cloud data set to obtain the second point cloud data set.

[0065] Preferably, the image processing module further includes a feature extraction sub-module;

[0066] The feature extraction sub-module is used to obtain all the point cloud data adjacent to the point cloud data pi in the original point cloud data for each point cloud data pi in the second point cloud dataset as the third point cloud data;

[0067] Perform local plane or quadratic surface fitting by the least squares method, and then solve the curve equation according to multiple third point cloud data to obtain the curvature of the point cloud data pi;

[0068] Perform singular value decomposition on the coordinate matrix X of the third point cloud data, and use the eigenvector corresponding to the smallest singular value as the normal vector of the point cloud data pi.

[0069] Preferably, it further includes a training module, and the training module includes a construction sub-module, a training sub-module, and an update sub-module;

[0070] The construction sub-module is used to set the tensor for processing the image as: H×W×C, and the tensor of the convolution kernel as: h×w×C, and set the convolution operation output feature map F through the tensors of the processing image and the convolution kernel;

[0071] Among them, the convolution operation output feature map H and W are respectively the height and width of the processed image, h and w are respectively the height and width of the convolution kernel, c is the number of channels, i and j are respectively the coordinates of the convolution operation output feature map, the ranges of w′ and h′ are 0≤h′≤H - h + 1 and 0≤w′≤W - w + 1 respectively, I(w′ + m, h′ + n, c) represents the pixel value at the position (w′ + m, h′ + n) and channel c, and K(w′, h′, c) represents the pixel value of the processed image at the position (w′, h′) and channel c;

[0072] The training sub-module is used to set the predicted probability distribution of the recognition model as The true label is e=(e 1 ,e 2 …e k ), e i and respectively represent the true probability and the predicted probability that the sample belongs to the i-th class, and its cross-entropy loss is:

[0073] Let the output of the recognition model be Q, and obtain the predicted probability through the activation function σ

[0074] Obtain the gradient of the cross-entropy loss function with respect to the output Q:

[0075]

[0076] Then, the gradient of the cross-entropy loss function with respect to the model parameter λ is calculated using the chain rule. Assume that the output Q of the recognition model is obtained through a linear transformation of Q = Wr + b and an activation function, where W is the weight matrix, r is the input, and b is the bias. Then, the gradient with respect to the weight matrix is:

[0077]

[0078] The gradient with respect to the bias is:

[0079]

[0080] The update sub-module is used to update the model parameters using a Stochastic Gradient Descent (SGD) optimizer. The update formulas for the weight matrix W and the bias b are respectively:

[0081]

[0082] η is the learning rate.

[0083] One of the technical solutions in the above technical solutions has the following advantages or beneficial effects: The processed image after preprocessing can reduce the impact of noise on the image quality, improve the clarity and stability of the image. It is convenient to use the recognition model to recognize the processed image subsequently to identify different components of the automotive skid. Then, the size parameters of different components are calculated to obtain all the size information of the automotive skid. Finally, through the size information, the shooting distance of the 3D scanning device, and the corresponding pixel relationship, the sizes of different components in the automotive skid can be accurately obtained, thereby determining whether the production of the skid meets the specification requirements. This method can adapt to different automotive skids and improve the versatility of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 is a flowchart of an embodiment of the method of the present invention.

[0085] Figure 2 is a schematic structural diagram of an embodiment of the system of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0086] The embodiments of the present invention are described in detail below. The examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

[0087] In the description of the embodiments of the present invention, the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the embodiments of the present invention, the meaning of "a plurality" is two or more unless otherwise specifically defined.

[0088] In addition, the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise stated, the meaning of "a plurality" is two or more. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0089] As Figures 1-2 shown, a method for measuring the size of an automotive skid based on three-dimensional reconstruction includes the following steps:

[0090] Step S1: Control the robotic arm to take images of the automotive skid according to a preset path and angle, and obtain a plurality of processed images;

[0091] In the present invention, coordinate markings of each key point or component on the automotive skid are performed in a three-dimensional space through a three-dimensional scanning device, and its precise position can be determined;

[0092] Due to the complex structure of the automotive skid, a single-angle photograph can only show some of its surfaces and features and cannot cover all the surfaces of the automotive skid to obtain complete external shape data. Therefore, in the present invention, a preset path needs to be set, and the robotic arm moves and takes pictures along the preset path to obtain information on all the surfaces of the automotive skid; the preset path includes directly in front of the skid, the left side of the skid, the right side, and the diagonally above the four corners of the automotive skid. When taking pictures directly in front, on the left side, and on the right side, the three-dimensional scanning device needs to be perpendicular to the side of the skid. Then, through the relative position relationships between the various angles: the relative positions between the components, such as distance, parallelism, perpendicularity, etc., the overall structural layout of the skid is reflected.

[0093] Step S2: Preprocess the processed images, where the preprocessing includes data cleaning and point cloud feature extraction;

[0094] Step S3: Input the preprocessed processed images into a trained recognition model to obtain component information including component classification and positioning;

[0095] Step S4: Calculate the dimensional parameters of the component information to obtain the size of the automotive skid.

[0096] The preprocessed processed image can reduce the impact of noise on the image quality, improve the clarity and stability of the image, which is convenient for subsequent use of the recognition model to recognize the processed image to identify different components of the automotive skid. Then, calculate the dimensional parameters of different components to obtain all the dimensional information of the automotive skid. Finally, through the dimensional information, the shooting distance of the 3D scanning device, and the corresponding pixel relationship, the dimensions of different components in the automotive skid can be accurately obtained, so as to determine whether the production of the skid meets the specification requirements. Through this method, different automotive skids can be adapted, and the versatility of detection can be improved.

[0097] Preferably, the steps of data cleaning in step S2 are as follows:

[0098] Step A1: Construct a 3D point set P from the 3D point cloud data captured by the 3D scanning device, where P = {p1, p2,... pn}, and pn represents the nth point cloud data;

[0099] Step A2: Obtain any point cloud data pi in the 3D point set P, and obtain all the point cloud data adjacent to the point cloud data pi in the 3D point set P as the second point cloud data;

[0100] Calculate the average distance and standard deviation of all the second point cloud data from the point cloud data pi;

[0101] Calculate the selection distance through the average distance and standard deviation. If the distance between the point cloud data pi and the adjacent second point cloud data is greater than the selection distance, delete the second point cloud data. If the distance between the point cloud data pi and the adjacent second point cloud data is less than the selection distance, save the second point cloud data;

[0102] Step A3: Repeat step A2 until all the point cloud data in the 3D point set P have been calculated;

[0103] The formula for calculating the selection distance through the average distance and standard deviation is as follows:

[0104] d c = d avg + t × σ, where d avg is the average distance, σ is the standard deviation, and t is a constant term with a value range between 2 and 3.

[0105] Obtain the first point cloud data set after cleaning;

[0106] Step A4: Perform normalization processing on the first point cloud data set to obtain the second point cloud data set.

[0107] When scanning through a three-dimensional scanning device, there is a lot of noisy data (outliers) although it is highly accurate. If the outliers are retained, they will affect the calculation results in subsequent dimensional parameter calculations. In the case of automotive skids, due to their shape and structure, the sparsity of the point cloud data obtained by the three-dimensional scanning device is different. If a conventional fixed value is used to divide the outliers, the point cloud data of some relatively sparse components (such as fixing screws) will be cleared, which also affects the dimensional inspection of this part. Therefore, in the present invention, the average distance and standard deviation of the neighboring points of the point are used to dynamically generate a selection distance, so as to better adapt to the removal of outliers with different sparsities, adapt to point cloud regions with different densities or noise distributions, and improve the robustness.

[0108] Preferably, the point cloud feature extraction in step S2 includes curvature feature extraction and normal vector feature extraction;

[0109] The curvature feature extraction steps are as follows:

[0110] For each point cloud data pi in the second point cloud dataset, all the point cloud data adjacent to the point cloud data pi are obtained in the original point cloud data as the third point cloud data;

[0111] Perform local plane or quadratic surface fitting by the least squares method, and then solve the curve equation according to multiple third point cloud data to obtain the curvature of the point cloud data pi;

[0112] The normal vector feature extraction steps are as follows:

[0113] Perform singular value decomposition on the coordinate matrix X of the third point cloud data, and use the eigenvector corresponding to the smallest singular value as the normal vector of the point cloud data pi.

[0114] Obtaining the curvature and normal vector of the point cloud data in the preprocessing step can help the subsequent recognition model quickly identify the position of this part. For example, points with a large curvature may be edge points or corner points, and the change of the normal vector can help identify edges and contours. For example, the direction of the normal vector will change suddenly at the edge.

[0115] Preferably, the training steps of the recognition model include:

[0116] Step S31: Model construction:

[0117] Set the tensor for processing the image as: H×W×C, and the tensor of the convolutional kernel as: h×w×C, and set the convolutional operation output feature map F through the tensors of the processing image and the convolutional kernel;

[0118] Among them, the convolutional operation output feature map Let \(H\) and \(W\) be the height and width of the processed image respectively, \(h\) and \(w\) be the height and width of the convolutional kernel respectively, \(c\) be the number of channels, \(i\) and \(j\) be the coordinates of the output feature map of the convolutional operation respectively. The ranges of \(w'\) and \(h'\) are \(0\leq h'\leq H - h + 1\) and \(0\leq w'\leq W - w + 1\) respectively. \(I(w'+m,h'+n,c)\) represents the pixel value at position \((w'+m,h'+n)\) and channel \(c\), and \(K(w',h',c)\) represents the pixel value of the processed image at position \((w',h')\) and channel \(c\);

[0119] Step S32: Training of the recognition model:

[0120] Let the predicted probability distribution of the recognition model be The true label is \(e=(e 1 ,e 2 …e k ), where \(e i and represent the true probability and predicted probability that the sample belongs to the \(i\)-th class respectively, and the cross-entropy loss is:

[0121] Let the output of the recognition model be \(Q\), and the predicted probability

[0122] is obtained through the activation function \(\sigma\).

[0123]

[0124] Then, the gradient of the cross-entropy loss function with respect to the output \(Q\) is obtained:

[0125]

[0126] The gradient with respect to the bias is:

[0127]

[0128] The cross-entropy loss function can well measure the difference between the predicted probability distribution of the recognition model and the true label. By minimizing this loss function, the prediction of the recognition model can be made more accurate. At the same time, by calculating the gradient of the cross-entropy loss function with respect to the parameters of the recognition model through the chain rule, we can accurately know the influence degree of each parameter on the loss function, so as to carry out targeted optimization.

[0129] Step S33: Update of the bias and weights:

[0130] Update the model parameters using the Stochastic Gradient Descent (SGD) optimizer. The update formulas for the weight matrix W and the bias b are as follows:

[0131]

[0132] η is the learning rate.

[0133] The SGD optimizer updates the gradient of only a small batch of data each time, enabling the model to converge to the optimal solution faster. At the same time, SGD also has a certain regularization effect, which can prevent the model from overfitting.

[0134] Preferably, the dimensional parameters in step S4 include the dimensional length of the component, the angle of the component, and the flatness of the component;

[0135] The formula for obtaining the dimensional length is as follows:

[0136]

[0137] where (x 1 , y 1 , z 1 ) and (x 2 , y 2 , z 2 ) are the coordinates of two points respectively;

[0138] The formula for obtaining the angle of the component is as follows:

[0139] where when the measurement target is a line segment, are the direction vectors of two line segments respectively. When the measurement target is a plane, are the normal vectors of two planes respectively;

[0140] The steps for obtaining the flatness of the component are as follows:

[0141] Fit a reference plane, where the equation of the reference plane is as follows:

[0142] C = ax + by + o;

[0143] Solve for the plane parameters a, b, and o by minimizing the error function ;

[0144] Obtain the distance d from each point in the plane to the reference plane i ,

[0145] Take the standard deviation of all distances d i as the flatness of the plane.

[0146] An automobile skid size measurement system based on three-dimensional reconstruction, using the method for measuring the size of an automobile skid based on three-dimensional reconstruction, includes an image acquisition module, an image processing module, an identification module, and a calculation module;

[0147] The image acquisition module is used to control the robotic arm to take images of the automobile skid according to a preset path and angle, and obtain a plurality of processed images;

[0148] The image processing module is used to preprocess the processed images, where the preprocessing includes data cleaning and point cloud feature extraction;

[0149] The identification module is used to input the preprocessed processed images into a trained identification model to obtain component information including component classification and positioning;

[0150] The calculation module is used to calculate the size parameters of the component information to obtain the size of the automobile skid.

[0151] Preferably, the image processing module includes a first cleaning sub-module, a second cleaning sub-module, a third cleaning sub-module, and a fourth cleaning sub-module;

[0152] The first cleaning sub-module is used to construct a three-dimensional point set P from the three-dimensional point cloud data captured by the three-dimensional scanning device;

[0153] The second cleaning sub-module is used to obtain any point cloud data pi in the three-dimensional point set P, and obtain all the point cloud data adjacent to the point cloud data pi in the three-dimensional point set P as the second point cloud data;

[0154] Calculate the selection distance through the average distance and the standard deviation. If the distance between the point cloud data pi and the adjacent second point cloud data is greater than the selection distance, the second point cloud data is deleted. If the distance between the point cloud data pi and the adjacent second point cloud data is less than the selection distance, the second point cloud data is saved;

[0155] The third cleaning sub-module is used to replace the next point cloud data of the three-dimensional point set P and re-invoke the second cleaning sub-module until all the point cloud data in the three-dimensional point set P have been calculated to obtain the first cleaned point cloud data set;

[0156] The fourth cleaning sub-module is used to perform normalization processing on the first point cloud data set to obtain the second point cloud data set.

[0157] Preferably, the image processing module further includes a feature extraction sub-module;

[0158] The feature extraction sub-module is used to obtain all the point cloud data adjacent to the point cloud data pi in the original point cloud data for each point cloud data pi in the second point cloud dataset as the third point cloud data;

[0159] Perform local plane or quadratic surface fitting by the least squares method, and then solve the curve equation based on multiple third point cloud data to obtain the curvature of the point cloud data pi;

[0160] Perform singular value decomposition on the coordinate matrix X of the third point cloud data, and use the eigenvector corresponding to the smallest singular value as the normal vector of the point cloud data pi.

[0161] Preferably, it further includes a training module, and the training module includes a construction sub-module, a training sub-module, and an update sub-module;

[0162] The construction sub-module is used to set the tensor of the processed image as: H×W×C, and the tensor of the convolution kernel as: h×w×C, and set the output feature map F of the convolution operation through the tensors of the processed image and the convolution kernel;

[0163] Among them, the output feature map of the convolution operation H and W are respectively the height and width of the processed image, h and w are respectively the height and width of the convolution kernel, c is the number of channels, i and j are respectively the coordinates of the output feature map of the convolution operation, the ranges of w′ and h′ are 0≤h′≤H - h + 1 and 0≤w′≤W - w + 1 respectively, I(w + m, h′ + n, c) represents the pixel value at the position (w′ + m, h′ + n) and channel c, and K(w′, h′, c) represents the pixel value of the processed image at the position (w′, h′) and channel c;

[0164] The training sub-module is used to set the predicted probability distribution of the recognition model as The true label is e = (e 1 , e 2 … e k ), e i and respectively represent the true probability and predicted probability that the sample belongs to the i-th class, and its cross-entropy loss is:

[0165] Let the output of the recognition model be Q, and obtain the predicted probability through the activation function σ

[0166] Obtain the gradient of the cross-entropy loss function with respect to the output Q:

[0167]

[0168] Then, the gradient of the cross-entropy loss function with respect to the model parameter λ is calculated by the chain rule. Assume that the output Q of the recognition model is obtained through a linear transformation of Q = Wr + b and an activation function, where W is the weight matrix, r is the input, and b is the bias. Then, the gradient with respect to the weight matrix is as follows:

[0169]

[0170] The gradient with respect to the bias is as follows:

[0171]

[0172] The update sub-module is used to update the model parameters using a stochastic gradient descent (SGD) optimizer. The update formulas for the weight matrix W and the bias b are respectively:

[0173]

[0174] η is the learning rate.

[0175] In the description of this specification, the descriptions referring to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0176] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for measuring the size of a car skid based on three-dimensional reconstruction, characterized in that: The steps include: Step S1: controlling the robot arm to capture images of the car skid according to a preset path and angle to obtain multiple processed images; Step S2: preprocessing the processed image, wherein the preprocessing includes data cleaning and point cloud feature extraction; Step S3: input the pre-processed image into the trained recognition model to obtain component information including component classification and positioning; Step S4: Calculate the size parameters of the component information to obtain the size of the automobile skid.

2. A method for measuring the size of a car skid based on three-dimensional reconstruction according to claim 1, characterized in that: The steps of data cleaning in step S2 are as follows: Step A1: construct a three-dimensional point set P from the three-dimensional point cloud data captured by the three-dimensional scanning device, where P = {p1, p2, ... pn}, and pn represents the nth point cloud data; Step A2: obtaining any point cloud data pi in the three-dimensional point set P, and obtaining all point cloud data adjacent to the point cloud data pi in the three-dimensional point set P as the second point cloud data; Calculate the average and standard deviation of the distances between all second point cloud data and point cloud data pi; The selected distance is calculated by the distance average and standard deviation. If the distance between the point cloud data pi and the adjacent second point cloud data is greater than the selected distance, the second point cloud data is deleted. If the distance between the point cloud data pi and the adjacent second point cloud data is less than the selected distance, the second point cloud data is saved. Step A3: Repeat step A2 until all point cloud data in the three-dimensional point set P are calculated; Get the first point cloud data set after cleaning; Step A4: normalize the first point cloud data set to obtain a second point cloud data set.

3. A method for measuring the size of a car skid based on three-dimensional reconstruction according to claim 2, characterized in that: The point cloud feature extraction in step S2 includes curvature feature extraction and normal vector feature extraction; The steps of curvature feature extraction are as follows: For each point cloud data pi in the second point cloud data set, all point cloud data adjacent to the point cloud data pi are obtained in the original point cloud data as the third point cloud data; The local plane or quadratic surface is fitted by the least square method, and then the curve equation is solved according to the plurality of third point cloud data to obtain the curvature of the point cloud data pi; The steps of normal vector feature extraction are as follows: The coordinate matrix X of the third point cloud data is subjected to singular value decomposition, and the eigenvector corresponding to the smallest singular value is used as the normal vector of the point cloud data pi.

4. A method for measuring the size of a car skid based on three-dimensional reconstruction according to claim 3, characterized in that: The training steps of the recognition model include: Step S31: Model building: Set the tensor of the processed image to: H×W×C, the tensor of the convolution kernel to: h×w×c, and set the convolution operation output feature map F by processing the tensor of the image and the convolution kernel; The convolution operation outputs the feature map H and W are the height and width of the processed image, h and w are the height and width of the convolution kernel, c is the number of channels, i and j are the coordinates of the output feature map of the convolution operation, w′ and h′ ranges are 0≤h′≤H-h+1 and 0≤w′≤W-w+1, I(w′+m,h′+n,c) represents the pixel value at position (w′+m,h′+n) and channel c, and K(w′,h′,c) represents the pixel value at position (w′,h′) and channel c of the processed image; Step S32: Training of recognition model: Assume that the predicted probability distribution of the recognition model is The real label is e=(e1,e2…e k ), e i and They represent the true probability and predicted probability of the sample belonging to the i-th category, respectively, and the cross entropy loss is: Assume that the output of the recognition model is Q, and the predicted probability is obtained through the activation function σ Get the gradient of the cross entropy loss function with respect to the output Q: Then, the gradient of the cross entropy loss function to the model parameter λ is calculated by the chain rule. Assume that the output Q of the recognition model is obtained by the linear transformation and activation function of Q=Wr+b, where W is the weight matrix, r is the input, and b is the bias. Then the gradient of the weight matrix is: The gradient for the bias is: Step S33: Update of bias and weight: The model parameters are updated using the stochastic gradient descent (SGD) optimizer, where the update formulas for the weight matrix W and the bias b are: η is the learning rate.

5. A method for measuring the size of a car skid based on three-dimensional reconstruction according to claim 4, characterized in that: The size parameters in step S4 include the size length of the component, the angle of the component and the flatness of the component; The formula for obtaining the size length is as follows: Where (x1, y1, z1) and (x2, y2, z2) are the coordinates of two points respectively; The formula for obtaining the angle of a component is as follows: When the measurement target is a line segment, are the direction vectors of the two line segments respectively. When the measurement target is a plane, are the normal vectors of the two planes respectively; The steps for obtaining the flatness of the component are as follows: A reference plane is fitted, where the equation of the reference plane is as follows: C = ax + by + o; By minimizing the error function To solve the plane parameters a, b and o; Get the distance d from each point in the plane to the reference plane i , For all distances d i The standard deviation of is taken as the flatness of the plane.

6. A system for measuring the size of a car skid based on three-dimensional reconstruction, using the method for measuring the size of a car skid based on three-dimensional reconstruction according to any one of claims 1 to 5, characterized in that: It includes an image acquisition module, an image processing module, a recognition module and a calculation module; The image acquisition module is used to control the mechanical arm to take images of the automobile skid according to a preset path and angle, and acquire multiple processed images; The image processing module is used to preprocess the processed image, wherein the preprocessing includes data cleaning and point cloud feature extraction; The recognition module is used to input the pre-processed processed image into the trained recognition model to obtain component information including component classification and positioning; The calculation module is used to calculate the size parameters of the component information to obtain the size of the automobile skid.

7. The automobile skid dimension measurement system based on three-dimensional reconstruction according to claim 6, characterized in that: The image processing module includes a first cleaning submodule, a second cleaning submodule, a third cleaning submodule and a fourth cleaning submodule; The first cleaning submodule is used to construct a three-dimensional point set P from the three-dimensional point cloud data captured by the three-dimensional scanning device; The second cleaning submodule is used to obtain any point cloud data pi in the three-dimensional point set P, and obtain all point cloud data adjacent to the point cloud data pi in the three-dimensional point set P as second point cloud data; The selected distance is calculated by the distance average and standard deviation. If the distance between the point cloud data pi and the adjacent second point cloud data is greater than the selected distance, the second point cloud data is deleted. If the distance between the point cloud data pi and the adjacent second point cloud data is less than the selected distance, the second point cloud data is saved. The third cleaning submodule is used to replace the next point cloud data of the three-dimensional point set P, and re-call the second cleaning submodule until all the point cloud data in the three-dimensional point set P are calculated to obtain a cleaned first point cloud data set; The fourth cleaning submodule is used to normalize the first point cloud data set to obtain a second point cloud data set.

8. The automobile skid dimension measurement system based on three-dimensional reconstruction according to claim 7, characterized in that: The image processing module also includes a feature extraction submodule; The feature extraction submodule is used for obtaining, for each point cloud data pi in the second point cloud data set, all point cloud data adjacent to the point cloud data pi in the original point cloud data as the third point cloud data; The local plane or quadratic surface is fitted by the least square method, and then the curve equation is solved according to the plurality of third point cloud data to obtain the curvature of the point cloud data pi; The coordinate matrix X of the third point cloud data is subjected to singular value decomposition, and the eigenvector corresponding to the smallest singular value is used as the normal vector of the point cloud data pi.

9. The automobile skid dimension measurement system based on three-dimensional reconstruction according to claim 7, characterized in that: Also includes a training module, the training module includes a building submodule, a training submodule and an updating submodule; The construction submodule is used to set the tensor of the processed image to be: H×W×C, the tensor of the convolution kernel to be: h×w×C, and the convolution operation output feature map F is set by the tensor of the processed image and the convolution kernel; The convolution operation outputs the feature map H and W are the height and width of the processed image, h and w are the height and width of the convolution kernel, c is the number of channels, i and j are the coordinates of the output feature map of the convolution operation, w′ and h′ ranges are 0≤h′≤H-h+1 and 0≤w′≤W-w+1, I(w′+m,h′+n,c) represents the pixel value at position (w′+m,h′+n) and channel c, and K(w′,h′,c) represents the pixel value at position (w′,h′) and channel c of the processed image; The training submodule is used to set the predicted probability distribution of the recognition model as The real label is e=(e1,e2…e k ), e i and They represent the true probability and predicted probability of the sample belonging to the i-th category, respectively, and the cross entropy loss is: Assume that the output of the recognition model is Q, and the predicted probability is obtained through the activation function σ Get the gradient of the cross entropy loss function with respect to the output Q: Then, the gradient of the cross entropy loss function to the model parameter λ is calculated by the chain rule. Assume that the output Q of the recognition model is obtained by the linear transformation and activation function of Q=Wr+b, where W is the weight matrix, r is the input, and b is the bias. Then the gradient of the weight matrix is: The gradient for the bias is: The update submodule is used to update the model parameters using a stochastic gradient descent (SGD) optimizer, where the update formulas of the weight matrix W and the bias b are respectively: η is the learning rate.

Images