Vehicle body scratch detection method

Abstract

The invention relates to the technical field of scratch detection, and discloses a vehicle body scratch detection method, which comprises the following steps: step S101, acquiring an RGB image and a depth image of a detection object through a three-dimensional vision technology; step S102, carrying out preprocessing on the RGB image and the depth image; step S103, performing data fusion on the preprocessed RGB image and depth image based on a multi-channel fusion method to obtain a multi-channel matrix, constructing a scratch detection model, and outputting a scratch bounding box coordinate, scratch depth and confidence; and S104, according to a scratch detection model output result, carrying out severity grading on scratches, and taking different measures. According to the method, the scratch detection model is constructed, and comprehensive analysis and detection of scratches are realized through operations such as maximum pooling, average pooling, weight adjustment and the like.

Description

Technical Field

[0001] The invention belongs to the technical field of scratch detection, and in particular relates to a method for detecting scratches on a vehicle body. Background Art

[0002] In the process of automobile manufacturing, the quality inspection of the body surface has always been one of the key links that affect the appearance and performance of the whole vehicle. In particular, scratches on the body surface not only directly affect the appearance of the car, but also may reduce the anti-corrosion performance of the body, thereby affecting the durability of the car.

[0003] Current methods for detecting scratches on vehicle bodies mainly include: manual visual inspection and automated scratch detection based on image processing. Manual visual inspection is the most traditional method for detecting scratches on vehicle bodies. It is usually performed by professional inspectors who visually inspect the surface of the vehicle body under strong light, and use experience and intuition to determine whether there are defects such as scratches, dents or rust spots on the vehicle body. Although this method is intuitive and does not require complex equipment support, it is highly subjective, and the detection results are often affected by personal factors, lack consistency, and are affected by the fatigue level of the inspectors themselves, which can easily lead to missed detections and false detections. In addition, manual detection is slow and difficult to adapt to the needs of efficient and mass production of modern automobiles. The automated scratch detection method based on image processing obtains images of the body surface through industrial cameras installed on the production line, and detects scratches, rust spots and other defects in combination with image processing algorithms. Compared with manual visual inspection, it has higher efficiency and consistency, but the existing detection methods based on 2D image processing usually rely on algorithms such as edge detection and morphological processing. When there are complex backgrounds, textures or reflections on the body surface, the detection system cannot accurately separate scratches from the background, resulting in false detection or missed detection. 2D images can only capture the planar information of the body surface, and cannot effectively detect subtle depth changes (such as dents or deeper scratches), especially when the scratches or dents are shallow, their grayscale or color changes in the plane image are not obvious and difficult to be recognized by the system.

[0004] Therefore, a vehicle body scratch detection method is urgently needed to solve the above problems. Summary of the invention

[0005] The present invention provides a vehicle body scratch detection method, which solves the technical problems of low efficiency, easy false detection and missed detection in the related technology.

[0006] The present invention provides a method for detecting scratches on a vehicle body, comprising the following steps:

[0007] Step S101, collecting an RGB image and a depth image of a detection object through three-dimensional vision technology;

[0008] Step S102, preprocessing the RGB image and the depth image;

[0009] Step S103, performing data fusion on the preprocessed RGB image and the depth image based on a multi-channel fusion method to obtain a multi-channel matrix, and constructing a scratch detection model, the input of the scratch detection model is the multi-channel matrix, and the output is a scratch result sequence, and a sequence unit of the result sequence includes: scratch boundary box coordinates, scratch depth and confidence;

[0010] Step S104: classify the severity of the scratches according to the result sequence of the scratches output by the scratch detection model, and take different measures.

[0011] Furthermore, in step S102, the RGB image and the depth image are preprocessed, and the specific steps include:

[0012] Step S201, scaling the RGB image by a bilinear interpolation method according to the resolution of the depth image to make it consistent with the size of the depth image;

[0013] Step S202, converting the RGB image into a grayscale image;

[0014] Step S203, using a Gaussian filter algorithm to smooth the RGB image, and using a median filter algorithm to denoise the depth image;

[0015] Step S204, adjusting the contrast of the RGB image using histogram equalization;

[0016] Step S205, normalizing the depth image using a z-score normalization method;

[0017] Step S206: perform image calibration on the RGB image and the depth image.

[0018] Furthermore, the step of calibrating the RGB image and the depth image in step S206 includes:

[0019] Step S301, calibrate the intrinsic and extrinsic parameters of the camera that collects the RGB image and the depth image. The intrinsic parameters are used to describe the internal optical characteristics of each camera, including focal length, principal point position and distortion coefficient; the extrinsic parameters are used to describe the position and posture relationship between the two cameras, including rotation matrix and translation vector, which represent the spatial position relationship between the two cameras relative to each other;

[0020] Step S302, aligning the RGB image and the depth image through perspective transformation and pixel mapping; perspective transformation refers to transforming the coordinate system of the RGB camera into the coordinate system of the depth camera according to the relative posture of the two cameras, so that the two can be aligned in the same space; pixel mapping refers to using the calibrated camera parameters to transform the coordinates of each pixel in the RGB image and map them to the corresponding pixel points in the depth image;

[0021] Step S303: performing distortion correction on the RGB image and the depth image according to the intrinsic parameters of the camera.

[0022] Furthermore, the step of fusing the preprocessed RGB image and the depth image based on a multi-channel fusion method in step S103 includes:

[0023] Step S401, extracting the red, green, and blue channels of each pixel of the RGB image, and generating three corresponding values: R(i, j), G(i, j), and B(i, j);

[0024] Step S402, extracting depth information D(i, j) of the depth image, indicating the distance from the pixel to the camera;

[0025] Step S403, combine the three channels (R, G, B) of the RGB image and the single channel D of the depth image to form a four-channel matrix 9R, G, B, D); each pixel will have four eigenvalues: [R(i, j), G9i, j), B(i, j), D(i, j)].

[0026] Furthermore, the scratch detection model includes: a convolutional layer, a first hidden layer, a second hidden layer and a target detection head;

[0027] The convolution layer is used to input the multi-channel matrix, extract the features of the multi-channel matrix, and output a first feature matrix;

[0028] The first hidden layer is used to input the first feature matrix, adjust the weight of each channel, and output the second feature matrix;

[0029] The second hidden layer is used to input the second feature matrix, adjust the weights at the spatial position, and output the third feature matrix;

[0030] The object detection head is used to input the third feature matrix and output the bounding box coordinates, depth and confidence of the scratch.

[0031] Furthermore, the specific steps of the first hidden layer include:

[0032] Step S501, performing global average pooling on the first feature matrix of each channel;

[0033] Step S502, using two fully connected layers to calculate the weight of each channel, the first fully connected layer uses a scaling factor to compress the number of channels, and the second fully connected layer restores it to the original number of channels;

[0034] Step S503: weighting the first feature matrix of each channel by the weight of the channel.

[0035] Furthermore, the specific steps of the second hidden layer include:

[0036] Step S601, performing maximum pooling and average pooling on the second feature matrix;

[0037] Step S602, concatenating the results of maximum pooling and average pooling to obtain a feature map with two channels, and generating a spatial attention map through convolution;

[0038] Step S603: Use the spatial attention map to weight the second feature matrix to obtain a third feature matrix.

[0039] Furthermore, a scratch detection model based on YOLO is used to predict the position, depth and confidence of the bounding box where the scratch is located, and the target detection head of the scratch detection model simultaneously predicts k bounding box coordinates, depth and confidence. The bounding boxes of the same prediction batch represent a rectangular unit. The bounding box coordinates include: the coordinates of the center point of the bounding box and width and height The calculation formula of the YOLO target detection head includes:

[0040]

[0041]

[0042]

[0043]

[0044] D s =W d *F d +b d ;

[0045] p obj =sigmoid(t obj );

[0046] where t x and t y Indicates the offset of the center point of the bounding box, c x and c y Indicates the coordinates of the upper left corner of the current rectangular unit, indicating that the predicted center point is the offset relative to the rectangular unit; p w and p h represents the width and height of the predefined anchor box, t w and t h represents the scale of the model prediction relative to the anchor box, D s Indicates the depth of the scratch, W d represents the weight matrix for depth prediction, F drepresents the depth information extracted from the first feature matrix, b d represents the bias parameter for depth prediction, p obj represents the confidence level, t obj Represents the intermediate confidence score output by the object detection head.

[0047] Furthermore, manual annotation is used as sample labels for the scratch detection model.

[0048] Furthermore, the loss function of the scratch detection model is a cross entropy loss function.

[0049] The beneficial effects of the present invention are as follows: the present invention uses three-dimensional vision technology to obtain the RGB image and depth image of the detection object, thereby being able to simultaneously combine the color information and geometric depth information of the surface, and being able to more comprehensively analyze and detect scratches;

[0050] The present invention combines the color information of the RGB image and the geometric information of the depth image through multi-channel fusion technology to generate a multi-channel matrix as the input of the scratch detection model. This fusion method realizes the comprehensive judgment of scratches from the perspectives of color, texture and depth, and improves the accuracy and robustness of the model;

[0051] The scratch detection model of the present invention enhances the model's attention to key feature channels through operations such as maximum pooling, average pooling, and weight adjustment, helping the model to focus on areas where scratches may appear in the spatial dimension, thereby enhancing more important channels in the scratch detection task and improving the model's scratch detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of a vehicle body scratch detection method of the present invention;

[0053] Figure 2 It is a flow chart of preprocessing RGB images and depth images of the present invention. DETAILED DESCRIPTION

[0054] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the contents of this specification. Each example may omit, replace or add various processes or components as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0055] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in one or more embodiments of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0056] like Figure 1-Figure 2 As shown, a vehicle body scratch detection method comprises the following steps:

[0057] Step S101, collecting an RGB image and a depth image of a detection object through three-dimensional vision technology;

[0058] Step S102, preprocessing the RGB image and the depth image;

[0059] Step S103, performing data fusion on the preprocessed RGB image and the depth image based on a multi-channel fusion method to obtain a multi-channel matrix, and constructing a scratch detection model, the input of the scratch detection model is the multi-channel matrix, and the output is a scratch result sequence, and a sequence unit of the result sequence includes: scratch boundary box coordinates, scratch depth and confidence;

[0060] Step S104: classify the severity of the scratches according to the result sequence of the scratches output by the scratch detection model, and take different measures.

[0061] In one embodiment of the present invention, the RGB image reflects the color and texture information of the car and is collected using a high-definition camera, and the depth image reflects the depth of scratches on the car body and is collected using a structured light camera; the RGB image and the depth image need to be captured simultaneously through synchronous collection of the device.

[0062] In one embodiment of the present invention, the RGB image and the depth image are preprocessed in step S102, and the specific steps include:

[0063] Step S201, scaling the RGB image by a bilinear interpolation method according to the resolution of the depth image to make it consistent with the size of the depth image;

[0064] Step S202, converting the RGB image into a grayscale image;

[0065] Step S203, using a Gaussian filter algorithm to smooth the RGB image, and using a median filter algorithm to denoise the depth image;

[0066] Step S204, adjusting the contrast of the RGB image using histogram equalization;

[0067] Step S205, normalizing the depth image using a z-score normalization method;

[0068] Step S206: perform image calibration on the RGB image and the depth image.

[0069] In one embodiment of the present invention, the step of calibrating the RGB image and the depth image in step S206 includes:

[0070] Step S301, calibrate the internal and external parameters of the camera that collects the RGB image and the depth image. The internal parameters are used to describe the internal optical characteristics of each camera, including focal length, principal point position and distortion coefficient; the external parameters are used to describe the position and posture relationship between the two cameras, including rotation matrix and translation vector, which represent the spatial position relationship between the two cameras relative to each other; the calibration can be completed by shooting a calibration image with known size and features (such as a checkerboard image);

[0071] Step S302, aligning the RGB image and the depth image through perspective transformation and pixel mapping; perspective transformation refers to transforming the coordinate system of the RGB camera into the coordinate system of the depth camera according to the relative posture of the two cameras, so that the two can be aligned in the same space; pixel mapping refers to using the calibrated camera parameters to transform the coordinates of each pixel in the RGB image and map them to the corresponding pixel points in the depth image;

[0072] Step S303, performing distortion correction on the RGB image and the depth image according to the intrinsic parameters of the camera; specifically, the distortion correction can be performed using the function of OpenCV according to the distortion coefficient obtained during camera calibration.

[0073] In one embodiment of the present invention, the step of fusing the preprocessed RGB image and the depth image based on a multi-channel fusion method in step S103 includes:

[0074] Step S401, extracting the red, green, and blue channels of each pixel of the RGB image, and generating three corresponding values: R(i, j), G(i, j), and B(i, j);

[0075] Step S402, extracting depth information D(i, j) of the depth image, indicating the distance from the pixel to the camera;

[0076] Step S403, combine the three channels (R, G, B) of the RGB image and the single channel D of the depth image to form a four-channel matrix 9R, G, B, D); each pixel will have four eigenvalues: [R(i, j), G(i, j), B(i, j), D(i, j)]; for example, the size of the RGB image is: H*W*3, the size of the depth image is: H*W*1, and the fused four-channel matrix is: H*W*4.

[0077] In one embodiment of the present invention, the scratch detection model includes: a convolutional layer, a first hidden layer, a second hidden layer and an object detection head;

[0078] The convolution layer is used to input the multi-channel matrix, extract the features of the multi-channel matrix, and output a first feature matrix;

[0079] The first hidden layer is used to input the first feature matrix, adjust the weight of each channel, and output the second feature matrix;

[0080] The second hidden layer is used to input the second feature matrix, adjust the weights at the spatial position, and output the third feature matrix;

[0081] The object detection head is used to input the third feature matrix and output the bounding box coordinates, depth and confidence of the scratch.

[0082] The first hidden layer enhances the model's attention to key feature channels through global average pooling and weight adjustment. The information of each channel is weighted so that more important channels in the scratch detection task are enhanced; the second hidden layer generates a spatial attention map through maximum pooling and average pooling operations to help the model focus on areas where scratches may appear in the spatial dimension. The attention map weights each spatial position (pixel point).

[0083] In one embodiment of the present invention, the specific steps of the first hidden layer include:

[0084] Step S501, perform global average pooling on the first feature matrix of each channel, and the calculation formula is: where s C represents the global information of the channel, H and W represent the height and width of the first feature matrix respectively, X C (i, j) represents the value of channel C at position (i, j);

[0085] Step S502, use two fully connected layers to calculate the weight of each channel. The first fully connected layer uses a scaling factor to compress the number of channels, and the second fully connected layer restores it to the original number of channels. The calculation formula is: w = sigmoid (W2*ReLU (W1*s C)), where w represents the weight of each channel, W1 represents the reduced-dimensional weight matrix, and its size is W2 represents the dimension-enhanced weight matrix, which is sigmoid represents the sigmoid activation function;

[0086] Step S503, weighting the first feature matrix of each channel by the weight of the channel, and the calculation formula is: Represents the weighted second feature matrix.

[0087] In one embodiment of the present invention, the specific steps of the second hidden layer include:

[0088] Step S601, performing maximum pooling and average pooling on the second feature matrix, the calculation formula is: X max =maxX k , where X avg The size is H*W*1, which is the average value map of each pixel, X max is the maximum value map of each pixel;

[0089] Step S602, the results of maximum pooling and average pooling are concatenated to obtain a feature map with two channels, and a spatial attention map is generated by convolution. The calculation formula is: con =concat(X avg , X max ), M s =sigmoid(Conv(X con )), where X con represents the concatenated feature map, M s represents a spatial attention map, concat represents a concatenation operation, and Conv represents a 1*1 convolution operation;

[0090] Step S603, using the spatial attention map to weight the second feature matrix to obtain a third feature matrix, the calculation formula is: in represents the third characteristic matrix, M s (i, j) represents the spatial attention weight of position (i, j), Represents the value of the second characteristic matrix at this position.

[0091] In one embodiment of the present invention, a scratch detection model based on YOLO is used to predict the position, depth and confidence of the bounding box where the scratch is located, and the target detection head of the scratch detection model simultaneously predicts k bounding box coordinates, depths and confidences. The bounding boxes of the same predicted batch represent a rectangular unit. Preferably, k is set to 3. The bounding box coordinates include: the center point coordinates of the bounding box and width and height The calculation formula of the target detection head of the scratch detection model includes:

[0092]

[0093]

[0094]

[0095]

[0096] D s =W d *F d +b d ;

[0097] p obj =sigmoid(t obj );

[0098] where t x and t y Indicates the offset of the center point of the bounding box, c x and c y Indicates the coordinates of the upper left corner of the current rectangular unit, indicating that the predicted center point is the offset relative to the rectangular unit; p w and p h represents the width and height of the predefined anchor box, t w and t h represents the scale of the model prediction relative to the anchor box, D s Indicates the depth of the scratch, W d represents the weight matrix for depth prediction, F d represents the depth information extracted from the first feature matrix, b d represents the bias parameter for depth prediction, p obj represents the confidence level, t obj Represents the intermediate confidence score output by the object detection head.

[0099] The final output format of the scratch detection model is:

[0100] In one embodiment of the present invention, manual annotation is used as sample labels of the scratch detection model.

[0101] In one embodiment of the present invention, the loss function of the scratch detection model is a cross entropy loss function.

[0102] In one embodiment of the present invention, thresholds are set for grading according to the width and height of the bounding box, the depth of the scratch and the confidence level. Only when the confidence level reaches the threshold, the width and height of the bounding box and the depth of the scratch are judged.

[0103] The above describes an embodiment of the present embodiment, but the present embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present embodiment, ordinary technicians in this field can also make many forms, all of which are within the protection of the present embodiment.

Claims

1. A method for detecting scratches on a vehicle body, characterized in that: The following steps are involved: Step S101, collecting an RGB image and a depth image of a detection object through three-dimensional vision technology; Step S102, preprocessing the RGB image and the depth image; Step S103, performing data fusion on the preprocessed RGB image and the depth image based on a multi-channel fusion method to obtain a multi-channel matrix, and constructing a scratch detection model, the input of the scratch detection model is the multi-channel matrix, and the output is a scratch result sequence, and a sequence unit of the result sequence includes: scratch boundary box coordinates, scratch depth and confidence; Step S104: classify the severity of the scratches according to the result sequence of the scratches output by the scratch detection model, and take different measures.

2. A vehicle body scratch detection method according to claim 1, characterized in that: In step S102, the RGB image and the depth image are preprocessed, and the specific steps include: Step S201, scaling the RGB image by a bilinear interpolation method according to the resolution of the depth image; Step S202, converting the RGB image into a grayscale image; Step S203, using a Gaussian filter algorithm to smooth the RGB image, and using a median filter algorithm to denoise the depth image; Step S204, adjusting the contrast of the RGB image using histogram equalization; Step S205, normalizing the depth image using a z-score normalization method; Step S206: perform image calibration on the RGB image and the depth image.

3. A vehicle body scratch detection method according to claim 2, characterized in that: The step of calibrating the RGB image and the depth image in step S206 includes: Step S301, calibrate the intrinsic and extrinsic parameters of the camera that collects the RGB image and the depth image. The intrinsic parameters are used to describe the internal optical characteristics of each camera, including focal length, principal point position and distortion coefficient; the extrinsic parameters are used to describe the position and posture relationship between the two cameras, including rotation matrix and translation vector, which represent the spatial position relationship between the two cameras relative to each other; Step S302, aligning the RGB image and the depth image through perspective transformation and pixel mapping; perspective transformation refers to transforming the coordinate system of the RGB camera into the coordinate system of the depth camera according to the relative posture of the two cameras, so that the two can be aligned in the same space; pixel mapping refers to using the calibrated camera parameters to transform the coordinates of each pixel in the RGB image and map them to the corresponding pixel points in the depth image; Step S303: performing distortion correction on the RGB image and the depth image according to the intrinsic parameters of the camera.

4. A vehicle body scratch detection method according to claim 1, characterized in that: The step of fusing the preprocessed RGB image and the depth image based on a multi-channel fusion method in step S103 includes: Step S401, extracting the red, green, and blue channels of each pixel of the RGB image, and generating three corresponding values: R(i, j), G(i, j), and B(i, j); Step S402, extracting depth information D(i, j) of the depth image, indicating the distance from the pixel to the camera; Step S403, combine the three channels (R, G, B) of the RGB image and the single channel D of the depth image to form a four-channel matrix (R, G, B, D); each pixel will have four eigenvalues: [R(i, j), G(i, j), B(i, j), D(i, j)].

5. The vehicle body scratch detection method according to claim 1, characterized in that: The scratch detection model includes: a convolutional layer, a first hidden layer, a second hidden layer and a target detection head; The convolution layer is used to input the multi-channel matrix, extract the features of the multi-channel matrix, and output a first feature matrix; The first hidden layer is used to input the first feature matrix, adjust the weight of each channel, and output the second feature matrix; The second hidden layer is used to input the second feature matrix, adjust the weights at the spatial position, and output the third feature matrix; The object detection head is used to input the third feature matrix and output the bounding box coordinates, depth and confidence of the scratch.

6. A vehicle body scratch detection method according to claim 5, characterized in that: The specific steps of the first hidden layer include: Step S501, performing global average pooling on the first feature matrix of each channel; Step S502, using two fully connected layers to calculate the weight of each channel, the first fully connected layer uses a scaling factor to compress the number of channels, and the second fully connected layer restores it to the original number of channels; Step S503: weighting the first feature matrix of each channel by the weight of the channel.

7. A vehicle body scratch detection method according to claim 5, characterized in that: The specific steps of the second hidden layer include: Step S601, performing maximum pooling and average pooling on the second feature matrix; Step S602, concatenating the results of maximum pooling and average pooling to obtain a feature map with two channels, and generating a spatial attention map through convolution; Step S603: Use the spatial attention map to weight the second feature matrix to obtain a third feature matrix.

8. The vehicle body scratch detection method according to claim 1, characterized in that: The scratch detection model built based on YOLO is used to predict the position, depth and confidence of the bounding box where the scratch is located. The target detection head of the scratch detection model predicts k bounding box coordinates, depth and confidence at the same time. The bounding boxes of the same prediction batch represent a rectangular unit. The bounding box coordinates include: the coordinates of the center point of the bounding box and width and height The calculation formula of the YOLO target detection head includes: D s =W d *F d +b d ; p obj =sigmoid(t obj ); where t x and t y Indicates the offset of the center point of the bounding box, c x and c y Indicates the coordinates of the upper left corner of the current rectangular unit, indicating that the predicted center point is the offset relative to the rectangular unit; p w and p h represents the width and height of the predefined anchor box, t w and t h represents the scale of the model prediction relative to the anchor box, D s Indicates the depth of the scratch, W d represents the weight matrix for depth prediction, F d represents the depth information extracted from the first feature matrix, b d represents the bias parameter for depth prediction, p obj represents the confidence level, t obj Represents the intermediate confidence score output by the object detection head.

9. A vehicle body scratch detection method according to claim 1, characterized in that: Manual annotation is used as sample labels for the scratch detection model.

10. The vehicle body scratch detection method according to claim 1, characterized in that: The loss function of the scratch detection model is the cross entropy loss function.

Images