Vehicle paint defect detection system and working method thereof

Abstract

The application provides a whole vehicle paint surface defect detection system, which comprises a hardware device and an edge computer. The hardware device comprises a camera module, an illumination module, a display module and a triggering device. The camera module is used for shooting defect images to realize detection of small defects on the paint surface of the whole vehicle. The illumination module is used for illuminating the paint surface defects to realize high image acquisition quality. The display module is used for displaying images with correct defects. The triggering device is used for realizing automatic image acquisition. The edge computer is used as a transmission center of computing power and data, realizes calling of the hardware device and forwarding of data, and thus realizes control of the whole system. The network structure of the application is improved based on the YOLOv7 algorithm, GAM attention mechanism and SPPFCSP pyramid structure are integrated, the detection precision of the model for detail characteristics and the fusion effect of the model for multi-scale features are enhanced, the detection precision of the model for small targets is improved, and high-precision detection of the paint surface defects is realized.

Description

Technical Field

[0001] This invention relates to the fields of defect detection and machine vision technology, specifically to a whole vehicle paint defect detection system and its working method. Background Technology

[0002] With the introduction of the "intelligent manufacturing" development strategy, the automation and intelligent development of the manufacturing industry has become the core direction of my country's intelligent manufacturing. As a crucial support for industrial production, the level of industrial intelligence in automobile production is a significant step in my country's intelligent manufacturing strategy. In recent years, with the continuous promotion and development of the new energy vehicle market in my country, domestic automobile sales have steadily increased domestically, while exports have also reached new highs. The increase in domestic automobile sales has driven the development of the domestic automobile industry, gradually moving towards the high-end market. Furthermore, the quality of automobile appearance, as a crucial aspect of automobile quality control, and the quality of automobile paint, being the most direct indicator of automobile appearance quality, are undeniably important. This not only relates to the level of product appearance quality indicators but also reflects the production attitude of the enterprise.

[0003] As one of the most influential factors in a car's overall appearance, good paint quality can enhance consumer perception of the product and improve brand image. In automobile production, the inspection line is the final production line to check the appearance quality of vehicles before they leave the factory. Its control over appearance quality directly impacts the overall quality control level of the product. Higher appearance quality control can effectively increase car sales. And with increasing automation, the level of automated detection for automotive paint defects is constantly improving and developing.

[0004] In the field of paint defect detection, manual inspection and traditional methods still dominate. Traditional methods primarily utilize image processing techniques, processing grayscale data to detect paint defects. Common traditional methods can be categorized as follows: threshold segmentation detection under bright field, bright and dark field scattering detection, phase deflection detection, and 3D scanning detection. These traditional methods remain dominant in automotive defect detection. However, with the rise of computer vision, traditional methods are limited by their high cost and complex systems. Furthermore, traditional methods often rely on classifiers to detect defects, thus having limitations. They cannot meet the demands for high-precision, high-efficiency detection of paint defect locations and the simultaneous presence of multiple defect types. Image processing-based methods also suffer from high requirements for lighting conditions, expensive equipment, and limited detection areas. Deep learning-based image detection algorithms address these issues to some extent. Currently, mainstream detection algorithms include two-stage and one-stage algorithms. While two-stage algorithms have seen some industrial applications and validation, their detection speed has remained limited, hindering their industrial development. One-stage algorithms, with their end-to-end network structure, have significantly improved detection speed and gradually surpassed two-stage algorithms in accuracy, becoming the mainstream approach. Typical examples include the SSD and YOLO series. YOLOv1 transformed bounding box selection into a regression problem, but its detection accuracy was not outstanding. However, its faster detection speed garnered attention, leading to subsequent iterations such as YOLOv2, YOLOv3, YOLOv4, YOLOv5, and the recently released YOLOv7. As the state-of-the-art (SOTA) model of its time, YOLOv7 achieved detection speeds ranging from 5 to 160 FPS, surpassing existing known object detection networks in both speed and accuracy, and achieving a peak detection accuracy of 56.8% on the COCO dataset. The high accuracy and speed of the YOLOv7 network meet the inspection needs of industrial production lines, enabling rapid paint surface inspection and real-time display of vehicles on the production line. However, because the YOLO network abandons the use of target candidate boxes and transforms the target bounding box localization into a regression problem, the YOLO network tends to overlook feature extraction for small targets, resulting in low detection accuracy for small target defects. Currently, most automotive paint surface defects are concentrated in the range of several centimeters to several hundred micrometers, which brings difficulties to the detection of automotive paint surface defects.

[0005] In response to the problems existing in the current technology, there is an urgent need for a more sophisticated vehicle paint defect detection system that can achieve high-precision detection of even small defects on the car paint surface. Summary of the Invention

[0006] To address the problems and shortcomings of existing technologies, this invention adopts the latest YOLOv7 target detection method, whose excellent performance ensures the real-time performance of the system. Furthermore, this application is geared towards the practical scenario of automotive paint defect detection. Based on the YOLOv7 algorithm, its network structure is improved by incorporating the GAM attention mechanism and the SPPFCSP pyramid structure, which enhances the model's detection accuracy for detailed features and the fusion effect of multi-scale features, improves the model's detection accuracy for small targets, and achieves high-precision detection of paint defects.

[0007] To achieve the above objectives, the present invention proposes a vehicle paint defect detection system, including hardware devices and an edge computer. The hardware devices include a camera module, a lighting module, a display module, and a triggering device.

[0008] The camera module is used to capture defect images to detect minute defects in the paint surface of the entire vehicle.

[0009] The lighting module is used to illuminate defects in the car paint to achieve high image acquisition quality;

[0010] The display module is used to display images with correct defects;

[0011] The triggering device is used to achieve automatic image acquisition;

[0012] The edge computer serves as a transmission center for computing power and data, enabling the invocation of hardware devices and the forwarding of data, thereby achieving control over the entire system.

[0013] To address the issues of small defect features and glare in automotive paint surface inspection, this application designs and builds a paint surface defect detection system based on an actual automotive production line to obtain clear defect feature images. To achieve automated design for automotive paint surface defects, the defect detection system is modularly designed. The entire paint surface defect detection system needs to realize sensor triggering and reception, camera access and image storage, VIN identification and forwarding, and image recognition and forwarding.

[0014] Preferably, the lighting module includes an LED diffuse reflective surface light source and a light shield;

[0015] The LED diffuse surface light source is used in conjunction with the coaxial light shooting method to reduce the occurrence of reflected light spots;

[0016] The light-shielding plate is used to improve the consistency of the light source, so as to reduce the impact of changes in the external light source on the paint surface inspection system.

[0017] Preferably, the camera module includes at least eight high-resolution industrial cameras.

[0018] Preferably, the triggering device includes two high-precision medium-sized distance sensors; after the two medium-sized distance sensors receive the trigger signal simultaneously, they can transmit the rising edge signal to the edge computer and realize the automatic acquisition of paint defect images by configuring camera SDK functions.

[0019] Preferably, the acquired paint defect image categories include bubbles, dirt spots, fibers, scratches, and pinholes; the acquired image data is augmented to improve the ability to extract paint defect features, and the data augmentation methods include image rotation, image mirroring, and changes in image brightness and contrast.

[0020] Preferably, it also includes a programming module, which includes an API interface file, an OCR recognition algorithm file, and an improved YOLOv7 detection algorithm file;

[0021] The API interface file enables the camera module to be invoked: a local area network was built between the camera module and the edge computer to enable separate invocation of eight cameras; data from the eight cameras is collected through a switch based on their IP addresses; simultaneous transmission of data from multiple cameras is achieved through 10 Gigabit fiber optic cable, which ensures complete transmission of images from all eight cameras without frame loss or data corruption; the simultaneous triggering of the eight cameras is accomplished using a Python thread pool, which calls the cameras separately based on their IP addresses.

[0022] The API interface can be used to acquire and recognize VIN code images: by calling the corresponding camera, it acquires the image through a trigger signal, and then calls the OCR recognition algorithm file to recognize the image. The OCR recognition algorithm uses the open-source Baidu PaddleOCR algorithm model to improve the detection accuracy of Chinese characters. After the OCR algorithm recognizes the image, it matches and verifies the recognition result with the MES system. If it is correct, the image will be allowed to be displayed; otherwise, the transmitted data will be recorded as an error.

[0023] The improved YOLOv7 detection algorithm specifically employs the YOLOv7-w6 network model, which has four detection heads. The network input image yields feature matrices of sizes 80×80×256, 40×40×512, 20×20×768, and 10×10×1024. The YOLOv7-w6 network model adopts the PANet structure to further fuse images of different sizes, and also incorporates SPPCSPC, ELAN, and CBS modules, improving the overall performance and balance of the network model.

[0024] Preferably, the CBS is a basic convolutional module, which consists of convolutional layers, batch normalization layers, and SiLU activation functions; the CBS basic convolutional module includes three types: convolutional kernel size of 1×1 with a stride of 1, convolutional kernel size of 3×3 with a stride of 1, and convolutional kernel size of 3×3 with a stride of 2, wherein the convolutional kernel with a stride of 2 is used to replace the pooling layer to achieve downsampling of the image;

[0025] The ELAN module is mainly composed of CBS modules. The ELAN module adopts a CSP structure for module construction. The input image is divided into two parts for convolution. One part is directly connected to the connection layer after convolution, and the other part is fed into the connection layer by two CBS modules as a group, and finally passes through another CBS module. The CSP structure can effectively reduce redundant gradient information and improve the learning ability of the network.

[0026] The SPPCSPC module also adopts the CSP structure and combines the pyramid pooling layer and the CBS module. The pyramid pooling layer uses a parallel connection of pooling layers of sizes 5×5, 9×9 and 13×13 to improve the multi-scale fusion capability of the entire model.

[0027] Preferably, the improved YOLOv7 detection algorithm uses the K-means++ clustering method to reselect the prior boxes. The K-means++ clustering method clusters the prior box data based on the size of the labeled anchor boxes. First, a fixed number of initial cluster centers are set. Using the IOU metric, all samples are assigned to nearby cluster centers. Then, the data is updated and iterated according to the average width and height of the boxes in each cluster until the optimal box size is obtained. The calculation formula is shown in the following formula (1):

[0028]

[0029] In the formula, Let i be the size of the region containing the i-th labeled anchor box in the dataset; is the region size of the j-th cluster center anchor; n is the number of detected targets in the dataset; k is the number of cluster center anchors.

[0030] Preferably, the improved YOLOv7 detection algorithm also incorporates a GAM attention mechanism; the GAM redesigns the Channel Attention and Spatial Attention sub-modules of CBAM; in the Channel Attention sub-module, the input data is first transformed into different dimensions, then converted back to its original dimensions after passing through an MLP, and finally processed by a Sigmoid function for output, thus achieving the fusion of information from different dimensions through dimensional changes; in the Spatial Attention sub-module, a convolutional approach is used, first convolving the input data, then compressing the number of data channels, then passing it through a 7×7 convolution kernel, increasing the number of data channels to maintain consistency, and finally processing it by a Sigmoid function for output;

[0031] Furthermore, the ELAN module is improved: based on the CSP structure, the number of convolutional layers in the ELAN module is further reduced, and it is named ELAN-S. The ELAN-S module can divide the data into two parts, each passing through a 1×1 convolutional kernel. One of the parts then passes through a 3×3 convolutional kernel, and then a concat operation is performed before inputting it into a 1×1 convolution to complete the processing of the entire data.

[0032] The working method of the vehicle paint defect detection system includes the following steps:

[0033] Step 1: When the car arrives at the sensor sensing area, using the single-point trigger mode under the dual-sensor strategy, when both sensors receive the trigger signal at the same time, the signal is sent back to the PLC for logic processing.

[0034] Step 2: After receiving the sensor signal, the PLC processes the logic to determine the position status of the vehicle body, vehicle, and production line, sends out the RFID reading signal, reads complete vehicle information and obtains real-time production line position information, and binds the RFID to the production queue.

[0035] Step 3: When the GTI subscription queue information changes, the edge computer is triggered to call the camera SDK file via the API interface to collect paint surface images at various vehicle body points. The camera module is invoked, and a local area network is built between the cameras and the edge computer to enable separate calls to the cameras, distinguishing them by IP address. Data is collected from multiple cameras via a switch, and then transmitted simultaneously via 10 Gigabit fiber optic cable. The 10 Gigabit fiber optic cable ensures complete transmission of images from all eight cameras without frame drops or data loss. Simultaneous camera triggering is accomplished using a Python thread pool, calling the cameras separately based on their IP addresses.

[0036] Step 4: After image acquisition, perform OCR recognition algorithm on the image acquired by the camera that obtained the vehicle VIN according to the camera IP, and match and verify the recognition result with the VIN number format information in the MES system;

[0037] Step 5: If the verification result is correct, the improved YOLOv7 detection algorithm model is called to identify and label the images one by one, the identification results are processed, and the images and detection results are forwarded to the MES system and the images are allowed to be displayed. Otherwise, the transmitted data will be recorded as errors.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] 1. This application has minimal impact on existing production lines and will not be difficult to implement due to stringent environmental requirements.

[0040] 2. This application employs the latest YOLOv7 object detection method, which surpasses traditional detection methods in both detection speed and accuracy. This superior performance ensures the system's real-time performance. Furthermore, this application primarily addresses the insufficient accuracy issues in automotive paint defect detection by proposing an improved YOLOv7 algorithm. To address the difficulty in improving the accuracy of small-scale defects such as pinholes, this application incorporates the GAM attention mechanism and the SPPFCSPC module to enhance the extraction capability of defect features. Simultaneously, an improved ELAN module is used to enhance the model's ability to detect small-scale defects.

[0041] 3. The average detection accuracy of the improved YOLOv7 algorithm in this application reaches 88.9%, which is 17% higher than that of the original YOLOv7 algorithm, effectively improving the detection accuracy of the model for small-sized defects. Attached Figure Description

[0042] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the sensor location distribution according to the present invention;

[0044] Figure 2 This is a schematic diagram of the sensor triggering method of the present invention;

[0045] Figure 3 This is a schematic diagram of the workflow of the vehicle paint defect detection system of the present invention;

[0046] Figure 4 This is a network structure diagram of the YOLOv7 target detection algorithm of the present invention;

[0047] Figure 5 This is the CBS module of the present invention;

[0048] Figure 6 This is a schematic diagram of the structure of the improved ELAN-S module of the present invention;

[0049] Figure 7 This is a schematic diagram illustrating the paint defect detection accuracy under different numbers of ELAN-S according to the present invention;

[0050] Figure 8 This is a schematic diagram illustrating the working principle of the vehicle paint defect detection system of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0052] I. Hardware Setup

[0053] This embodiment proposes a paint defect detection system, including hardware devices and a programming module. The hardware devices include a camera module, an illumination module, a display module, and a triggering device. The illumination system is mainly used to illuminate automotive paint defects. To achieve high image acquisition quality, an LED diffuse surface light source is selected, and a coaxial light acquisition method is adopted. This effectively reduces the occurrence of reflected light spots, improves image acquisition quality, and reduces the occlusion of paint defects by light spots. In addition, to reduce the influence of external light sources, this embodiment uses a light shield to improve the consistency of the light source, effectively reducing the impact of changes in external light sources on the paint detection system, reducing the difficulty of adjusting the light source, and facilitating compatibility with vehicles of different colors. The camera module uses a Basler industrial camera with 20 megapixels, which can detect minute defects. The entire detection system is equipped with a total of 8 high-pixel cameras.

[0054] This embodiment selects equipment based on the size of the paint surface defects and the detection distance, as shown in Table 1.1.1 below. Table 1.1.1 Equipment Configuration Parameter Table

[0055]

[0056] II. Software System Setup and Data Collection

[0057] In order to obtain clear defect feature images, this embodiment designs and builds a paint defect detection system based on an actual automobile production line to address the problems of small defect features and easy reflection in automobile paint inspection.

[0058] To achieve automated design for automotive paint defects, the defect detection system was modularly designed. The entire paint defect detection system in this embodiment can realize sensor triggering and reception, camera access and image storage, VIN identification and forwarding, and image identification and forwarding.

[0059] 2.1 Installation and debugging of the triggering device:

[0060] Since the acquisition equipment platform is deployed on an automated production line, it needs to achieve automatic image acquisition. Therefore, this embodiment adopts a distance sensor triggering method, using two medium-sized distance sensors as triggering devices, model SickDT35-B15851, with an accuracy of ±10mm. When the two sensors receive the trigger signal simultaneously, they transmit the rising edge signal to the edge computer, and automatically acquire the paint defect image by configuring the camera SDK function.

[0061] Based on the environmental characteristics of the commodity inspection line, this embodiment uses a distance sensor as the triggering device. The sensor is connected to the PLC system of the production line. When the car passes the trigger point on the conveyor belt, the sensor receives and transmits the signal, which is then transmitted by the PLC to the Andon system. The Andon system processes the signal and transmits it to the GTI platform. The GTI platform then transmits the signal to the edge computer to realize the transmission of the trigger signal.

[0062] The sensor used is the SICK DT35 mid-range distance sensor, with parameters shown in Table 2.1.1. It can detect objects within a range of 0.05m to 12m, and features fast response time, strong resistance to environmental interference, and the ability to avoid the influence of background light. It also has three trigger modes: single-point mode, window mode, and background mode. In single-point mode, when an object appears within the sensor's receiving range, the sensor's voltage level changes from low to high, and returns to low when the object disappears. Window mode and background mode are similar, both using pulse levels. The former remains at a low level, generating a rising pulse signal when an object enters the range, while the latter remains at a high level, generating a falling pulse signal when an object appears.

[0063] Table 2.1.1 Sensor Parameters

[0064] model resolution Rereading precision Measurement accuracy Detection distance SICKDT35 0.1mm 0.5-5mm -10mm-10mm 0.5-12m

[0065] To improve sensor stability and reduce the frequency of false triggers, and to prevent false triggering caused by workers passing by, this embodiment employs a dual-sensor strategy. That is, signals are only transmitted when both sensors trigger simultaneously. The sensors are installed on both sides of the vehicle's front, such as... Figure 1 As shown.

[0066] This embodiment originally used a window-triggered mode: it acquired the rising edge signal, and the detection range was the edge of the car's front bumper and hood, such as... Figure 2 As shown in (a), the sensor is triggered to transmit a signal when a car passes by. However, during the trial operation of this scheme, it was found that the sensor had multiple false triggering phenomena. The sensor would generate trigger signals when passing the windshield and the rear of the car. The multiple false triggering signals seriously affected the accuracy of image acquisition. At the same time, the triggering signal at the rear would cause the image processing of the next car to be delayed, resulting in missed detection. This affected the stability of the automatic image acquisition scheme. Analysis revealed that the main reason was that the transparent windshield allowed the sensor to penetrate and detect the car floor, thus causing false triggering. Similarly, the rear of the car was caused by the rear bumper. These two false triggerings could not be processed from the detection end, so this scheme was abandoned.

[0067] This embodiment ultimately adopts a single-point mode to acquire the rising edge signal transmitted by the sensor, such as... Figure 2As shown in (b), when a vehicle enters the detection range, the sensor signal changes from low to high and remains high until the vehicle leaves the detection range. This ensures that only one rising edge signal is generated for the entire vehicle. While this scheme also exhibits false triggering, there is no false triggering at the rear of the vehicle, which does not affect the detection of the next vehicle, thus resolving the issue of missed detections. For false triggering of the windshield and sunroof in this scheme, a dual strategy of trigger delay and OCR recognition is employed. The delay triggering mainly prevents the generation of a trigger signal for the next 3 seconds after a high level is generated, ensuring the signal is unaffected by false triggering. For false triggering of the sunroof, the format of the VIN code obtained by OCR recognition is judged to determine whether the correct VIN information has been acquired, thereby obtaining the corresponding image. This dual strategy effectively improves the detection accuracy, prevents the impact of false triggering, and achieves accurate acquisition of vehicle paint surface images.

[0068] 2.2 Creation of paint surface defect dataset:

[0069] This embodiment focuses on image acquisition of five types of defects (bubbles, dirt spots, fibers, scratches, and pinholes) that frequently appear on the surface of automobile bodies and have a significant impact on vehicle quality. Most of these defects are characterized by their small size and similarity to the background color. Pinhole defects, in particular, vary greatly in color and shape due to differences in painting processes and materials used on vehicles of different colors. Therefore, pinhole defects are separated. Training data confirms that the defect detection accuracy is improved after separation. However, when displayed on the front end, they are logically reclassified into one category.

[0070] Meanwhile, to improve the model's learning performance, data augmentation is necessary. Data augmentation can effectively improve the network's robustness, enhance the algorithm's ability to extract defect features, and prevent overfitting. The data augmentation methods used in this embodiment include image rotation, image mirroring, and changes in image brightness and contrast. The original dataset contained 1341 images, which were augmented to obtain 4023 images. The LabelImg tool was used for image annotation, thus creating a dataset of paint surface defects.

[0071] 2.3 Design of Paint Defect Detection System

[0072] The programming module of the paint defect detection system is mainly divided into several parts: API interface files, OCR recognition algorithm files, and improved YOLOv7 detection algorithm files. The API interface files are the primary files in the programming, calling multiple modules to implement the invocation of multiple functions and the forwarding of detection results, such as... Figure 3 As shown.

[0073] In this embodiment, to enable the use of the camera module, a local area network (LAN) was established between the cameras and the edge computer, differentiated by IP address. Data from multiple cameras is collected via a switch, and simultaneous transmission of data from multiple cameras is achieved through a 10 Gigabit fiber optic cable. The 10 Gigabit fiber ensures complete transmission of images from all eight cameras without frame drops or data loss. Simultaneous triggering of the cameras is handled using a Python thread pool, with calls made based on IP address to achieve simultaneous camera access. For the acquisition and recognition of the VIN code image, the camera corresponding to the rear triangular window is primarily responsible. It acquires the image via a trigger signal and then calls an OCR recognition algorithm for recognition. The OCR uses the open-source Baidu PaddleOCR algorithm model, which can recognize most text, especially Chinese characters, exhibiting high detection accuracy. After the OCR algorithm recognizes the rear triangular image, the recognition result is matched and verified against the MES system. If correct, the image is allowed to be displayed; otherwise, the transmitted data is recorded as an error. To call the algorithm, the API interface file of the algorithm model needs to be modified to make calling the model more convenient and faster. The algorithm model identifies and labels the images one by one, processes the recognition results, and forwards the images and detection results to the MES system.

[0074] 2.3.1 Improved YOLOv7 algorithm file

[0075] This embodiment primarily uses the YOLOv7-w6 network model from the YOLOv7 series, such as... Figure 4 As shown, due to the large size range of the dataset, this embodiment uses a network model with 4 detector heads. The network input image can obtain feature matrices of sizes 80×80×256, 40×40×512, 20×20×768 and 10×10×1024. In order to further fuse images of different sizes, this embodiment adopts the PANet (Path Aggregation Network) structure, but improves and adds modules to the network, adding SPPCSPC, ELAN-2 and CBS modules to improve the overall performance and balance of the network.

[0076] In the improved YOLOv7 network structure of this embodiment, CBS is the basic convolutional module, consisting of convolutional layers, batch normalization (BN) layers, and SiLU activation functions, as follows. Figure 5As shown, there are three types of basic convolutional modules in the YOLOv7 algorithm: 1×1 kernel size with a stride of 1, 3×3 kernel size with a stride of 1, and 3×3 kernel size with a stride of 2. Different kernel sizes and strides have different functions. The convolution with a stride of 2 is mainly used to replace pooling layers to achieve downsampling of the image. Compared with pooling layers, convolutional downsampling can learn downsampling parameters, optimize the feature extraction ability during downsampling, and improve the performance of the model.

[0077] The ELAN module is mainly composed of CBS modules and uses a CSP (Cross Stage Partial) structure to build the modules. The input image is divided into two parts for convolution. One part is directly connected to the connection layer after convolution, and the other part is composed of two CBS modules as a group and then fed into the connection layer, and finally fed into another CBS module. In ELAN-W, each CBS module in the branch is fed into the connection layer. The CSP structure can effectively reduce the repetitive gradient information and improve the learning ability of the network.

[0078] The SPPCSPC module also adopts the CSP structure, combining the pyramid pooling layer and the CBS module. The pyramid pooling layer mainly uses a parallel connection of pooling layers of sizes 5×5, 9×9 and 13×13 to improve the multi-scale fusion capability of the entire model.

[0079] 2.3.2 Prior Framework:

[0080] Prior boxes, also known as anchors, are used in most object detection models. The quality of prior box settings greatly affects the performance of object detection. YOLOv7 usually comes with prior box data from public datasets such as COCO, which are suitable for ordinary object detection. However, for different types of objects, such as defects in car paint, most of them are relatively small objects, which will reduce the overall performance of the model. Therefore, this embodiment uses the K-means++ clustering method to reselect prior boxes. It mainly clusters them based on the size of the labeled anchor boxes. First, a fixed number of initial cluster centers are set. Using the IOU metric, all samples are assigned to nearby cluster centers (anchors). Then, the boxes are updated and iterated according to the average width and height of each cluster until the optimal box size is obtained. The calculation formula is as follows: Formula (1):

[0081]

[0082] In the formula, Let i be the size of the region containing the i-th labeled anchor box in the dataset; is the region size of the j-th cluster center anchor; n is the number of detected targets in the dataset; k is the number of cluster center anchors.

[0083] 2.3.3 GAM Attention Mechanism and SPPFCSPC Module

[0084] (1) GAM attention

[0085] In this embodiment, the improved YOLOv7 detection algorithm also incorporates the GAM attention mechanism. GAM attention is an improvement on CBAM attention and also consists of two modules. GAM redesigns the sub-modules ChannelAttention and SpatialAttention of CBAM. Compared with CBAM, GAM takes into account cross-dimensional information, improves the interaction capability in the global dimension, and performs better in detecting small targets.

[0086] In the Channel Attention submodule, GAM attention first transforms the input data into different dimensions, then transforms it back into its original dimensions after passing through an MLP, and finally processes it through a Sigmoid function to output the result. By changing the dimensions, it achieves the fusion of information from different dimensions.

[0087] The Spatial Attention submodule uses convolution, which is more similar to the SE attention mechanism. The input data is first convolved, then the number of data channels is compressed, and then after passing through a 7×7 convolution kernel, the number of data channels is increased to maintain the consistency of the number of data channels. Finally, it is also processed by Sigmoid to output the data. This processing method can improve the fusion ability of data from different dimensions.

[0088] (2) SPPFCSPC module

[0089] The SPPFCSPC module replaces the SPP module with the SPPF module. Compared to the SPP module, it has certain improvements in training and inference speeds, while SPPFCSPC also improves the accuracy of defect detection.

[0090] In this embodiment, during the model improvement and training process, it was found that the detection accuracy for different types of defects was improved to varying degrees, but the detection accuracy for pinhole defects was less than ideal. Through analysis of the model and dataset, it was found that pinhole defect features are small and inconspicuous. In the confusion matrix of the training results, pinhole defects are easily confused with the background, indicating that the model extracts fewer features for pinhole defects, which can easily lead to omissions. To improve the model's detection accuracy for small target defects, the network was modified specifically for pinhole defects. Because the YOLOv7 network model is relatively deep, the small features of pinhole defects are easily lost during convolution, resulting in insufficient extraction of feature information. Therefore, to improve the accuracy for pinhole defects, the ELAN module was improved by reducing the number of convolutional layers in the ELAN module, while still using the CSP structure, and named ELAN-S. The module structure is as follows. Figure 6 As shown, the ELAN-S module divides the data into two parts, each passing through a 1×1 convolution kernel. One of the parts then passes through a 3×3 convolution kernel, and a concat operation is performed before the data is input into a 1×1 convolution kernel to complete the processing of the entire data.

[0091] In this embodiment, the ELAN-S module replaces the original network's ELAN-W module, resulting in a significant performance improvement in the improved model and enhanced detection capability for pinholes.

[0092] 2.4 Usability Verification of the Improved Algorithm Model

[0093] To verify the effectiveness of the improved algorithm, this embodiment conducted ablation experiments on the improved module and comparative experiments with commonly used algorithms, and also conducted experiments on the number of ELAN-S modules that could be replaced. The experimental environment is shown in Table 2.4.1 below. All algorithms used in the experiments were developed using Python, based on the PyTorch framework, and run on Ubuntu.

[0094] Table 2.4.1 Experimental Environment Parameters

[0095]

[0096] 2.4.1 Evaluation Criteria

[0097] The evaluation metrics used in this embodiment are precision (P), recall (R), average precision (AP), and mean average precision (mAP). The definitions of P, R, and AP are as follows:

[0098]

[0099]

[0100]

[0101]

[0102] In the formula, TP represents the positive sample of the predicted result, indicating a correct result (true positive); FP represents the positive sample of the predicted result, indicating an incorrect result (false positive); FN represents the negative sample of the predicted result, indicating an incorrect result (false negative); P(r) is the PR curve; mAP is the average detection precision, among which the more common ones are mAP@0.5 and mAP@0.5:0.95. mAP@0.5 refers to the average precision when the confidence threshold is 0.5, and mAP@0.5:0.95 refers to the average mAP from different thresholds from 0.5 to 0.95.

[0103] 2.4.2 Experimental and Comparative Results

[0104] In this embodiment, the collected dataset is divided into training set, validation set and test set according to the ratio. The data is randomly allocated in a ratio of 8:1:1. The total number of datasets is 4023 images, and the data of each type of image is shown in Table 2.4.2.

[0105] Table 2.4.2 Dataset of Paint Defects

[0106]

[0107] (1) Experiment comparing the number of ELAN-S modules

[0108] To address the characteristics of pinhole defects, this embodiment designs an ELAN-S module. The pruned ELAN module reduces the overall number of network layers, avoiding feature data loss caused by excessive network depth. To obtain a better network model, comparative experiments were conducted on the number of ELAN-S modules. ELAN-W modules were replaced, with two modules per group. To test the appropriateness of the replacement number, the last two ELAN modules in the network backbone were also replaced. During the experiments, the number of ELAN-S modules was set to 4, 6, and 8, with 400 iterations per iteration. The input image size was 1280×1280, the batch size was 8, the momentum was 0.937, the learning rate was 0.01, and the weight decay was 0.0005. The paint defect detection accuracy under different numbers of ELAN-S modules is as follows: Figure 7 As shown.

[0109] from Figure 7As can be seen, when the number of ELAN modules replaced is 6, the model's detection accuracy is the highest at 88.9%. When the number increases or decreases, the model's detection accuracy fluctuates significantly, with the most obvious change in the detection accuracy of pinhole defects. The highest detection accuracy is 0.741, which is about 10% higher than the other detection accuracies. The detection accuracy of pinhole defects has been significantly improved, verifying the effectiveness of the module. While reducing the number of network layers, the detection accuracy of defects has been improved, and the features of small pinhole defects have been preserved. Therefore, the method of replacing all ELAN-W modules is adopted as the network structure of the model.

[0110] (2) Ablation test

[0111] This embodiment mainly focuses on improving the head part of the network. In order to verify the effectiveness of the improved module, ablation experiments were conducted on different modules, and the comparison results are shown in Table 2.4.3.

[0112] Table 2.4.3 Ablation experiments of the improved YOLOv7 algorithm

[0113]

[0114]

[0115] As shown in Table 2.4.3 above, the network's detection performance improved after introducing different modules. The GAM attention mechanism increased the model's detection accuracy by 9.2%, enhancing its attention to small-scale defects and improving its feature extraction capabilities. This indicates that the attention mechanism significantly improves the detection accuracy of the paint defect dataset. Adding the SPPFCSPC module further enhanced the model's fusion ability across different scales, further improving its detection capabilities. After analyzing the network model, the improved ELAN module further improved the detection accuracy by 5.9%, reaching 88.9%. This effectively reduced the loss of small-sized defect features and decreased the confusion with the background, demonstrating the effectiveness of this module in improving the detection accuracy of paint defects.

[0116] To further verify the improvement effect of each module on the network, heatmap visualizations were created for networks with different modules introduced, thus more clearly demonstrating the performance improvement of each module on the network.

[0117] With the addition of improved modules to the network, the model's sensitivity to defects has been significantly enhanced. After adding GAM attention, the fineness of the region of interest has been improved. After adding SPPFCSPC and ELAN-S modules, the model is more accurate in detecting key defect regions. At the same time, the robustness of the network is also gradually improved, effectively reducing the possibility of false detections.

[0118] This embodiment demonstrates the detection effect of each type of defect. The smallest defect is about 0.5mm. At the same time, some defects are very similar to the background color of the vehicle body, which significantly increases the difficulty of defect detection.

[0119] (3) Algorithm Comparison Experiment

[0120] The improved YOLOv7 model achieved good detection results. To further verify the model's detection performance, different algorithms were used to train and detect on the paint surface dataset. The comparison algorithms included Faster-RCNN, YOLOv4, YOLOR, and YOLOv7, with Faster-RCNN using ResNet50 as the backbone network. All models were developed based on the PyTorch framework and experiments were conducted on the same dataset and hardware configuration. Performance comparison experiments were performed using mAP@0.5 and mAP@0.5:0.95 as metrics, and the results are shown in Table 2.4.4.

[0121] Table 2.4.4 Comparison results of different detection methods

[0122]

[0123] As can be seen from Table 2.4.4, the YOLOv4 algorithm has the lowest accuracy and detection efficiency, with an accuracy of 70.6%, while the Faster-RCNN algorithm has an accuracy of 80.9%, which is higher. However, its detection efficiency is lower and it cannot achieve real-time detection of paint defects. The YOLOR algorithm has a similar detection accuracy to the Faster-RCNN algorithm, but it still has a certain gap in detection accuracy compared to the improved YOLOv7 model in this embodiment.

[0124] 2.4.3 Conclusion

[0125] This embodiment addresses the issue of insufficient accuracy in automotive paint defect detection by proposing an improved YOLOv7 algorithm. To tackle the difficulty in improving the detection accuracy of small-scale defects such as pinholes, the algorithm incorporates a GAM attention mechanism and an SPPFCSPC module to enhance the extraction of defect features. Simultaneously, an improved ELAN module is used to enhance the model's ability to detect small-scale defects. The model's performance was validated using a self-collected and created automotive paint dataset. The results show that the improved YOLOv7 algorithm achieves an average detection accuracy of 88.9%, a 17% improvement compared to the original YOLOv7 algorithm, effectively enhancing the model's detection accuracy for small-sized defects.

[0126] 2.5 Implementation of the Paint Surface Defect Detection System

[0127] as follows Figure 8 As shown, the system establishes a link for image acquisition, detection, recognition, and display. An edge computer acts as the computing power and data transmission center, enabling the access to hardware devices and data forwarding, thereby controlling the entire system. The entire system achieves image acquisition and real-time detection, fulfilling the defect detection requirements of a commercial inspection line.

[0128] Step 1: When the car arrives at the sensor sensing area, the single-point triggering mode under the dual-sensor strategy (the principle is detailed in 2.1) is used. When both sensors receive the trigger signal at the same time, the signal is sent back to the PLC for logic processing.

[0129] Step 2: After receiving the sensor signals, the PLC processes the signals logically to determine the presence of the vehicle body, vehicle, and production line. It then sends out RFID reading signals to read complete vehicle information and obtain real-time production line location information, and binds the RFID tags to the production queue.

[0130] Step 3: When the GTI subscription queue information changes, the edge computer is triggered to call the camera SDK file via the API interface to collect paint surface images at various vehicle body points. In the camera invocation module, a local area network (LAN) was built between the cameras and the edge computer. To enable separate calls to the cameras, they are distinguished by IP address. Data collection from multiple cameras is achieved through a switch, and then simultaneous transmission of data from multiple cameras is realized through 10 Gigabit fiber optic cable. The 10 Gigabit fiber optic cable ensures complete transmission of images from all eight cameras without frame drops or data loss. Simultaneous triggering of the cameras is accomplished using a Python thread pool, which calls the cameras separately based on their IP addresses, enabling simultaneous access to each camera.

[0131] Step 4: After image acquisition, perform OCR recognition algorithm on the image acquired by the camera that obtained the vehicle VIN according to the camera IP, and match and verify the recognition result with the VIN number format information in the MES system;

[0132] The fifth step involves correctly calling the algorithm model to identify and label each image, processing the identification results, and forwarding the image and detection results to the MES system for display. Otherwise, the transmitted data will be recorded as an error. To call the algorithm, the API interface file of the algorithm model needs to be modified to make calling the model more convenient and faster.

[0133] The paint defect detection system in this embodiment is deployed on an edge computer equipped with a CentOS 7.6 system. The system environment needs to be configured and related software needs to be installed, including the Basler camera driver, the installation and configuration of the PyTorch environment, and the installation of Nginx. After the software is installed, the trigger API interface needs to be debugged, that is, to realize the signal transmission and feedback signals from the sensor to the GTI. The main HTTP protocol is used for information transmission and feedback to complete the debugging of the trigger.

[0134] After the automatic triggering implementation in this embodiment is completed, it is necessary to realize the real-time display of the image. Similarly, the HTTP protocol is used to transmit information, and Nginx is used to transmit the image. The image address is fixed and the name is transmitted to realize the access to the image address, thereby realizing the successful display of the image and the successful transmission of detection information.

[0135] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes or modifications made to the structure, features and principles described in the claims of the present invention should be included in the scope of the claims of the present invention.

[0136] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A vehicle paint defect detection system, characterized in that, It includes hardware devices and edge computers, wherein the hardware devices include a camera module, a lighting module, a display module, and a triggering device; The camera module is used to capture defect images to detect minute defects in the paint surface of the entire vehicle. The lighting module is used to illuminate defects in the car paint to achieve high image acquisition quality; The display module is used to display images with correct defects; The triggering device is used to achieve automatic image acquisition; The edge computer is used as a transmission center for computing power and data, enabling the calling of hardware devices and the forwarding of data, thereby achieving control over the entire system; It also includes a programming module, which includes API interface files, OCR recognition algorithm files, and an improved YOLOv7 detection algorithm file; The improved YOLOv7 detection algorithm also incorporates a GAM attention mechanism. GAM redesigns the Channel Attention and Spatial Attention sub-modules of CBAM. In the Channel Attention sub-module, the input data undergoes transformation across different dimensions, is converted back to its original dimensions after passing through an MLP, and finally processed by a Sigmoid function for output. This fusion of information from different dimensions is achieved through dimensional changes. In the Spatial Attention sub-module, convolution is used. The input data is first convolved, then the number of data channels is compressed, and after passing through a 7×7 convolution kernel, the number of data channels is increased to maintain consistency. Finally, it is also processed by a Sigmoid function for output. Furthermore, the ELAN module is improved: based on the CSP structure, the number of convolutional layers in the ELAN module is further reduced, and it is named the ELAN-S module. The ELAN-S module can divide the data into two parts, each passing through a 1×1 convolutional kernel. One of the parts then passes through a 3×3 convolutional kernel, and then a concat operation is performed before inputting it into a 1×1 convolution to complete the processing of the entire data.

2. The vehicle paint defect detection system according to claim 1, characterized in that, The lighting module includes an LED diffuse surface light source and a light shield; The LED diffuse surface light source is used in conjunction with the coaxial light shooting method to reduce the occurrence of reflected light spots; The light-shielding plate is used to improve the consistency of the light source, so as to reduce the impact of changes in the external light source on the paint surface inspection system.

3. The vehicle paint defect detection system according to claim 1, characterized in that, The camera module includes at least eight high-resolution industrial cameras.

4. The vehicle paint defect detection system according to claim 3, characterized in that, The triggering device includes two high-precision medium-sized distance sensors; after the two medium-sized distance sensors receive the trigger signal simultaneously, they can transmit the rising edge signal to the edge computer, and automatically acquire the image of paint defects by configuring the camera SDK function.

5. The vehicle paint defect detection system according to claim 4, characterized in that, The acquired paint defect image categories include bubbles, dirt spots, fibers, scratches, and pinholes; the acquired image data is augmented to improve the ability to extract paint defect features, and the data augmentation methods include image rotation, image mirroring, and changes in image brightness and contrast.

6. The vehicle paint defect detection system according to claim 4, characterized in that, The API interface file enables the camera module to be invoked: a local area network was built between the camera module and the edge computer to enable separate invocation of eight cameras; data from the eight cameras is collected through a switch based on their IP addresses; simultaneous transmission of data from multiple cameras is achieved through 10 Gigabit fiber optic cable, which ensures complete transmission of images from all eight cameras without frame loss or data corruption; the simultaneous triggering of the eight cameras is accomplished using a Python thread pool, which calls the cameras separately based on their IP addresses. The API interface can be used to acquire and recognize VIN code images: by calling the corresponding camera, it acquires the image through a trigger signal, and then calls the OCR recognition algorithm file to recognize the image. The OCR recognition algorithm uses the open-source Baidu PaddleOCR algorithm model to improve the detection accuracy of Chinese characters. After the OCR algorithm recognizes the image, it matches and verifies the recognition result with the MES system. If it is correct, the image will be allowed to be displayed; otherwise, the transmitted data will be recorded as an error. The improved YOLOv7 detection algorithm specifically employs the YOLOv7-w6 network model, which has four detection heads. The network input image yields feature matrices of sizes 80×80×256, 40×40×512, 20×20×768, and 1010×1024. The YOLOv7-w6 network model adopts the PANet structure to further fuse images of different sizes, and also incorporates SPPCSPC, ELAN, and CBS modules, improving the overall performance and balance of the network model.

7. The vehicle paint defect detection system according to claim 6, characterized in that, The CBS is a basic convolutional module, which consists of convolutional layers, batch normalization layers, and SiLU activation functions. The CBS basic convolutional module includes three types: a convolutional kernel size of 1×1 with a stride of 1, a convolutional kernel size of 3×3 with a stride of 1, and a convolutional kernel size of 3×3 with a stride of 2. The convolutional kernel with a stride of 2 is used to replace the pooling layer to achieve downsampling of the image. The ELAN module is mainly composed of CBS modules. The ELAN module adopts a CSP structure for module construction. The input image is divided into two parts for convolution. One part is directly connected to the connection layer after convolution, and the other part is fed into the connection layer by two CBS modules as a group, and finally passes through another CBS module. The CSP structure can effectively reduce redundant gradient information and improve the learning ability of the network. The SPPCSPC module also adopts the CSP structure and combines the pyramid pooling layer and the CBS module. The pyramid pooling layer uses a parallel connection of pooling layers of sizes 5×5, 9×9 and 13×13 to improve the multi-scale fusion capability of the entire model.

8. The vehicle paint defect detection system according to claim 6, characterized in that, The improved YOLOv7 detection algorithm employs K-means++ clustering to reselect prior bounding boxes. K-means++ clustering clusters the prior bounding box data based on the size of the labeled anchor boxes. It first sets a fixed number of initial cluster centers, uses the Intersection over Union (IoU) metric to assign all samples to nearby cluster centers, and then updates and iterates based on the average width and height of the boxes in each cluster until the optimal box size is obtained. The calculation formula is shown below: In the formula, Let i be the size of the region containing the i-th labeled anchor box in the dataset; is the region size of the j-th cluster center anchor; n is the number of detected targets in the dataset; k is the number of cluster center anchors.

9. The working method of the vehicle paint defect detection system according to any one of claims 4-7, characterized in that, Includes the following steps: Step 1: When the car arrives at the sensor sensing area, using the single-point trigger mode under the dual-sensor strategy, when both sensors receive the trigger signal at the same time, the signal is sent back to the PLC for logic processing. Step 2: After receiving the sensor signal, the PLC processes the logic to determine the position status of the vehicle body, vehicle, and production line, sends out the RFID reading signal, reads complete vehicle information and obtains real-time production line position information, and binds the RFID to the production queue. Step 3: When the GTI subscription queue information changes, the edge computer is triggered to call the camera SDK file via the API interface to collect paint surface images at various vehicle body points. The camera module is invoked, and a local area network is built between the cameras and the edge computer to enable separate calls to the cameras, distinguishing them by IP address. Data is collected from multiple cameras via a switch, and then transmitted simultaneously via 10 Gigabit fiber optic cable. The 10 Gigabit fiber optic cable ensures complete transmission of images from all eight cameras without frame drops or data loss. Simultaneous camera triggering is accomplished using a Python thread pool, calling the cameras separately based on their IP addresses. Step 4: After image acquisition, perform OCR recognition algorithm on the image acquired by the camera that obtained the vehicle VIN according to the camera IP, and match and verify the recognition result with the VIN number format information in the MES system; Step 5: If the verification result is correct, the improved YOLOv7 detection algorithm model is called to identify and label the images one by one, the identification results are processed, and the images and detection results are forwarded to the MES system and the images are allowed to be displayed. Otherwise, the transmitted data will be recorded as errors.

Images