Vehicle body paint defect dynamic detection method, device, system and equipment

Abstract

The present application relates to a method, device, system, and equipment for dynamic detection of vehicle body paint defects, wherein the method includes: obtaining a first vehicle body image captured by a fixed bracket image acquisition component and a second vehicle body image captured by a robotic arm handheld image acquisition component; identifying a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image; jointly calibrating the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result; based on the camera joint calibration result, mapping the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result. During the entire process, paint defect detection using both fixed bracket and robotic arm handheld modes is compatible, and dynamic and accurate vehicle body paint defect detection can be achieved without the vehicle body remaining relatively still.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, system, computer equipment, storage medium, and computer program product for dynamic detection of vehicle body paint defects. Background Art

[0002] Painting is a crucial step in the automotive production process. The painted surface protects the vehicle from corrosion and enhances its appearance and visibility. Unavoidable defects such as pits and particles can severely impact the product's aesthetics, performance, and service life. Therefore, timely detection and repair of defects arising during the production process are crucial. Traditional production lines still primarily rely on manual visual inspection for defect detection and repair, resulting in low detection efficiency and a high rate of missed detections.

[0003] With the advancement of technology, some manufacturers have gradually begun to develop online automobile paint inspection equipment. These inspection devices usually require the car body to remain relatively still and the automobile production line to be paused for data collection. The inspection time is long, which is difficult to meet the increasingly accelerated production pace. Moreover, as the car model changes, the system configuration usually needs to be significantly modified and adapted.

[0004] Therefore, there is an urgent need for a dynamic and accurate vehicle body paint defect detection solution. Summary of the Invention

[0005] Based on this, it is necessary to provide a dynamic and accurate method, device, system, computer equipment, computer-readable storage medium and computer program product for dynamic detection of vehicle body paint defects in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for dynamic detection of vehicle body paint defects. The method comprises:

[0007] Acquire a first vehicle body image captured by the fixed bracket image acquisition component and a second vehicle body image captured by the robotic arm handheld image acquisition component;

[0008] identifying a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image;

[0009] Performing joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0010] Based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result.

[0011] In one embodiment, mapping the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system based on the camera joint calibration result to obtain the vehicle body paint defect detection result includes:

[0012] Based on the camera joint calibration result, mapping the first vehicle body paint surface defect result to a vehicle body three-dimensional coordinate system;

[0013] Performing hand-eye calibration on the camera in the handheld image acquisition component of the robotic arm;

[0014] Mapping the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system based on the hand-eye calibration result and the camera joint calibration result;

[0015] The defect positions are mapped to the three-dimensional coordinate system of the vehicle body to obtain the vehicle body paint defect detection results.

[0016] In one embodiment, the joint calibration of the camera in the fixed support image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain the camera joint calibration result includes:

[0017] Obtaining calibration data of a standard vehicle body captured by a camera in the fixed bracket image acquisition component and a camera in the robotic arm handheld image acquisition component, wherein calibration plates are distributed on the standard vehicle body, and each calibration plate corresponds to a unique ID;

[0018] Based on the calibration data, the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component are jointly calibrated to obtain a camera joint calibration result.

[0019] In one embodiment, the joint calibration of the camera in the fixed support image acquisition component and the camera in the robotic arm handheld image acquisition component based on the calibration data to obtain the camera joint calibration result includes:

[0020] Using a calibration feature point detection algorithm to extract corner feature coordinates and corresponding IDs in the calibration data;

[0021] Traversing the corner feature coordinates based on the same ID to generate feature point matching pairs of different cameras;

[0022] Generate an essential matrix based on a multi-view sparse reconstruction algorithm and according to the corner point feature coordinates and the feature point matching pairs;

[0023] Solve the rotation matrix R and translation vector of each camera according to the essential matrix and the PnP algorithm;

[0024] Restore all camera poses according to the rotation matrix R and translation vector of each camera;

[0025] Based on all restored camera poses, triangulation and bundle adjustment methods are used to obtain the optimized camera internal and external parameters.

[0026] In one embodiment, mapping the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system based on the camera joint calibration result to obtain the vehicle body paint defect detection result includes:

[0027] Establishing a reflected ray from the defect to the optical center of the camera in a reference coordinate system based on the camera joint calibration result, the first vehicle body paint defect result, and the second vehicle body paint defect result;

[0028] The intersection of the reflected light and the three-dimensional model data of the vehicle body is calculated by real-time simulation to obtain a real-time simulation calculation result;

[0029] According to the real-time simulation calculation result, the first vehicle body paint surface defect and the second vehicle body paint surface defect are mapped from pixel coordinates in the image to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint surface defect detection result.

[0030] In one embodiment, based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to a vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result;

[0031] Based on the camera joint calibration result, mapping the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system to obtain an initial vehicle body paint defect detection result;

[0032] Identifying repeated defect points in the initial vehicle body paint defect detection results where the distance between defects is less than a preset error tolerance;

[0033] The repeated defect points in the initial vehicle body paint surface defect detection result are eliminated to obtain the vehicle body paint surface defect detection result.

[0034] In a second aspect, the present application also provides a dynamic detection device for vehicle body paint defects. The device comprises:

[0035] An image acquisition module is used to acquire a first vehicle body image acquired by the fixed bracket image acquisition component and a second vehicle body image acquired by the robotic arm handheld image acquisition component;

[0036] an identification module, configured to identify a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image;

[0037] a calibration module, configured to jointly calibrate the camera in the fixed support image acquisition assembly and the camera in the robotic arm handheld image acquisition assembly to obtain a camera joint calibration result;

[0038] A detection module is used to map the first vehicle body paint defect result and the second vehicle body paint defect result to the vehicle body three-dimensional coordinate system based on the camera joint calibration result to obtain a vehicle body paint defect detection result.

[0039] In a third aspect, the present application also provides a vehicle body paint defect dynamic detection system, comprising a fixed bracket image acquisition component, a robotic arm handheld image acquisition component, and a processing and computing unit;

[0040] The fixed bracket image acquisition component acquires a first vehicle body image and sends the first vehicle body image to the processing and computing unit; the robotic arm handheld image acquisition component acquires a second vehicle body image and sends the second vehicle body image to the processing and computing unit. The processing and computing unit uses the above-mentioned method to perform dynamic detection of vehicle body paint defects.

[0041] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0042] Acquire a first vehicle body image captured by the fixed bracket image acquisition component and a second vehicle body image captured by the robotic arm handheld image acquisition component;

[0043] identifying a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image;

[0044] Performing joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0045] Based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result.

[0046] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0047] Acquire a first vehicle body image captured by the fixed bracket image acquisition component and a second vehicle body image captured by the robotic arm handheld image acquisition component;

[0048] identifying a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image;

[0049] Performing joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0050] Based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result.

[0051] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0052] Acquire a first vehicle body image captured by the fixed bracket image acquisition component and a second vehicle body image captured by the robotic arm handheld image acquisition component;

[0053] identifying a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image;

[0054] Performing joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0055] Based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result.

[0056] The above-mentioned dynamic vehicle body paint defect detection method, apparatus, computer equipment, storage medium, and computer program product obtain a first vehicle body image captured by a fixed-bracket image acquisition component and a second vehicle body image captured by a robotic arm handheld image acquisition component; identify a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image; jointly calibrate the camera in the fixed-bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result; and based on the camera joint calibration result, map the first vehicle body paint defect result and the second vehicle body paint defect result to the vehicle body's three-dimensional coordinate system to obtain a vehicle body paint defect detection result. Throughout this process, paint defect detection using both fixed-bracket and robotic arm handheld methods is compatible, and dynamic and accurate vehicle body paint defect detection can be achieved without the vehicle body remaining relatively stationary. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1This is a diagram of an application environment of a method for dynamic detection of vehicle body paint defects in one embodiment;

[0058] Figure 2 1 is a flow chart of a method for dynamic detection of vehicle body paint defects in one embodiment;

[0059] Figure 3 is a flow chart of a method for dynamic detection of vehicle body paint defects in another embodiment;

[0060] Figure 4 is a structural block diagram of a dynamic detection device for vehicle body paint defects in one embodiment;

[0061] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] The dynamic detection method for vehicle body paint defects provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The entire vehicle body paint defect dynamic detection system mainly includes a fixed bracket image acquisition component 102, a robotic arm handheld image acquisition component 104 and a controller 106. In addition, the entire system can also include PLC modules, encoders, light sources and other devices. When implementing the basic vehicle body paint defect dynamic detection, the fixed bracket image acquisition component 102 acquires a first vehicle body image, the robotic arm handheld image acquisition component 104 acquires a second vehicle body image, and the controller 106 obtains the first vehicle body image acquired by the fixed bracket image acquisition component 102 and the second vehicle body image acquired by the robotic arm handheld image acquisition component 104; identifies the first vehicle body paint defect result in the first vehicle body image and the second vehicle body paint defect result in the second vehicle body image; jointly calibrates the camera in the fixed bracket image acquisition component 102 and the camera in the robotic arm handheld image acquisition component 104 to obtain a camera joint calibration result; based on the camera joint calibration result, maps the first vehicle body paint defect result and the second vehicle body paint defect result to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result. Furthermore, the above-mentioned PLC module is mainly responsible for controlling the overall hardware and determining the timing of the camera taking pictures; the encoder is used to determine the position of the vehicle body; the photoelectric sensor is used to detect whether the vehicle has reached the detection area and start the defect detection program. The encoder is used to detect the forward distance of the vehicle body, which is used to control the camera's shooting rhythm and for the precise positioning of vehicle body defects. The displacement change of the vehicle body for each picture should not exceed 1 / 4 of the width of the short side of the binary stripe light source.

[0064] Furthermore, the cameras in the fixed-mount image acquisition component 102 and the robotic arm handheld image acquisition component 104 utilize layout simulation to ensure that the captured images cover the entire vehicle body. More specifically, the light source utilizes a combination of binary stripe LEDs and multiple white LEDs at intervals. During the inspection process, as the vehicle passes through a tunnel, the robotic arm handheld image acquisition component switches between test points, while the fixed-mount image acquisition component captures images of the vehicle body at the fixed test points. The light source creates a quasi-binary stripe of light on the vehicle body surface, creating a more pronounced difference in the defect area relative to the surrounding normal paint surface. This highlights the defect area and facilitates defect detection, effectively enhancing the characteristics of defects such as protrusions, depressions, particles, and shrinkage cavities. Simultaneously, the cameras capture images of the stripe of light reflected from the vehicle body as it moves relative to the inspection system. The multiple cameras are mounted so that the captured images cover the entire illuminated cross-section of the vehicle body, and the field of view between cameras must be continuous to prevent data gaps caused by data splicing, which can affect defect detection. Furthermore, appropriate camera lenses are selected to cover a wider inspection area within the depth of field and maximize the camera's spatial resolution to detect smaller defects. Specifically, the camera layout can be simulated based on the vehicle body CAD model to check the camera's field of view on the vehicle body model (which can be displayed on the vehicle body surface in the simulation environment). If there is a missing area, consider adjusting the camera position or angle. If the missing area is too large, consider increasing the camera configuration until a relatively good camera layout is achieved. Based on the simulation layout results, the actual system is built, and some inherent features of the vehicle body and hardware structure are combined for fine-tuning. There needs to be a certain overlap between adjacent cameras.

[0065] In one embodiment, Figure 2 As shown, a dynamic detection method for vehicle body paint defects is provided. Figure 1 The controller in the example is used to illustrate the following steps:

[0066] S200: Acquire a first vehicle body image captured by the fixed bracket image acquisition component and a second vehicle body image captured by the robotic arm handheld image acquisition component.

[0067] A fixed bracket image acquisition component refers to an image acquisition component fixed at a preset specific position, such as a camera fixed at a preset specific position. In actual applications, a fixed bracket method can be used to set up an image acquisition component to capture vehicle body images for the left and right sides of the vehicle body whose height positions are relatively fixed. A robotic arm handheld image acquisition component refers to an image acquisition component that is set on a robotic arm and can adjust its position as the robotic arm moves, such as a camera set at the end of the robotic arm. In actual applications, a relatively flexible robotic arm handheld method is used for areas whose height positions change greatly with the vehicle model, such as the roof, hood, and tail, and the detection points are adjusted in real time as the vehicle body moves.

[0068] S400: Identify a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image.

[0069] Image-based paint recognition technology can be used to identify a first vehicle body paint defect result in a first vehicle body image and a second vehicle body paint defect result in a second vehicle body image. Specifically, a trained vehicle paint defect recognition model can be used to identify paint defects in images. Furthermore, the location of defects on the vehicle body can be identified and located, the defect type determined, and the defect size calculated. Subsequently, the specific location of the defect on the vehicle body can be determined by combining the vehicle body CAD 3D model, the vehicle body reference position at the time of the photo, and the camera calibration parameters. This can then be mapped to the vehicle body 3D model, and the defect location information on each part of the vehicle body can be further displayed on the software UI.

[0070] S600: Jointly calibrate the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result.

[0071] Here, a joint calibration method is used for the cameras in the fixed-bracket image acquisition component and the cameras in the robotic arm handheld image acquisition component to obtain the intrinsic parameters of all cameras and the camera extrinsic parameters (specifically, the rotation matrix R and the translation matrix T) in a unified reference coordinate system. Furthermore, based on a pre-set calibration plate on the vehicle body, calibration data can be collected and then feature point analysis can be performed on the calibration data to complete the calibration of all cameras in a single coordinate system or a unified world coordinate system.

[0072] S800: Based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result.

[0073] Based on the results of the S600 camera joint calibration, the first body paint defect result and the second body paint defect result are mapped to the body three-dimensional coordinate system, thus obtaining a complete body paint defect detection result.

[0074] The above-mentioned dynamic vehicle body paint defect detection method obtains a first vehicle body image captured by a fixed-bracket image acquisition component and a second vehicle body image captured by a robotic arm handheld image acquisition component; identifies a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image; jointly calibrates the camera in the fixed-bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result; and based on the camera joint calibration result, maps the first vehicle body paint defect result and the second vehicle body paint defect result to the vehicle body's three-dimensional coordinate system to obtain a vehicle body paint defect detection result. Throughout this process, paint defect detection using both fixed-bracket and robotic arm handheld methods is compatible, and dynamic and accurate vehicle body paint defect detection can be achieved without the vehicle body remaining relatively stationary.

[0075] like Figure 3 As shown, in one embodiment, S800 includes:

[0076] S820: Based on the camera joint calibration result, map the first vehicle body paint surface defect result to the vehicle body three-dimensional coordinate system.

[0077] Camera joint calibration refers to unifying the imaging systems of multiple cameras (such as cameras mounted on a robotic arm and cameras in fixed positions) into a common reference coordinate system. This is usually achieved by photographing a set of calibration plates of known shape and position (such as a checkerboard calibration plate), and using the relationship between the positions of the feature points on these calibration plates in the image and their positions in the real world to calculate the intrinsic and extrinsic parameters of each camera (including focal length, optical center, distortion coefficient, and the position and posture of the camera in the world coordinate system). Using the camera joint calibration results, the first body paint defect results (such as scratches, pits, etc.) captured and identified by the fixed-position camera can be converted from the two-dimensional image coordinate system to the three-dimensional coordinate system of the body. This step depends on the intrinsic and extrinsic parameters of the camera and the position information of the defect in the image.

[0078] S840: Perform hand-eye calibration on the camera in the handheld image acquisition component of the robotic arm.

[0079] Hand-eye calibration is the process of determining the relative position and posture between the robotic arm's end effector (such as a camera) and the robotic arm itself. Because the robotic arm may move during the inspection process, hand-eye calibration is necessary to ensure that the camera accurately reflects the position of objects within its field of view, regardless of the robotic arm's position. Hand-eye calibration also typically involves photographing a calibration plate, but in this case, the plate needs to be moved to different positions by the robotic arm for photographing.

[0080] S860: Based on the hand-eye calibration result and the camera joint calibration result, the second vehicle body paint defect result is mapped to the vehicle body three-dimensional coordinate system.

[0081] Based on the results of hand-eye calibration and joint camera calibration, the paint defect on the second vehicle body, captured and identified by the robot arm's handheld camera, can be mapped into the vehicle body's 3D coordinate system. This step takes into account not only the camera's imaging characteristics but also the robot arm's motion, enabling precise localization of defects captured by the moving camera.

[0082] Specifically, for the method of using a robotic arm holding a vision unit, hand-eye calibration is also required. Specifically, it is necessary to collect the image of the calibration plate taken by the camera and save the position of the robotic arm in the corresponding teach pendant. The transformation relationship between the camera coordinate system and the robotic arm end coordinate system and the base coordinate system is obtained through calculation, and finally the image pixel coordinates at different robotic arm points are converted into the three-dimensional coordinates of the reference coordinate system.

[0083] S880: Collect and map defect positions in the three-dimensional coordinate system of the vehicle body to obtain vehicle body paint defect detection results.

[0084] Finally, all defect locations mapped to the vehicle body's 3D coordinate system are aggregated to form a complete paint defect detection result. This result not only includes information on the type and size of the defect, but also their precise location on the vehicle body, providing an important reference for subsequent repair work.

[0085] like Figure 3 As shown, in one embodiment, S600 includes:

[0086] S620: Obtain calibration data of a standard vehicle body captured by a camera in a fixed bracket image acquisition component and a camera in a robotic arm handheld image acquisition component. Calibration plates are distributed on the standard vehicle body, and each calibration plate corresponds to a unique ID.

[0087] Standard vehicle body: A standard vehicle body with known geometric and surface properties is selected as the calibration object. This vehicle body should be representative of the actual production vehicle body, and its surface needs to be sufficiently flat and uniform to accurately place the calibration plate. Calibration plate: Multiple calibration plates are distributed on the standard vehicle body, each with a unique ID for easy identification and differentiation. Calibration plates are typically flat plates with known geometric shapes and patterns (such as a checkerboard, circular dot array, etc.), which are easy to detect and identify in images.

[0088] Use the camera in the fixed-bracket image acquisition assembly and the camera in the handheld robotic arm image acquisition assembly to capture images of the calibration plate on the standard vehicle body from different angles and positions. Ensure that each camera captures a sufficient number of calibration plate images and that these images contain sufficient feature points for subsequent processing. Record the calibration plate images captured by each camera, their corresponding ID information, and the camera parameters used during the capture (such as focal length and exposure time).

[0089] S640: Jointly calibrate the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component based on the calibration data to obtain a camera joint calibration result.

[0090] Joint calibration specifically involves the following stages: 1) Feature Point Extraction: First, feature points are extracted from the calibration plate image captured by each camera. These feature points are typically easy-to-detect and locate points on the calibration plate, such as corners and circle centers. 2) Matching and Correspondence: Next, based on the calibration plate ID information, feature points in images from different cameras are matched and corresponded. This typically involves image processing and computer vision techniques, such as feature descriptor matching and the RANSAC algorithm, to eliminate false matches and noise. 3) Optimization: Using matched feature point pairs, an optimization algorithm (such as the least squares method or the iterative closest point algorithm) is used to determine the camera's intrinsic and extrinsic parameters. These parameters include the camera's focal length, optical center position, distortion coefficient, and its position and pose in the world coordinate system. 4) Joint Calibration Result: Finally, the parameters of the cameras in the fixed-mount image acquisition component and the robotic arm's handheld image acquisition component are unified into a common reference coordinate system, resulting in the joint camera calibration result. This result describes how each camera maps its captured image to the real-world 3D coordinate system.

[0091] Generally speaking, in practical applications, the entire calibration process involves the following: small calibration plates (such as AprilTag, ArUco, etc.) are evenly distributed on the body of a standard prototype vehicle. Each small calibration plate has its own ID, ensuring that each calibration plate detected by the camera has a unique ID corresponding to it. After the calibration plates are attached, the vehicle body is sent into the inspection system. According to the standard process, the vehicle body is moved and calibration data is collected multiple times at equal intervals. After image acquisition, feature points in the calibration data are extracted. The IDs of the captured calibration plates can be used to link adjacent cameras, thereby achieving joint calibration of all cameras. This allows all cameras to be calibrated to one of the camera coordinate systems or a unified world coordinate system, thus completing the calibration of all cameras. After the initial camera calibration, the camera intrinsic parameters M, distortion coefficient D, and camera extrinsic parameters (rotation matrix R, translation matrix T) in the reference coordinate system can be obtained.

[0092] In one embodiment, a camera in a fixed support image acquisition assembly and a camera in a robotic arm handheld image acquisition assembly are jointly calibrated based on the calibration data, and the camera joint calibration result obtained includes:

[0093] Step 1: Use the calibration feature point detection algorithm to extract the corner feature coordinates and corresponding IDs in the calibration data.

[0094] Use a calibration feature point detection algorithm (such as AprilTag's detection algorithm) to extract corner feature coordinates and corresponding IDs from the calibration data. These corner points are usually obvious and easy to identify points on the calibration board, such as the corners of a checkerboard. Each corner feature coordinate is associated with its unique ID on the calibration board, which facilitates subsequent feature point matching.

[0095] Step 2: Traverse the corner feature coordinates based on the same ID to generate feature point matching pairs of different cameras.

[0096] Based on the same ID, the coordinates of the corner points of all cameras are traversed to generate matching pairs of feature points between different cameras. This step is critical to ensuring that images from different cameras can be correctly matched. The matching process may use various matching strategies, such as nearest neighbor matching and bidirectional matching, to reduce the occurrence of mismatches.

[0097] Step 3: Generate the essential matrix based on the multi-view sparse reconstruction algorithm and according to the corner feature coordinates and feature point matching pairs.

[0098] Using a multi-view sparse reconstruction algorithm, the Essential Matrix is ​​generated based on the matched feature point coordinate pairs. The Essential Matrix describes the geometric relationship between the two camera views and includes information about the camera's rotation and translation (though not directly). Solving the Essential Matrix typically involves mathematical optimization methods, such as the Eight-Point Algorithm.

[0099] Step 4: Solve the rotation matrix R and translation vector of each camera based on the essential matrix and PnP algorithm.

[0100] According to the essential matrix and PnP (Perspective-n-Point) algorithm, the rotation matrix R and translation vector t of each camera are solved. The PnP algorithm is a method that uses n known 3D points and their projections on the image to estimate the camera pose.

[0101] Step 5: Restore all camera poses based on the rotation matrix R and translation vector of each camera.

[0102] Recover the poses of all cameras based on the rotation matrix R and translation vector t of each camera. The pose of a camera refers to the position and attitude of the camera in the world coordinate system, which determines how the camera observes the world.

[0103] Step 6: Based on all restored camera poses, triangulation and bundle adjustment are used to obtain the optimized camera internal and external parameters.

[0104] Based on the recovered camera pose, triangulation and bundle adjustment are used to optimize the camera's intrinsic and extrinsic parameters. Triangulation is the process of estimating the position of 3D points using matched feature point coordinate pairs, while bundle adjustment is a global optimization algorithm that optimizes the camera's intrinsic and extrinsic parameters and the positions of 3D points by minimizing the reprojection error.

[0105] In addition, in practical applications, the camera pose estimation and camera parameter calibration in the multi-view stereo reconstruction system can also be completed through open source algorithm libraries such as colmap and OpenMVG.

[0106] In one embodiment, based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint defect detection result, including:

[0107] Step 1: Based on the camera joint calibration results, the first body paint defect results, and the second body paint defect results, establish the reflected light from the defect to the camera optical center in the reference coordinate system.

[0108] First, based on the results of joint camera calibration, we know the intrinsic parameters (focal length, optical center, etc.) and extrinsic parameters (the camera's position and orientation in the world coordinate system) of each camera. For each defect in the first and second body paint defect results, we need to determine its pixel coordinates in the image. Using the camera's intrinsic parameters and these pixel coordinates, we can calculate a reflected light ray (or ray) that starts from the camera's optical center, passes through the defect point in the image, and points to three-dimensional space. This ray represents the possible location of the defect in three-dimensional space as seen from the camera's perspective.

[0109] Step 2: Calculate the intersection of the reflected light and the three-dimensional model data of the vehicle body through real-time simulation to obtain the real-time simulation calculation results.

[0110] Next, we need a 3D model of the vehicle body, typically a CAD model or scanned point cloud data. Using real-time simulation techniques (such as ray tracing algorithms), we can simulate the interaction of these reflected rays with the 3D vehicle body model. By calculating, we find the intersection of each reflected ray with the 3D vehicle body model. These intersections represent the possible locations of defects in the 3D vehicle body coordinate system.

[0111] Step 3: Based on the real-time simulation calculation results, the first vehicle body paint defect and the second vehicle body paint defect are mapped from the pixel coordinates in the image to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint defect detection results.

[0112] Based on the results of real-time simulation calculations, we can map the coordinates of each defective pixel in the image to one or more points in the three-dimensional coordinate system of the vehicle body (depending on the number of intersections of the reflected light and the vehicle body). If there are multiple intersections (for example, when the defect is located on the curved surface of the vehicle body), further analysis may be required to determine the most likely defect location. This can be achieved by considering the shape, size, color and other characteristics of the defect, combined with the geometry of the vehicle body. Finally, all defect locations mapped to the three-dimensional coordinate system of the vehicle body are aggregated to form a complete vehicle body paint defect detection result. This result not only contains the location information of the defect in three-dimensional space, but may also include detailed information such as the type, size, and severity of the defect.

[0113] In one embodiment, based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint defect detection result;

[0114] Step 1: Based on the camera joint calibration results, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body 3D coordinate system to obtain the initial vehicle body paint defect detection result;

[0115] Step 2: Identify repeated defect points in the initial vehicle body paint defect detection results where the distance between defects is less than a preset error tolerance;

[0116] Step 3: Remove duplicate defect points from the initial vehicle body paint defect detection results to obtain the vehicle body paint defect detection results.

[0117] In this embodiment, duplicate defects are removed from the initial vehicle body paint defect detection results. Specifically, the initial vehicle body paint defect detection results may contain data from the same defect captured by multiple cameras or by the same camera at different times. This can be effectively removed after mapping to the vehicle body's three-dimensional coordinate system. Considering the presence of errors in the coordinate mapping process, an error tolerance value Δd is set. If the Euclidean distance between defects is less than the error tolerance value, the defect is considered a duplicate point. This tolerance value is generally set to a fixed threshold, such as 10mm, but it may need to be fine-tuned based on the coordinate mapping errors of the final system.

[0118] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0119] Based on the same inventive concept, embodiments of the present application also provide a vehicle body paint defect dynamic detection device for implementing the aforementioned vehicle body paint defect dynamic detection method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more of the following embodiments of the vehicle body paint defect dynamic detection device can be found in the aforementioned limitations of the vehicle body paint defect dynamic detection method and will not be further elaborated here.

[0120] In one embodiment, Figure 4 As shown, a dynamic detection device for vehicle body paint defects is provided, comprising:

[0121] An image acquisition module 200 is configured to acquire a first vehicle body image acquired by the fixed bracket image acquisition component and a second vehicle body image acquired by the robotic arm handheld image acquisition component;

[0122] An identification module 400 is configured to identify a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image;

[0123] The calibration module 600 is used to jointly calibrate the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0124] The detection module 800 is used to map the first vehicle body paint defect result and the second vehicle body paint defect result to the vehicle body three-dimensional coordinate system based on the camera joint calibration result to obtain the vehicle body paint defect detection result.

[0125] In one embodiment, the detection module 800 is also used to map the first vehicle body paint defect result to the vehicle body three-dimensional coordinate system based on the camera joint calibration result; perform hand-eye calibration on the camera in the robotic arm handheld image acquisition component; map the second vehicle body paint defect result to the vehicle body three-dimensional coordinate system based on the hand-eye calibration result and the camera joint calibration result; and aggregate the defect positions mapped in the vehicle body three-dimensional coordinate system to obtain the vehicle body paint defect detection result.

[0126] In one embodiment, the calibration module 600 is also used to obtain calibration data of a standard vehicle body captured by the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component. Calibration plates are distributed on the standard vehicle body, and each calibration plate corresponds to a unique ID; based on the calibration data, the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component are jointly calibrated to obtain a camera joint calibration result.

[0127] In one embodiment, the calibration module 600 is further used to extract corner feature coordinates and corresponding IDs in the calibration data using a calibration feature point detection algorithm; traverse the corner feature coordinates based on the same ID to generate feature point matching pairs of different cameras; generate an essential matrix based on a multi-view sparse reconstruction algorithm and according to the corner feature coordinates and feature point matching pairs; solve the rotation matrix R and translation vector of each camera based on the essential matrix and the PnP algorithm; restore all camera poses based on the rotation matrix R and translation vector of each camera; based on all restored camera poses, use triangulation and bundle adjustment to obtain optimized camera internal and external parameter results.

[0128] In one embodiment, the detection module 800 is also used to establish the reflected light from the defect to the camera optical center in the reference coordinate system based on the camera joint calibration result, the first body paint defect result and the second body paint defect result; obtain the real-time simulation calculation result by calculating the intersection of the reflected light and the body three-dimensional model data through real-time simulation; and map the first body paint defect and the second body paint defect from the pixel coordinates in the image to the body three-dimensional coordinate system according to the real-time simulation calculation result to obtain the body paint defect detection result.

[0129] In one embodiment, the detection module 800 is also used to map the first body paint defect result and the second body paint defect result to the body three-dimensional coordinate system based on the camera joint calibration result to obtain an initial body paint defect detection result; identify repeated defect points in the initial body paint defect detection result where the distance between defects is less than a preset error tolerance value; and eliminate repeated defect points in the initial body paint defect detection result to obtain a body paint defect detection result.

[0130] Each module in the aforementioned dynamic vehicle body paint defect detection device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor within a computer device in hardware form, or stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.

[0131] In one embodiment, the present application further provides a vehicle body paint defect dynamic detection system, comprising a fixed bracket image acquisition component, a robotic arm handheld image acquisition component, and a processing and computing unit;

[0132] The fixed bracket image acquisition component acquires a first vehicle body image and sends the first vehicle body image to the processing and computing unit; the robotic arm handheld image acquisition component acquires a second vehicle body image and sends the second vehicle body image to the processing and computing unit, and the processing and computing unit uses the above-mentioned vehicle body paint defect dynamic detection method to perform dynamic detection of vehicle body paint defects.

[0133] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a dynamic detection method for vehicle body paint defects is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0134] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0135] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned dynamic detection method for vehicle body paint defects when executing the computer program.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned dynamic detection method for vehicle body paint defects is implemented.

[0137] In one embodiment, a computer program product is provided, comprising a computer program, which implements the above-mentioned method for dynamic detection of vehicle body paint defects when executed by a processor.

[0138] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0139] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A dynamic detection method for vehicle body paint defects, characterized in that: The method comprises: Acquire a first vehicle body image captured by the fixed bracket image acquisition component and a second vehicle body image captured by the robotic arm handheld image acquisition component; identifying a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image; Performing joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result; Based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result, including: Based on the camera joint calibration result, mapping the first vehicle body paint surface defect result to a vehicle body three-dimensional coordinate system; Performing hand-eye calibration on the camera in the handheld image acquisition component of the robotic arm; Mapping the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system based on the hand-eye calibration result and the camera joint calibration result; The defect positions are mapped to the three-dimensional coordinate system of the vehicle body to obtain the vehicle body paint defect detection results.

2. The method according to claim 1, characterized in that The joint calibration of the camera in the fixed support image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result includes: Obtaining calibration data of a standard vehicle body captured by a camera in the fixed bracket image acquisition component and a camera in the robotic arm handheld image acquisition component, wherein calibration plates are distributed on the standard vehicle body, and each calibration plate corresponds to a unique ID; Based on the calibration data, the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component are jointly calibrated to obtain a camera joint calibration result.

3. The method according to claim 2, characterized in that The step of jointly calibrating the camera in the fixed support image acquisition component and the camera in the robotic arm handheld image acquisition component based on the calibration data to obtain a camera joint calibration result includes: Using a calibration feature point detection algorithm to extract corner feature coordinates and corresponding IDs in the calibration data; Traversing the corner feature coordinates based on the same ID to generate feature point matching pairs of different cameras; Generate an essential matrix based on a multi-view sparse reconstruction algorithm and according to the corner point feature coordinates and the feature point matching pairs; Solve the rotation matrix R and translation vector of each camera according to the essential matrix and the PnP algorithm; Restore all camera poses according to the rotation matrix R and translation vector of each camera; Based on all restored camera poses, triangulation and bundle adjustment methods are used to obtain the optimized camera internal and external parameters.

4. The method according to claim 1, wherein The step of mapping the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system based on the camera joint calibration result to obtain a vehicle body paint defect detection result includes: Establishing a reflected ray from the defect to the optical center of the camera in a reference coordinate system based on the camera joint calibration result, the first vehicle body paint defect result, and the second vehicle body paint defect result; The intersection of the reflected light and the three-dimensional model data of the vehicle body is calculated by real-time simulation to obtain a real-time simulation calculation result; According to the real-time simulation calculation result, the first vehicle body paint surface defect and the second vehicle body paint surface defect are mapped from pixel coordinates in the image to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint surface defect detection result.

5. The method according to any one of claims 1 to 4, characterized in that include: Based on the camera joint calibration result, mapping the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system to obtain an initial vehicle body paint defect detection result; Identifying repeated defect points in the initial vehicle body paint defect detection results where the distance between defects is less than a preset error tolerance; The repeated defect points in the initial vehicle body paint surface defect detection result are eliminated to obtain the vehicle body paint surface defect detection result.

6. A dynamic detection device for vehicle body paint defects, characterized in that: The device comprises: An image acquisition module is used to acquire a first vehicle body image acquired by the fixed bracket image acquisition component and a second vehicle body image acquired by the robotic arm handheld image acquisition component; an identification module, configured to identify a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image; a calibration module, configured to jointly calibrate the camera in the fixed support image acquisition assembly and the camera in the robotic arm handheld image acquisition assembly to obtain a camera joint calibration result; a detection module, configured to map the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system based on the camera joint calibration result, to obtain a vehicle body paint defect detection result; The detection module is also used to map the first vehicle body paint defect result to the vehicle body three-dimensional coordinate system based on the camera joint calibration result; perform hand-eye calibration on the camera in the robotic arm handheld image acquisition component; map the second vehicle body paint defect result to the vehicle body three-dimensional coordinate system based on the hand-eye calibration result and the camera joint calibration result; and aggregate the defect positions mapped in the vehicle body three-dimensional coordinate system to obtain the vehicle body paint defect detection result.

7. A dynamic detection system for vehicle body paint defects, characterized in that: It includes a fixed bracket image acquisition component, a robotic arm handheld image acquisition component and a processing and calculation unit; The fixed bracket image acquisition component acquires a first vehicle body image and sends the first vehicle body image to the processing and computing unit; the robotic arm handheld image acquisition component acquires a second vehicle body image and sends the second vehicle body image to the processing and computing unit, and the processing and computing unit uses the method described in any one of claims 1 to 5 to perform dynamic detection of vehicle body paint defects.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images