Vehicle body paint surface defect dynamic detection method, device, system and equipment

Abstract

The invention relates to a vehicle body paint surface defect dynamic detection method, device, system and equipment, and the method comprises the steps: obtaining a first vehicle body image collected by a fixed support image collection assembly, and a second vehicle body image collected by a mechanical arm handheld image collection assembly; identifying a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image; joint calibration is conducted on a camera in the fixed support image acquisition assembly and a camera in the mechanical arm handheld mode image acquisition assembly, and a camera joint calibration result is obtained; and based on a camera joint calibration result, mapping the first vehicle body paint surface defect result and the second vehicle body paint surface defect result into a vehicle body three-dimensional coordinate system to obtain a vehicle body paint surface defect detection result. In the whole process, paint surface defect detection in a fixed support and mechanical arm handheld mode is compatible, and dynamic and accurate vehicle body paint surface defect detection can be achieved without keeping the vehicle body relatively static.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, system, computer equipment, storage medium and computer program product for dynamic detection of vehicle body paint defects. Background Art

[0002] Painting is an important part of the automobile production process. The painted surface of the car body protects the car body from corrosion and improves its appearance and recognition. In the process of automobile painting, some defects such as pits and particles are inevitable. The existence of defects seriously affects the appearance, performance and service life of the product. Therefore, it is necessary to detect the defects generated in the production process in time and repair them. Traditional production lines still mainly use manual visual methods to detect and repair defects, which has the problems of low detection efficiency and high missed detection rate.

[0003] With the advancement of technology, some manufacturers have gradually begun to develop online inspection equipment for automotive paint. These inspection devices usually require the car body to remain relatively still and the automobile production line to be suspended for data collection. The inspection time is long and it is difficult to meet the increasingly accelerated production rhythm. Moreover, as the car model changes, the system configuration usually needs to be significantly changed and adapted.

[0004] Therefore, there is an urgent need for a dynamic and accurate vehicle body paint defect detection solution. Summary of the invention

[0005] Based on this, it is necessary to provide a dynamic and accurate method, device, system, computer equipment, computer-readable storage medium and computer program product for dynamic detection of vehicle body paint defects in response to the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a method for dynamic detection of vehicle body paint defects. The method comprises:

[0007] Acquire a first vehicle body image acquired by the fixed bracket image acquisition component and a second vehicle body image acquired by the robotic arm handheld image acquisition component;

[0008] Identifying a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image;

[0009] Performing joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0010] Based on the camera joint calibration result, the first vehicle body paint surface defect result and the second vehicle body paint surface defect result are mapped to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint surface defect detection result.

[0011] In one embodiment, based on the camera joint calibration result, mapping the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result includes:

[0012] Based on the camera joint calibration result, mapping the first vehicle body paint surface defect result to a vehicle body three-dimensional coordinate system;

[0013] Performing hand-eye calibration on the camera in the image acquisition component of the robotic arm handheld mode;

[0014] Based on the hand-eye calibration result and the camera joint calibration result, mapping the second vehicle body paint surface defect result to the vehicle body three-dimensional coordinate system;

[0015] The defect positions mapped in the three-dimensional coordinate system of the vehicle body are collected to obtain the vehicle body paint defect detection result.

[0016] In one embodiment, the joint calibration of the camera in the fixed support image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain the camera joint calibration result includes:

[0017] Obtaining calibration data of a standard vehicle body collected by a camera in the fixed bracket image acquisition component and a camera in the robotic arm handheld image acquisition component, wherein calibration plates are distributed on the standard vehicle body, and each calibration plate corresponds to a unique ID;

[0018] Based on the calibration data, the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component are jointly calibrated to obtain a camera joint calibration result.

[0019] In one embodiment, the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component are jointly calibrated based on the calibration data to obtain the camera joint calibration result, which includes:

[0020] Using a calibration feature point detection algorithm to extract corner point feature coordinates and corresponding IDs in the calibration data;

[0021] Traversing the corner point feature coordinates based on the same ID to generate feature point matching pairs of different cameras;

[0022] Generate an essential matrix based on a multi-view sparse reconstruction algorithm and according to the corner point feature coordinates and the feature point matching pairs;

[0023] Solve the rotation matrix R and translation vector of each camera according to the essential matrix and the PnP algorithm;

[0024] Restore all camera poses according to the rotation matrix R and translation vector of each camera;

[0025] Based on the restored camera poses, triangulation and bundle adjustment are used to obtain the optimized camera internal and external parameters.

[0026] In one embodiment, based on the camera joint calibration result, mapping the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result includes:

[0027] Based on the camera joint calibration result, the first vehicle body paint surface defect result and the second vehicle body paint surface defect result, establishing a reflected light from the defect to the optical center of the camera in the reference coordinate system;

[0028] By real-time simulation calculation of the intersection of the reflected light and the three-dimensional model data of the vehicle body, a real-time simulation calculation result is obtained;

[0029] According to the real-time simulation calculation result, the first vehicle body paint surface defect and the second vehicle body paint surface defect are mapped from pixel coordinates in the image to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint surface defect detection result.

[0030] In one of the embodiments, based on the camera joint calibration result, the first vehicle body paint surface defect result and the second vehicle body paint surface defect result are mapped to a vehicle body three-dimensional coordinate system to obtain a vehicle body paint surface defect detection result;

[0031] Based on the camera joint calibration result, mapping the first vehicle body paint surface defect result and the second vehicle body paint surface defect result to a vehicle body three-dimensional coordinate system to obtain an initial vehicle body paint surface defect detection result;

[0032] Identifying repeated defect points in the initial vehicle body paint defect detection result where the distance between defects is less than a preset error tolerance value;

[0033] The repeated defect points in the initial vehicle body paint surface defect detection result are eliminated to obtain the vehicle body paint surface defect detection result.

[0034] In a second aspect, the present application also provides a vehicle body paint defect dynamic detection device. The device comprises:

[0035] An image acquisition module, used to acquire a first vehicle body image acquired by the fixed bracket image acquisition component and a second vehicle body image acquired by the mechanical arm handheld image acquisition component;

[0036] an identification module, configured to identify a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image;

[0037] A calibration module, used for jointly calibrating the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0038] A detection module is used to map the first vehicle body paint defect result and the second vehicle body paint defect result into a three-dimensional coordinate system of the vehicle body based on the camera joint calibration result to obtain a vehicle body paint defect detection result.

[0039] In a third aspect, the present application also provides a vehicle body paint defect dynamic detection system, including a fixed bracket image acquisition component, a robotic arm handheld image acquisition component, and a processing and computing unit;

[0040] The fixed bracket image acquisition component acquires a first vehicle body image and sends the first vehicle body image to the processing and computing unit; the robotic arm handheld image acquisition component acquires a second vehicle body image and sends the second vehicle body image to the processing and computing unit, and the processing and computing unit uses the above-mentioned method to perform dynamic detection of vehicle body paint defects.

[0041] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0042] Acquire a first vehicle body image acquired by the fixed bracket image acquisition component and a second vehicle body image acquired by the robotic arm handheld image acquisition component;

[0043] Identifying a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image;

[0044] Performing joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0045] Based on the camera joint calibration result, the first vehicle body paint surface defect result and the second vehicle body paint surface defect result are mapped to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint surface defect detection result.

[0046] In a fifth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0047] Acquire a first vehicle body image acquired by the fixed bracket image acquisition component and a second vehicle body image acquired by the robotic arm handheld image acquisition component;

[0048] Identifying a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image;

[0049] Performing joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0050] Based on the camera joint calibration result, the first vehicle body paint surface defect result and the second vehicle body paint surface defect result are mapped to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint surface defect detection result.

[0051] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0052] Acquire a first vehicle body image acquired by the fixed bracket image acquisition component and a second vehicle body image acquired by the robotic arm handheld image acquisition component;

[0053] Identifying a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image;

[0054] Performing joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0055] Based on the camera joint calibration result, the first vehicle body paint surface defect result and the second vehicle body paint surface defect result are mapped to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint surface defect detection result.

[0056] The above-mentioned vehicle body paint defect dynamic detection method, device, computer equipment, storage medium and computer program product obtain the first vehicle body image captured by the fixed bracket image acquisition component and the second vehicle body image captured by the robotic arm handheld image acquisition component; identify the first vehicle body paint defect result in the first vehicle body image and the second vehicle body paint defect result in the second vehicle body image; jointly calibrate the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain the camera joint calibration result; based on the camera joint calibration result, map the first vehicle body paint defect result and the second vehicle body paint defect result to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint defect detection result. In the whole process, the paint defect detection of the fixed bracket and the robotic arm handheld mode is compatible, and dynamic and accurate vehicle body paint defect detection can be achieved without the vehicle body remaining relatively still. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1A diagram showing an application environment of a method for dynamic detection of vehicle body paint defects in an embodiment;

[0058] Figure 2 A schematic diagram of a process of a dynamic detection method for vehicle body paint defects in one embodiment;

[0059] Figure 3 A schematic diagram of a flow chart of a method for dynamic detection of vehicle body paint defects in another embodiment;

[0060] Figure 4 is a structural block diagram of a dynamic detection device for vehicle body paint defects in one embodiment;

[0061] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] The vehicle body paint surface defect dynamic detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The entire vehicle body paint defect dynamic detection system mainly includes a fixed bracket image acquisition component 102, a mechanical arm handheld image acquisition component 104 and a controller 106. In addition, the entire system may also include PLC modules, encoders, light sources and other devices. When implementing the basic vehicle body paint defect dynamic detection, the fixed bracket image acquisition component 102 acquires a first vehicle body image, the mechanical arm handheld image acquisition component 104 acquires a second vehicle body image, and the controller 106 obtains the first vehicle body image acquired by the fixed bracket image acquisition component 102 and the second vehicle body image acquired by the mechanical arm handheld image acquisition component 104; identifies the first vehicle body paint defect result in the first vehicle body image and the second vehicle body paint defect result in the second vehicle body image; performs joint calibration on the camera in the fixed bracket image acquisition component 102 and the camera in the mechanical arm handheld image acquisition component 104 to obtain the camera joint calibration result; based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint defect detection result. Furthermore, the above-mentioned PLC module is mainly responsible for the control of the overall hardware and determines the timing of the camera taking pictures; the encoder is used to determine the position of the vehicle body; the photoelectric sensor is used to detect whether the vehicle has reached the detection area and start the defect detection program. The encoder is used to detect the forward distance of the vehicle body, which is used to control the camera's shooting rhythm and for the precise positioning of vehicle body defects. The displacement change of the vehicle body for each photo should not exceed 1 / 4 of the short side width of the binary stripe light source.

[0064] In addition, the cameras in the above-mentioned fixed bracket image acquisition component 102 and the robotic arm handheld image acquisition component 104 are arranged in a simulation manner so that the images collected by the cameras cover the entire vehicle body. More specifically, the light source adopts a binary stripe LED light source / multiple white light LED light sources to illuminate in an interval combination manner. During the detection process, the car passes through the tunnel, and the robotic arm handheld image acquisition component switches between the points to be tested. The fixed bracket image acquisition component collects the vehicle body image at the fixed points to be tested. The light source irradiates the surface of the vehicle body to produce a binary stripe light band. The defect area will undergo a more obvious change relative to the surrounding normal paint surface, thereby highlighting the defect area and facilitating defect detection, which can effectively enhance the characteristics of defects such as protrusions, depressions, particles, and shrinkage cavities. At the same time, the camera collects the stripe light band image reflected on the vehicle body that is displaced relative to the detection system in real time. The installation between multiple cameras requires that the images collected by the cameras cover the entire illuminated cross-section of the vehicle body, and the field of view between cameras must be continuous to prevent data splicing from missing data, affecting the defect detection effect. At the same time, a suitable camera lens is selected to cover a wider detection area within the depth of field range, and the spatial resolution of the camera is improved as much as possible so that smaller defects can be detected. Specifically, the camera layout can be simulated based on the body CAD model to check the camera's field of view on the body model (which can be displayed on the body surface in the simulation environment). If there is a missing area, consider adjusting the camera position or angle. If the missing area is too large, consider increasing the camera configuration until a better camera layout is achieved. The actual system is built based on the simulation layout results, and fine-tuned based on some inherent features of the body and hardware structure. There needs to be a certain overlap between adjacent cameras.

[0065] In one embodiment, Figure 2 As shown, a dynamic detection method for vehicle body paint defects is provided, and the method is applied to Figure 1 The controller in the example is used to illustrate the following steps:

[0066] S200: Acquire a first vehicle body image acquired by a fixed bracket image acquisition component and a second vehicle body image acquired by a robot arm handheld image acquisition component.

[0067] A fixed bracket image acquisition component refers to an image acquisition component fixed at a preset specific position, such as a camera fixed at a preset specific position. In actual applications, a fixed bracket method can be used to set up an image acquisition component to capture vehicle body images for the left and right sides of the vehicle body with relatively fixed height positions. A robotic arm handheld image acquisition component refers to an image acquisition component that is set on a robotic arm and can adjust its position as the robotic arm moves, such as a camera set at the end of the robotic arm. In actual applications, a relatively flexible robotic arm handheld method is used for areas whose height positions change greatly with the vehicle model, such as the roof, hood, and tail, and the detection points are adjusted in real time as the vehicle body moves.

[0068] S400: Identify a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image.

[0069] The image-based paint recognition technology can be used to identify the first body paint defect result in the first body image and the second body paint defect result in the second body image. Specifically, a trained paint defect recognition model can be used to identify paint defects in the image. Further, the defect position on the body can be identified and located, the defect category can be determined, and the defect size can be calculated. Subsequently, the specific position of the defect on the body can be determined by combining the body CAD three-dimensional model and the body reference position when taking pictures and the camera calibration parameters, and mapped to the body three-dimensional model, and the defect position information on each part of the body can be further displayed on the software UI.

[0070] S600: Jointly calibrate the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result.

[0071] Here, a joint calibration method is adopted for the cameras in the fixed bracket image acquisition component and the cameras in the robotic arm handheld image acquisition component to obtain the intrinsic parameters of all cameras and the camera extrinsic parameters in a unified reference coordinate system (including rotation matrix R and translation matrix T, etc.). Furthermore, based on the calibration plate preset on the vehicle body, calibration data can be collected, and then feature point analysis can be performed on the calibration data to complete the calibration of all cameras in one coordinate system or a unified world coordinate system.

[0072] S800: Based on the camera joint calibration result, the first vehicle body paint surface defect result and the second vehicle body paint surface defect result are mapped to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint surface defect detection result.

[0073] Based on the result of the joint calibration of the S600 camera, the first body paint defect result and the second body paint defect result are mapped to the body three-dimensional coordinate system, so that a complete body paint defect detection result is obtained.

[0074] The above-mentioned dynamic detection method for vehicle body paint defects obtains a first vehicle body image captured by a fixed bracket image acquisition component and a second vehicle body image captured by a robotic arm handheld image acquisition component; identifies a first vehicle body paint defect result in the first vehicle body image and a second vehicle body paint defect result in the second vehicle body image; performs joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result; based on the camera joint calibration result, maps the first vehicle body paint defect result and the second vehicle body paint defect result to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint defect detection result. In the whole process, the paint defect detection compatible with the fixed bracket and the robotic arm handheld mode can realize dynamic and accurate vehicle body paint defect detection without the vehicle body remaining relatively still.

[0075] like Figure 3 As shown, in one embodiment, S800 includes:

[0076] S820: Based on the camera joint calibration result, map the first vehicle body paint surface defect result to the vehicle body three-dimensional coordinate system.

[0077] Camera joint calibration refers to unifying the imaging systems of multiple cameras (such as cameras mounted on a robotic arm and cameras in a fixed position) into a common reference coordinate system. This is usually achieved by photographing a set of calibration plates of known shape and position (such as a checkerboard calibration plate), and using the relationship between the positions of the feature points on these calibration plates in the image and their positions in the real world to calculate the intrinsic and extrinsic parameters of each camera (including focal length, optical center, distortion coefficient, and the position and posture of the camera in the world coordinate system). Using the camera joint calibration results, the first body paint defect results (such as scratches, pits, etc.) captured and identified by the fixed position camera can be converted from the two-dimensional image coordinate system to the three-dimensional coordinate system of the body. This step depends on the intrinsic and extrinsic parameters of the camera and the position information of the defect in the image.

[0078] S840: Perform hand-eye calibration on the camera in the image acquisition component of the robot handheld mode.

[0079] Hand-eye calibration is the process of determining the relative position and posture between the robot's end effector (such as a camera) and the robot itself. Since the robot may move during the inspection process, hand-eye calibration is required to ensure that the camera can accurately reflect the position of objects in its field of view regardless of the robot's position. Hand-eye calibration usually also involves photographing a calibration plate, but in this case the calibration plate needs to be moved to different positions by the robot for photographing.

[0080] S860: Based on the hand-eye calibration result and the camera joint calibration result, the second vehicle body paint surface defect result is mapped to the vehicle body three-dimensional coordinate system.

[0081] Based on the hand-eye calibration results and the camera joint calibration results, the second body paint defect results captured and identified by the robot arm's handheld camera can be mapped to the body's three-dimensional coordinate system. This step not only takes into account the camera's imaging characteristics, but also the robot arm's motion state, thereby achieving accurate positioning of the defects captured by the mobile camera.

[0082] Specifically, for the method of using a robotic arm holding a vision unit, hand-eye calibration is also required. Specifically, it is necessary to collect the image of the calibration plate taken by the camera and save the position and posture of the robotic arm in the corresponding teach pendant. The transformation relationship between the camera coordinate system and the robotic arm end coordinate system and the base coordinate system is obtained through calculation, and finally the image pixel coordinates at different robotic arm points are converted into the three-dimensional coordinates of the reference coordinate system.

[0083] S880: Collect and map defect positions in the three-dimensional coordinate system of the vehicle body to obtain vehicle body paint defect detection results.

[0084] Finally, all defect positions mapped to the three-dimensional coordinate system of the vehicle body are collected to form a complete body paint defect detection result. This result not only contains the type and size information of the defects, but also their precise location information on the vehicle body, providing an important reference for subsequent repair work.

[0085] like Figure 3 As shown, in one embodiment, S600 includes:

[0086] S620: Obtain calibration data of a standard vehicle body captured by a camera in a fixed bracket image acquisition component and a camera in a robotic arm handheld image acquisition component. Calibration plates are distributed on the standard vehicle body, and each calibration plate corresponds to a unique ID.

[0087] Standard sample body: Select a standard sample body with known geometric and surface characteristics as the calibration object. This sample body should be representative of the actual production body, and its surface needs to be flat and uniform enough to accurately place the calibration plate. Calibration plate: Multiple calibration plates are distributed on the standard sample body, and each calibration plate has a unique ID for easy identification and distinction. The calibration plate is usually a flat plate with known geometric shapes and patterns (such as checkerboards, circular dot arrays, etc.), which are easy to detect and identify in the image.

[0088] Use the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to photograph the calibration plate on the standard sample vehicle body from different angles and positions. Ensure that each camera can capture a sufficient number of calibration plate images and that these images contain enough feature points for subsequent processing. Record the calibration plate images captured by each camera and their corresponding ID information, as well as the camera parameters during shooting (such as focal length, exposure time, etc.).

[0089] S640: Based on the calibration data, jointly calibrate the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result.

[0090] Joint calibration specifically includes the following stages: 1) Feature point extraction: First, extract feature points from the calibration plate image taken by each camera. These feature points are usually corner points, center points on the calibration plate, and other points that are easy to detect and locate. 2) Matching and correspondence: Then, according to the ID information of the calibration plate, match and correspond the feature points in the images from different cameras. This usually involves image processing and computer vision techniques, such as feature descriptor matching, RANSAC algorithm, etc., to eliminate mismatches and noise. 3) Optimization solution: Using the matched feature point pairs, the internal and external parameters of the camera are solved by optimization algorithms (such as least squares method, iterative closest point algorithm, etc.). These parameters include the focal length, optical center position, distortion coefficient, and position and posture of the camera in the world coordinate system. 4) Joint calibration results: Finally, the parameters of the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component are unified into a common reference coordinate system to obtain the camera joint calibration result. This result describes how each camera maps the image it captures to the three-dimensional coordinate system in the real world.

[0091] In general, in practical applications, the entire calibration process includes the following: small calibration plates (such as AprilTag, ArUco, etc.) are evenly distributed on the body of the standard sample car. Each small calibration plate has its own ID, which ensures that each calibration plate detected by the camera has a unique ID. After the calibration plate is pasted, the body is sent to the detection system, and the body is moved according to the standard process and multiple calibration data are collected at equal intervals. After image acquisition, the feature points in the calibration data are extracted. The ID of the calibration plate captured can be used to link adjacent cameras, so that all cameras can be calibrated to one of the camera coordinate systems or a unified world coordinate system, thus completing the calibration of all cameras. After the initial calibration of the camera, the camera intrinsic parameters M, distortion coefficient D and camera extrinsic parameters (rotation matrix R, translation matrix T) in the reference coordinate system can be obtained.

[0092] In one embodiment, the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component are jointly calibrated based on the calibration data, and the camera joint calibration result is obtained including:

[0093] Step 1: Use the calibration feature point detection algorithm to extract the corner feature coordinates and corresponding IDs in the calibration data.

[0094] Use a calibration feature point detection algorithm (such as AprilTag's detection algorithm) to extract corner feature coordinates and corresponding IDs from the calibration data. These corner points are usually obvious and easy to identify points on the calibration board, such as the corners of a chessboard. Each corner feature coordinate is associated with its unique ID on the calibration board, which helps with subsequent feature point matching.

[0095] Step 2: Traverse the corner feature coordinates based on the same ID to generate feature point matching pairs of different cameras.

[0096] Based on the same ID, the corner feature coordinates of all cameras are traversed to generate feature point matching pairs between different cameras. This step is the key to ensure that images from different cameras can be correctly matched. Some matching strategies may be used in the matching process, such as nearest neighbor matching, bidirectional matching, etc., to reduce the occurrence of mismatching.

[0097] Step 3: Generate an essential matrix based on the multi-view sparse reconstruction algorithm and according to the corner feature coordinates and feature point matching pairs.

[0098] Using the multi-view sparse reconstruction algorithm, the Essential Matrix is ​​generated based on the matched feature point coordinate pairs. The Essential Matrix describes the geometric relationship between the two camera views. It contains the camera's rotation and translation information (but not directly given). Solving the Essential Matrix usually involves some mathematical optimization methods, such as the Eight-Point Algorithm.

[0099] Step 4: Solve the rotation matrix R and translation vector of each camera based on the essential matrix and PnP algorithm.

[0100] According to the essential matrix and PnP (Perspective-n-Point) algorithm, the rotation matrix R and translation vector t of each camera are solved. The PnP algorithm is a method for estimating the camera pose using n known three-dimensional points and their projections on the image.

[0101] Step 5: Restore all camera poses based on the rotation matrix R and translation vector of each camera.

[0102] According to the rotation matrix R and translation vector t of each camera, the pose of all cameras is restored. The pose of a camera refers to the position and attitude of the camera in the world coordinate system, which determines how the camera observes the world.

[0103] Step 6: Based on the restored camera poses, triangulation and bundle adjustment are used to obtain the optimized camera internal and external parameters.

[0104] Based on the restored camera pose, triangulation and bundle adjustment are used to optimize the camera's intrinsic and extrinsic parameters. Triangulation is the process of estimating the position of a 3D point using matched feature point coordinate pairs, while bundle adjustment is a global optimization algorithm that optimizes the camera's intrinsic and extrinsic parameters and the position of a 3D point by minimizing the reprojection error.

[0105] In addition, in practical applications, the camera posture estimation and camera parameter calibration in the multi-view stereoscopic reconstruction system can also be completed through open source algorithm libraries such as colmap and OpenMVG.

[0106] In one embodiment, based on the camera joint calibration result, the first vehicle body paint defect result and the second vehicle body paint defect result are mapped to the vehicle body three-dimensional coordinate system, and the vehicle body paint defect detection result is obtained, including:

[0107] Step 1: Based on the camera joint calibration result, the first body paint defect result and the second body paint defect result, establish the reflected light from the defect to the camera optical center in the reference coordinate system.

[0108] First, based on the joint camera calibration results, we know the intrinsic parameters (focal length, optical center, etc.) and extrinsic parameters (camera position and posture in the world coordinate system) of each camera. For each defect in the first body paint defect result and the second body paint defect result, we need to determine its pixel coordinates in the image. Using the camera's intrinsic parameters and these pixel coordinates, we can calculate a reflected light ray (or ray) starting from the camera's optical center, passing through the defect point in the image, and pointing to the three-dimensional space. This ray represents the position in the three-dimensional space where the defect may be located from the camera's perspective.

[0109] Step 2: Calculate the intersection of the reflected light and the three-dimensional model data of the vehicle body through real-time simulation to obtain the real-time simulation calculation result.

[0110] Next, we need a 3D model of the vehicle body, which is usually a CAD model or point cloud data obtained by scanning. Using real-time simulation technology (such as ray tracing algorithms), we can simulate the interaction between these reflected rays and the 3D model data of the vehicle body. Through calculation, we find the intersection of each reflected ray and the 3D model data of the vehicle body. These intersections are the possible locations of defects in the 3D vehicle body coordinate system.

[0111] Step 3: According to the real-time simulation calculation results, the first body paint defect and the second body paint defect are mapped from the pixel coordinates in the image to the body three-dimensional coordinate system to obtain the body paint defect detection result.

[0112] Based on the results of real-time simulation calculations, we can map the coordinates of each defective pixel in the image to one or more points in the three-dimensional coordinate system of the vehicle body (depending on the number of intersections between the reflected light and the vehicle body). If there are multiple intersections (for example, when the defect is located on the curved surface of the vehicle body), further analysis may be required to determine the most likely defect location. This can be achieved by considering the shape, size, color and other characteristics of the defect, combined with the geometry of the vehicle body. Finally, all defect locations mapped to the three-dimensional coordinate system of the vehicle body are collected to form a complete body paint defect detection result. This result not only contains the location information of the defect in three-dimensional space, but may also include detailed information such as the type, size, and severity of the defect.

[0113] In one of the embodiments, based on the camera joint calibration result, the first vehicle body paint surface defect result and the second vehicle body paint surface defect result are mapped to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint surface defect detection result;

[0114] Step 1: Based on the camera joint calibration result, the first body paint defect result and the second body paint defect result are mapped to the body three-dimensional coordinate system to obtain the initial body paint defect detection result;

[0115] Step 2: Identify repeated defect points in the initial vehicle body paint defect detection results where the distance between defects is less than a preset error tolerance;

[0116] Step 3: Remove repeated defect points in the initial vehicle body paint defect detection results to obtain the vehicle body paint defect detection results.

[0117] In this embodiment, the initial vehicle body paint defect detection results are subjected to defect deduplication processing. Specifically, in the initial vehicle body paint defect detection results, there may be multiple cameras or the same camera capturing the same defect data at different times. After mapping to the vehicle body three-dimensional coordinate system, the data can be removed well. Considering the existence of errors in the coordinate mapping process, the error tolerance Δd is set. If the Euclidean distance between defects is less than the error tolerance, it is judged as a duplicate point. Generally, the tolerance is set as a fixed threshold, such as 10mm, and it actually needs to be fine-tuned according to the different coordinate mapping errors of the final system.

[0118] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0119] Based on the same inventive concept, the embodiment of the present application also provides a vehicle body paint defect dynamic detection device for implementing the vehicle body paint defect dynamic detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more vehicle body paint defect dynamic detection device embodiments provided below can refer to the limitations of the vehicle body paint defect dynamic detection method above, and will not be repeated here.

[0120] In one embodiment, Figure 4 As shown, a vehicle body paint surface defect dynamic detection device is provided, comprising:

[0121] An image acquisition module 200 is used to acquire a first vehicle body image acquired by a fixed bracket image acquisition component and a second vehicle body image acquired by a mechanical arm handheld image acquisition component;

[0122] The recognition module 400 is used to recognize a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image;

[0123] The calibration module 600 is used to jointly calibrate the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result;

[0124] The detection module 800 is used to map the first vehicle body paint defect result and the second vehicle body paint defect result to the vehicle body three-dimensional coordinate system based on the camera joint calibration result to obtain the vehicle body paint defect detection result.

[0125] In one of the embodiments, the detection module 800 is also used to map the first vehicle body paint defect result to the vehicle body three-dimensional coordinate system based on the camera joint calibration result; perform hand-eye calibrate on the camera in the robot arm handheld image acquisition component; map the second vehicle body paint defect result to the vehicle body three-dimensional coordinate system based on the hand-eye calibration result and the camera joint calibration result; and collect the defect positions mapped in the vehicle body three-dimensional coordinate system to obtain the vehicle body paint defect detection result.

[0126] In one of the embodiments, the calibration module 600 is also used to obtain calibration data of a standard sample vehicle body captured by the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component. Calibration plates are distributed on the standard sample vehicle body, and each calibration plate corresponds to a unique ID; based on the calibration data, the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component are jointly calibrated to obtain a camera joint calibration result.

[0127] In one embodiment, the calibration module 600 is also used to extract the corner point feature coordinates and corresponding IDs in the calibration data using a calibration feature point detection algorithm; traverse the corner point feature coordinates based on the same ID to generate feature point matching pairs of different cameras; generate an essential matrix based on a multi-view sparse reconstruction algorithm and according to the corner point feature coordinates and the feature point matching pairs; solve the rotation matrix R and translation vector of each camera according to the essential matrix and the PnP algorithm; restore all camera poses according to the rotation matrix R and translation vector of each camera; based on the restored camera poses, triangulation and bundle adjustment are used to obtain optimized camera internal and external parameter results.

[0128] In one of the embodiments, the detection module 800 is also used to establish the reflected light from the defect to the camera optical center in the reference coordinate system based on the camera joint calibration result, the first body paint defect result and the second body paint defect result; obtain the real-time simulation calculation result by calculating the intersection of the reflected light and the body three-dimensional model data through real-time simulation; according to the real-time simulation calculation result, the first body paint defect and the second body paint defect are mapped from the pixel coordinates in the image to the body three-dimensional coordinate system to obtain the body paint defect detection result.

[0129] In one of the embodiments, the detection module 800 is also used to map the first body paint defect result and the second body paint defect result to the three-dimensional coordinate system of the body based on the camera joint calibration result to obtain an initial body paint defect detection result; identify repeated defect points in the initial body paint defect detection result where the distance between defects is less than a preset error tolerance value; eliminate repeated defect points in the initial body paint defect detection result to obtain a body paint defect detection result.

[0130] Each module in the above-mentioned vehicle body paint defect dynamic detection device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0131] In one of the embodiments, the present application also provides a vehicle body paint defect dynamic detection system, including a fixed bracket image acquisition component, a mechanical arm handheld image acquisition component, and a processing and computing unit;

[0132] The fixed bracket image acquisition component acquires a first vehicle body image and sends the first vehicle body image to the processing and computing unit; the robotic arm handheld image acquisition component acquires a second vehicle body image and sends the second vehicle body image to the processing and computing unit, and the processing and computing unit uses the above-mentioned vehicle body paint defect dynamic detection method to perform vehicle body paint defect dynamic detection.

[0133] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a dynamic detection method for vehicle body paint defects is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0134] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned dynamic detection method for vehicle body paint defects when executing the computer program.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned dynamic detection method for vehicle body paint defects is implemented.

[0137] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned vehicle body paint defect dynamic detection method when executed by a processor.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for dynamic detection of vehicle body paint defects, characterized in that: The method comprises: Acquire a first vehicle body image acquired by the fixed bracket image acquisition component and a second vehicle body image acquired by the robotic arm handheld image acquisition component; Identifying a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image; Performing joint calibration on the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result; Based on the camera joint calibration result, the first vehicle body paint surface defect result and the second vehicle body paint surface defect result are mapped to the vehicle body three-dimensional coordinate system to obtain the vehicle body paint surface defect detection result.

2. The method according to claim 1, characterized in that The step of mapping the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system based on the camera joint calibration result to obtain a vehicle body paint defect detection result includes: Based on the camera joint calibration result, mapping the first vehicle body paint surface defect result to a vehicle body three-dimensional coordinate system; Performing hand-eye calibration on the camera in the image acquisition component of the robotic arm handheld mode; Based on the hand-eye calibration result and the camera joint calibration result, mapping the second vehicle body paint surface defect result to the vehicle body three-dimensional coordinate system; The defect positions mapped in the three-dimensional coordinate system of the vehicle body are collected to obtain the vehicle body paint defect detection result.

3. The method according to claim 1, characterized in that The joint calibration of the camera in the fixed support image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain the camera joint calibration result includes: Obtaining calibration data of a standard vehicle body collected by a camera in the fixed bracket image acquisition component and a camera in the robotic arm handheld image acquisition component, wherein calibration plates are distributed on the standard vehicle body, and each calibration plate corresponds to a unique ID; Based on the calibration data, the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component are jointly calibrated to obtain a camera joint calibration result.

4. The method according to claim 3, characterized in that The step of jointly calibrating the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component based on the calibration data to obtain the camera joint calibration result includes: Using a calibration feature point detection algorithm to extract corner point feature coordinates and corresponding IDs in the calibration data; Traversing the corner point feature coordinates based on the same ID to generate feature point matching pairs of different cameras; Generate an essential matrix based on a multi-view sparse reconstruction algorithm and according to the corner point feature coordinates and the feature point matching pairs; Solve the rotation matrix R and translation vector of each camera according to the essential matrix and the PnP algorithm; Restore all camera poses according to the rotation matrix R and translation vector of each camera; Based on the restored camera poses, triangulation and bundle adjustment are used to obtain the optimized camera internal and external parameters.

5. The method according to claim 1, characterized in that The step of mapping the first vehicle body paint defect result and the second vehicle body paint defect result to a vehicle body three-dimensional coordinate system based on the camera joint calibration result to obtain a vehicle body paint defect detection result includes: Based on the camera joint calibration result, the first vehicle body paint surface defect result and the second vehicle body paint surface defect result, establishing a reflected light from the defect to the optical center of the camera in the reference coordinate system; By real-time simulation calculation of the intersection of the reflected light and the three-dimensional model data of the vehicle body, a real-time simulation calculation result is obtained; According to the real-time simulation calculation result, the first vehicle body paint surface defect and the second vehicle body paint surface defect are mapped from pixel coordinates in the image to the vehicle body three-dimensional coordinate system to obtain a vehicle body paint surface defect detection result.

6. The method according to claim 1, characterized in that Based on the camera joint calibration result, mapping the first vehicle body paint surface defect result and the second vehicle body paint surface defect result to a vehicle body three-dimensional coordinate system to obtain a vehicle body paint surface defect detection result; Based on the camera joint calibration result, mapping the first vehicle body paint surface defect result and the second vehicle body paint surface defect result to a vehicle body three-dimensional coordinate system to obtain an initial vehicle body paint surface defect detection result; Identifying repeated defect points in the initial vehicle body paint defect detection result where the distance between defects is less than a preset error tolerance value; The repeated defect points in the initial vehicle body paint surface defect detection result are eliminated to obtain the vehicle body paint surface defect detection result.

7. A dynamic detection device for vehicle body paint defects, characterized in that: The device comprises: An image acquisition module, used to acquire a first vehicle body image acquired by the fixed bracket image acquisition component and a second vehicle body image acquired by the mechanical arm handheld image acquisition component; an identification module, configured to identify a first vehicle body paint surface defect result in the first vehicle body image and a second vehicle body paint surface defect result in the second vehicle body image; A calibration module, used for jointly calibrating the camera in the fixed bracket image acquisition component and the camera in the robotic arm handheld image acquisition component to obtain a camera joint calibration result; A detection module is used to map the first vehicle body paint defect result and the second vehicle body paint defect result into a three-dimensional coordinate system of the vehicle body based on the camera joint calibration result to obtain a vehicle body paint defect detection result.

8. A dynamic detection system for vehicle body paint defects, characterized in that: It includes a fixed bracket image acquisition component, a mechanical arm handheld image acquisition component and a processing and computing unit; The fixed bracket image acquisition component acquires a first vehicle body image and sends the first vehicle body image to the processing and computing unit; the robotic arm handheld image acquisition component acquires a second vehicle body image and sends the second vehicle body image to the processing and computing unit, and the processing and computing unit uses the method described in any one of claims 1 to 6 to perform dynamic detection of vehicle body paint defects.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images