Methods, media and equipment for detecting paint defects

Abstract

This invention discloses a method, medium, and device for detecting paint defects. It involves creating an initial virtual scene by acquiring the intrinsic parameter matrices of each acquisition device and the first transformation matrix between them. Based on these matrices, the virtual acquisition devices and a car model are imported. The relative poses between the calibrated car and each acquisition device are obtained using prior engineering data. The relative poses between the virtual acquisition devices and the car model are then corrected based on these calibrated poses, resulting in the final virtual scene. Several sets of current 2D images of the car are acquired, and the coordinates of defect points in each set of 2D images are obtained and mapped to the car model to obtain the coordinates and normals of the defect points. This invention solves the problems of low efficiency, poor accuracy, insufficient compatibility, and limited precision in existing technologies for detecting automotive paint defects.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a method, medium, and equipment for detecting paint surface defects. Background Technology

[0002] In the automotive manufacturing industry, the painting process is a crucial step in ensuring the aesthetics and performance of the vehicle body. Paint quality not only affects the vehicle's appearance but also directly impacts its performance indicators such as abrasion resistance and corrosion resistance. However, traditional manual inspection methods have many shortcomings, such as visual fatigue, subjective misjudgment, low efficiency, and poor working environments, which limit the improvement of production quality and efficiency. Existing automated inspection methods for detecting defects in automotive paint, such as using robotic arms carrying camera light sources, still have the following problems:

[0003] Long inspection time: The vehicle body needs to remain stationary at the inspection station, which results in a long inspection time and affects the overall cycle time of the production line.

[0004] Poor compatibility: The tunnel-type vehicle body paint defect detection method is cumbersome in the calibration process and has poor compatibility with new models, which is not conducive to quickly adapting to changes in the production line.

[0005] Insufficient accuracy: Existing tunneling methods rely solely on encoder information from the vehicle body slide rails when calculating changes in vehicle body position, resulting in poor accuracy and failing to meet the requirements for high-precision detection.

[0006] Therefore, how to improve the existing technology for detecting defects in automotive paint, which suffers from low efficiency, poor accuracy, insufficient compatibility, and limited precision, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] Based on this, the purpose of this application is to provide a method, medium, and device for detecting paint defects, so as to solve at least one of the technical problems mentioned in the background art.

[0008] In a first aspect, this application provides a method for detecting paint surface defects, including:

[0009] Several sets of calibrated vehicle surface images were acquired to obtain the intrinsic parameter matrix of each acquisition device and the first transformation matrix between each acquisition device.

[0010] Create an initial virtual scene, and import the virtual acquisition devices and car models based on the intrinsic parameter matrices of each acquisition device and the first transformation matrix between each acquisition device;

[0011] Based on prior engineering data, the 3D surface point cloud of the calibrated vehicle is obtained. Several 2D image points in any surface image and the corresponding 3D surface point cloud on the calibrated vehicle are selected to construct feature point pairs and obtain the relative pose between the calibrated vehicle and each acquisition device.

[0012] The relative poses between the virtual acquisition devices and the car model are corrected based on the relative poses between the calibrated car and each acquisition device to obtain the final virtual scene.

[0013] Acquire several sets of current 2D images of the car, obtain the coordinates of the defect points in each set of 2D images, and map them to the car model to obtain the coordinates and normals of the defect points.

[0014] Furthermore, the steps of obtaining the intrinsic parameter matrix of each acquisition device and the first transformation matrix between the acquisition devices include:

[0015] Each acquisition device is installed in a set position, and the calibration vehicle is controlled to move in a set direction. Each acquisition device scans the surface of the calibration vehicle at a preset sampling distance to obtain several sets of surface images of the calibration vehicle.

[0016] Based on the surface images of all calibrated vehicles, several sets of relative poses between the calibrated vehicles and the acquisition devices are obtained. The acquisition devices are then transformed to the same coordinate system to obtain several relative acquisition devices.

[0017] Image analysis was performed on the surface images of each group of calibrated vehicles to obtain the image feature points in each group of surface images;

[0018] Three-dimensional reconstruction is performed on the image feature points in the coordinate systems of each relative acquisition device to obtain the first transformation matrix between the coordinate systems of each relative acquisition device and the intrinsic parameter matrix of each relative acquisition device.

[0019] Further, the steps of creating an initial virtual scene, importing the virtual acquisition devices and the car model based on the intrinsic parameter matrices of each acquisition device and the first transformation matrix between the acquisition devices, include:

[0020] Import the data acquisition device model and the design model of the calibrated car, which are the same models as those in the actual scene, into the virtual scene to obtain virtual data acquisition device and car models.

[0021] Configure the parameters of the virtual acquisition devices according to the intrinsic parameter matrix, and obtain the relative poses between the virtual acquisition devices in the virtual scene according to the first transformation matrix;

[0022] Based on the location of the car in the real scene, place the corresponding car model in the virtual scene.

[0023] Further, the steps for obtaining the relative pose between the calibrated vehicle and each data acquisition device include:

[0024] The 3D surface point cloud of the calibrated vehicle was obtained based on prior engineering data;

[0025] Select several 2D image points and the corresponding 3D surface point cloud from any surface image to construct several feature point pairs;

[0026] The feature point pairs are reprojected to obtain the second transformation matrix between the coordinate system of the acquisition device and the coordinate system of the calibration vehicle corresponding to the surface image, and the relative pose between the calibration vehicle and each acquisition device is obtained according to the first transformation matrix between each acquisition device.

[0027] Furthermore, the step of selecting several 2D image points and corresponding 3D surface point clouds from any surface image to construct several feature point pairs includes:

[0028] The coordinates of 2D feature points in the surface image are obtained based on the image detection algorithm;

[0029] The 3D surface point cloud is converted into a triangular mesh to obtain the points in the feature hole region;

[0030] Fit a plane to the points in the feature hole region and extract the plane contour;

[0031] The center of a circle is fitted to the planar contour to obtain 3D feature points, and several feature point pairs are constructed based on each 2D feature point and its corresponding 3D feature point.

[0032] Further steps to obtain the coordinates and normal of the defect point include:

[0033] Construct and train a neural network model that takes a 2D image as input and the coordinates of the defect point as output;

[0034] Acquire several sets of current 2D images of the car, and input all 2D images into a neural network model to obtain the coordinates of all defect points in each 2D image;

[0035] Project the coordinates of each defect point onto the car model to obtain the corresponding 3D coordinates, which are the defect model points;

[0036] Obtain the coordinates of several point clouds that are closest to each defect model point, and calculate the normal of each defect model point.

[0037] Furthermore, the step of obtaining the normals of each defect model point may optionally include:

[0038] Set the neighborhood size and obtain the coordinates of all points within the set neighborhood for each defect model point to construct the covariance matrix;

[0039] Eigenvalue decomposition is performed on the covariance matrix corresponding to each defect model point to obtain several eigenvalues ​​and eigenvectors corresponding to each defect model point.

[0040] Obtain the minimum eigenvalue corresponding to each defect model point, and use the eigenvector corresponding to the minimum eigenvalue as the normal vector of each defect model point.

[0041] Furthermore, the covariance matrix is:

[0042]

[0043] where C is the covariance matrix, k is the set neighborhood size, N is the number of points within the k-neighborhood of each defect model point, p i is the coordinate of the i-th point, 0 < i ≤ N, u is the average coordinate of all points within the k-neighborhood, and T is the matrix transpose symbol.

[0044] In a second aspect, the present application also provides a computer storage medium storing executable program code; the executable program code is used to execute the paint surface defect detection method according to any one of the first aspect.

[0045] In a third aspect, the present application also provides a terminal device including a memory and a processor; the memory stores program code executable by the processor; the program code is used to execute the paint surface defect detection method according to any one of the first aspect.

[0046] A paint surface defect detection method, medium and device provided by the present invention collect a number of calibrated automotive surface images to obtain the internal parameter matrix of each acquisition device and the first transformation matrix between each acquisition device, create an initial virtual scene, import virtual acquisition devices and automotive models according to the internal parameter matrix of each acquisition device and the first transformation matrix between each acquisition device, build an automotive model in the virtual scene to improve the efficiency of automotive paint surface defect detection, not limited to a certain type of vehicle, improve compatibility, obtain the 3D surface point cloud of the calibrated vehicle according to prior engineering data, select a number of 2D image points in any one surface image and the corresponding 3D surface point cloud on the calibrated vehicle to construct feature point pairs, obtain the relative pose between the calibrated vehicle and each acquisition device, correct the relative pose between each virtual acquisition device and the automotive model according to the relative pose between the calibrated vehicle and each acquisition device to obtain the final virtual scene, obtain a number of current automotive 2D images, obtain the coordinates of defect points in each group of 2D images, and map them to the automotive model to obtain the coordinates and normal vectors of the defect points. Mapping the defect points in the 2D image to the 3D model can more accurately locate the defects and improve the accuracy of detection. It solves the problems of low efficiency, poor accuracy, insufficient compatibility and limited precision in detecting automotive paint surface defects in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of the paint surface defect detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0049] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., these directional indications are only used to explain the relative positional relationships and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly. Furthermore, if the embodiments of the present invention involve descriptions such as "first," "second," "S1," "S2," "step one," "step two," etc., these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance, or implicitly indicating the number of technical features indicated or the order of method execution. Those skilled in the art will understand that anything that does not violate the inventive concept and is within the scope of the present invention should be included in the protection scope of the present invention.

[0050] like Figure 1 As shown, the present invention provides a method for detecting paint surface defects:

[0051] S1: Acquire several sets of surface images of the calibrated vehicle to obtain the intrinsic parameter matrix of each acquisition device and the first transformation matrix between each acquisition device;

[0052] Specifically, the sampling distance can be set, but is not limited to, by installing the coded pattern on the car surface. The center point or corner points of the coded pattern can be selected as image feature points. The car is controlled to move within the scanning range of the acquisition device. Each time the car moves a sampling distance, the acquisition device scans the car surface once, obtaining a set of car surface images. This process continues until the car leaves the scanning range of the acquisition device, resulting in several sets of car surface images. Image analysis is performed on all car surface images according to any existing technical solution to obtain the coordinates and corresponding numbers of the image feature points in each surface image under the coordinate system of each acquisition device. This allows for the acquisition of the intrinsic parameter matrix of each acquisition device and the first transformation matrix between each acquisition device. The sampling distance can be arbitrarily set by those skilled in the art. The coded pattern includes, but is not limited to, structured light coded patterns that can be used for pixel-by-pixel encoding, such as QR codes, step stripe patterns, Gray code stripe patterns, sine stripe patterns, binary stripe patterns, and phase-shift stripe patterns. The acquisition device can include commonly used acquisition devices such as digital cameras, video cameras, 3D line scan cameras, 3D structured light cameras, and TOF cameras.

[0053] Preferably, the step of acquiring several sets of surface images of the calibrated vehicle to obtain the intrinsic parameter matrix of each acquisition device and the first transformation matrix between the acquisition devices may include:

[0054] S11: Install each acquisition device in the set position, control the calibration vehicle to move in the set direction, and each acquisition device scans the surface of the calibration vehicle at a preset sampling distance to obtain several sets of surface images of the calibration vehicle.

[0055] Specifically, the location, direction, and preset sampling distance can be set, but are not limited to, and the acquisition device can be installed at the set location. Then, the calibration vehicle is controlled to move along the set direction. The calibration vehicle is scanned for the first time when it enters the tunnel. Then, the surface of the calibration vehicle is scanned at preset sampling distances. Each scan can obtain a set of corresponding images until the calibration vehicle leaves the tunnel, thus obtaining several sets of surface image data of the calibration vehicle. The location, direction, and preset sampling distance can be set arbitrarily by those skilled in the art.

[0056] For example, a calibration system may be provided, including a tunnel, several acquisition devices, a light source, a vehicle body, and an encoder. The acquisition devices are installed in the cross-section of the middle of the tunnel, the light source is evenly installed on the surface of the tunnel, and a preset coded pattern is placed on the vehicle body, so that the vehicle body moves along the tunnel. The encoder obtains the distance the vehicle body moves, so that the acquisition devices scan the vehicle body at set intervals to obtain a set of surface images of the car. Each acquisition device scans the surface of the vehicle body several times to obtain several sets of surface images of the car body in the coordinate system of each acquisition device.

[0057] S12: Based on the surface images of all calibrated vehicles, obtain several sets of relative poses between the calibrated vehicles and the acquisition devices, transform each acquisition device to the same coordinate system, and obtain several relative acquisition devices;

[0058] Specifically, according to step S11, several sets of surface images of the calibrated car in the coordinate system of the acquisition device can be obtained. The relative poses of the calibrated car and the corresponding acquisition device are different in each surface image, so several pairs of relative poses between the calibrated car and each acquisition device can be obtained. A calibration coordinate system is constructed with any point on the calibrated car as the origin. The acquisition devices corresponding to the calibrated car in all surface images are transformed to the calibration coordinate system, so several relative acquisition devices can be obtained.

[0059] For example, taking a data acquisition device as an example, the device controls the calibration car to move in a set direction. The data acquisition device acquires three surface images of the calibration car. Each surface image corresponds to a different relative pose between the calibration car and the data acquisition device. If the relative pose between the data acquisition device and the calibration car is transformed to the same coordinate system each time the surface of the calibration car is scanned, three data acquisition devices in the same coordinate system can be obtained, which are relative data acquisition devices.

[0060] S13: Perform image analysis on the surface images of each group of calibrated vehicles to obtain the image feature points in each group of surface images;

[0061] Specifically, but not limited to using the corner points or center points of the coded pattern as feature points, image analysis can be performed on the surface images of each group of calibrated vehicles using any existing image analysis method to obtain the image feature points in each surface image.

[0062] S14: Perform 3D reconstruction of image feature points in the coordinate systems of each relative acquisition device to obtain the first transformation matrix between the coordinate systems of each relative acquisition device and the intrinsic parameter matrix of each relative acquisition device.

[0063] Specifically, but not limited to, 3D reconstruction of image feature points in the coordinate systems of each relative acquisition device can be performed using any existing 3D reconstruction method to obtain the first transformation matrix between the coordinate systems of each relative acquisition device and the intrinsic parameter matrix of each relative acquisition device. The 3D reconstruction method may include commonly used methods and software for 3D reconstruction such as colmap-based methods, OpenMVG, OpenCV's SfM module, and Meshroom.

[0064] S2: Create an initial virtual scene, and import the virtual acquisition devices and car models based on the intrinsic parameter matrices of each acquisition device and the first transformation matrix between each acquisition device;

[0065] Specifically, optional but not limited to the following: a person skilled in the art places corresponding virtual acquisition devices and car models in the virtual scene according to the positions of each relative acquisition device and the calibrated car in step S1, corrects the relative pose of each virtual acquisition device according to the first transformation matrix between the coordinate systems of each relative acquisition device in step S1, and configures the parameters of the corresponding virtual acquisition devices according to the intrinsic parameter matrix of each relative acquisition device, so that the image information acquired by the acquisition devices in the virtual scene is the same as that in the real scene.

[0066] Preferably, the steps of creating an initial virtual scene and importing the virtual acquisition devices and car models based on the intrinsic parameter matrices of each acquisition device and the first transformation matrix between the acquisition devices may include:

[0067] S21: Import the data acquisition device model and the design model of the calibrated car, which are the same models as those in the actual scene, into the virtual scene to obtain the virtual data acquisition device and car model;

[0068] Specifically, but not limited to, importing the same model of acquisition equipment and the design model of the calibrated car as the actual scene from the virtual software platform library into the initial virtual scene to obtain virtual acquisition equipment and car models; the virtual software platform library may include commonly used virtualization platforms such as Unity, Unreal Engine, Godot, and CryEngine, and the order can be set arbitrarily by those skilled in the art.

[0069] S22: Configure the virtual acquisition device parameters according to the intrinsic parameter matrix, and obtain the relative poses between each virtual acquisition device in the virtual scene according to the first transformation matrix;

[0070] Specifically, but not limited to, setting the corresponding virtual acquisition device parameters according to the intrinsic parameter matrices of each relative acquisition device calibrated in step S1, so that when the virtual acquisition device takes pictures in the virtual scene, it can achieve the same shooting effect as the acquisition device in the real scene, so as to simulate various operations of the acquisition device in the real scene and reduce costs. At the same time, since the first transformation matrix between each relative acquisition device is also obtained in step S1, the relative pose between each virtual acquisition device can be obtained according to the first transformation matrix.

[0071] S23: Based on the position of the car in the real scene, place the corresponding car model in the virtual scene.

[0072] Specifically, optionally, but not limited to, placing a car model in a virtual scene by a person skilled in the art based on the position of the car calibrated in the real scene. It is worth noting that, since subsequent steps also include pose correction of the car model in the virtual scene, the position of the car model should not deviate too much from the initial position of the calibrated car in the real scene when a person skilled in the art places the car model.

[0073] S3: Based on prior engineering data, obtain the 3D surface point cloud of the calibrated vehicle, select several 2D image points from any surface image and the corresponding 3D surface point cloud on the calibrated vehicle to construct feature point pairs and obtain the relative pose between the calibrated vehicle and each acquisition device.

[0074] Specifically, the method may, but is not limited to, obtaining the 3D surface point cloud of the calibrated vehicle based on prior engineering data. Then, from the several surface images obtained in step S1, any number of 2D image points are selected from any one of the surface images. Based on the one-to-one correspondence between the pixels in the surface image of the calibrated vehicle and the 3D surface point cloud, the corresponding number of 3D point clouds in the 3D surface point cloud are obtained. Feature point pairs are constructed based on the one-to-one correspondence between the 2D image points and the 3D surface point clouds to obtain the second transformation matrix between the calibrated vehicle and the acquisition device corresponding to the selected surface image. The current pose of the calibrated vehicle in the coordinate system of each acquisition device is obtained based on the first transformation matrix between each acquisition device. The prior engineering data may include engineering data such as the vehicle body digital model design and assembly information.

[0075] Preferably, since there are several feature holes on the car, and feature holes are usually relatively stable features that can still maintain good recognition ability under the influence of noise and interference, thereby improving the robustness of the system, when constructing feature point pairs based on the one-to-one correspondence between each 2D image point and 3D model point, it is preferable to use the feature holes in the surface image as the selected 2D image points.

[0076] A further preferred step, which involves obtaining a 3D surface point cloud of the calibrated vehicle based on prior engineering data, selecting several 2D image points from any surface image and the corresponding 3D surface point cloud on the calibrated vehicle to construct feature point pairs, and obtaining the relative pose between the calibrated vehicle and each acquisition device, may include:

[0077] S31: Obtain the 3D surface point cloud of the calibration vehicle based on prior engineering data, and construct the calibration vehicle coordinate system;

[0078] Specifically, the 3D surface point cloud of the calibrated vehicle can be directly obtained from prior engineering data such as known vehicle body digital model design and assembly information, and the calibrated vehicle coordinate system can be constructed with any point in the 3D surface point cloud as the origin.

[0079] S32: Select several 2D image points and corresponding 3D surface point clouds from any surface image to construct several feature point pairs;

[0080] Specifically, since there are several feature areas on the vehicle body, such as feature holes, the center of the feature hole can be selected as the feature point to obtain several feature points in any surface image and the corresponding feature points in the 3D surface point cloud, and to construct several feature point pairs.

[0081] Preferably, when constructing feature point pairs from feature points in the surface image and the 3D surface point cloud, step S32 may optionally include:

[0082] S321: Obtain the coordinates of 2D feature points in the surface image based on the image detection algorithm;

[0083] Specifically, since there are several feature holes of various shapes on the vehicle body, an image detection algorithm can be used to obtain the center points of each feature hole in the surface image, which are the 2D feature point coordinates. Preferably, an ellipse detection algorithm is used to detect circular feature holes in the surface image to obtain the center points of the circular feature holes as 2D feature points.

[0084] S322: Convert the 3D surface point cloud into a triangular mesh to obtain the feature hole region points;

[0085] S323: Fit a plane to the points in the feature hole region and extract the plane contour;

[0086] S324: Fit the center of the circle to the planar contour to obtain 3D feature points, and construct several feature point pairs based on each 2D feature point and the corresponding 3D feature point.

[0087] Specifically, the 3D surface point cloud can be converted into a triangular mesh using Delaunay triangulation to ensure that the mesh covers the surrounding area of ​​the feature hole. Then, a person skilled in the art selects points around the feature hole from the triangular mesh as feature hole region points. The selected feature hole region points are then fitted using plane fitting methods such as least squares or RANSAC algorithm to obtain the local spatial plane equation of the feature hole. The boundary points of the feature hole can be identified by boundary extraction algorithms such as normal calculation and Canny edge detection to connect the boundary points and form the 3D contour of the feature hole. Finally, the circular model is fitted using least squares method, and the 3D center of the feature hole is calculated based on the extracted 3D contour points to obtain the 3D feature points.

[0088] S33: Reproject the feature point pairs to obtain the second transformation matrix between the coordinate system of the acquisition device and the coordinate system of the calibration vehicle corresponding to the surface image, and obtain the relative pose between the calibration vehicle and each acquisition device according to the first transformation matrix between each acquisition device.

[0089] Specifically, but not limited to, reprojecting the feature point pairs obtained in step S32 to minimize the distance between corresponding points, the second transformation matrix between the coordinate system of the acquisition device and the coordinate system of the calibration vehicle corresponding to the surface image is obtained, that is, the relative pose between the calibration vehicle and the acquisition device is obtained. Since the relative pose between each acquisition device has been obtained in step S1, the relative pose between the calibration vehicle and each acquisition device can be obtained according to the first transformation matrix between each acquisition device.

[0090] S4: Correct the relative poses between each virtual acquisition device and the car model based on the relative poses between the calibrated car and each acquisition device to obtain the final virtual scene;

[0091] Specifically, but not limited to, the pose of the car model in the virtual scene can be corrected based on the relative pose between the calibrated car and the acquisition device obtained in step S3, so that the relative pose between each device in the virtual scene is the same as the relative pose between each device in the real scene. This is so that when performing simulation operations in the virtual scene, the same results as in the real scene can be obtained, which greatly saves operating costs, improves operating accuracy, and improves operating efficiency.

[0092] S5: Obtain several sets of current 2D images of the car, get the coordinates of the defect points in each set of 2D images, and map them to the car model to obtain the coordinates and normals of the defect points.

[0093] Specifically, after obtaining the final virtual scene, the system can optionally, but is not limited to, control the car to be acquired to move in a set direction, acquire several sets of surface images of the current car according to step S1, then identify the coordinates of defect points in all surface images, map them to the car model, obtain the 3D coordinates of the defect points, then obtain the coordinates of several point clouds that are closest to each defect model point, construct the covariance equation, and perform eigenvalue decomposition on it to calculate the normal of each defect model point.

[0094] Preferably, the steps of acquiring several sets of current 2D images of the vehicle, obtaining the coordinates of defect points in each set of 2D images, and mapping them to the vehicle model to obtain the coordinates and normals of the defect points in the vehicle body coordinate system may include:

[0095] S51: Construct and train a neural network model that takes a 2D image as input and the coordinates of the defect point as output;

[0096] Specifically, optionally, but not limited to, constructing and training a neural network model that takes a 2D image as input and the coordinates of the defect point as output, based on any existing neural network model and existing technical solutions. It is worth noting that this step is preparatory work, which can be constructed and trained in advance, as long as it is constructed and trained before step S52, when the neural network model is needed for prediction.

[0097] S52: Obtain several sets of current 2D images of the car, and input all 2D images into the neural network model to obtain the coordinates of all defect points in each 2D image;

[0098] Specifically, it is optional, but not limited to, obtaining several sets of current 2D images of the car according to step S1, and then inputting all the obtained 2D images into the neural network model obtained in step S51 to obtain the coordinates of the defect points in the surface image coordinate system in each 2D image.

[0099] S53: Project the coordinates of each defect point onto the car model to obtain the corresponding 3D coordinates, which are the defect model points;

[0100] Specifically, it is possible to convert it into a ray according to the internal parameters of the acquisition device through ray tracing, obtain the intersection point between the ray and the vehicle model, and obtain the 3D model point corresponding to the defect point, that is, the defect model point.

[0101] S54: Obtain the coordinates of several point clouds closest to each defect model point, and calculate the normal vector of each defect model point.

[0102] Specifically, step S54 may include:

[0103] S541: Set the neighborhood size, obtain the coordinates of all points within the set neighborhood of each defect model point, and construct a covariance matrix;

[0104] Specifically, it is possible but not limited to setting the neighborhood size to k, obtaining the coordinates of all points within the k-neighborhood of each defect model point, and calculating the coordinate average u to construct the covariance matrix corresponding to each defect model point.

[0105] Preferably, the covariance matrix can be expressed as Equation 5-1:

[0106]

[0107] where C is the covariance matrix, N is the number of points within the k-neighborhood of each defect model point, p i is the coordinate of the i-th point, 0 < i ≤ N, u is the coordinate average of all points within the k-neighborhood, and T is the matrix transpose symbol.

[0108] S542: Perform eigenvalue decomposition on the covariance matrix corresponding to each defect model point to obtain several eigenvalues corresponding to each defect model point and the eigenvectors corresponding to the eigenvalues;

[0109] S543: Obtain the minimum eigenvalue corresponding to each defect model point, and use the eigenvector corresponding to the minimum eigenvalue as the normal vector of each defect model point.

[0110] Specifically, it is possible to perform eigenvalue decomposition on each covariance matrix obtained in step S541 to obtain several eigenvalues and the eigenvectors corresponding to the eigenvalues, sort the several eigenvalues corresponding to each defect point model point in a set order according to the eigenvalue size to obtain the minimum eigenvalue of each defect point model point, and use the eigenvector corresponding to the minimum eigenvalue as the normal vector corresponding to the corresponding defect model point.

[0111] This embodiment presents a paint defect detection method of the present invention. It involves acquiring several sets of calibrated vehicle surface images to obtain the intrinsic parameter matrices of each acquisition device and the first transformation matrix between the acquisition devices, creating an initial virtual scene, and importing virtual acquisition devices and a vehicle model based on the intrinsic parameter matrices of each acquisition device and the first transformation matrix between them. By constructing a vehicle model in the virtual scene, the efficiency of paint defect detection is improved, and the method is not limited to a specific vehicle model, thus improving compatibility. A 3D surface point cloud of the calibrated vehicle is obtained based on prior engineering data. Several 2D image points from any surface image and the corresponding 3D surface point cloud on the calibrated vehicle are selected to construct feature point pairs, obtaining the relative pose between the calibrated vehicle and each acquisition device. The relative pose between each virtual acquisition device and the vehicle model is corrected based on the relative pose between the calibrated vehicle and each acquisition device, resulting in the final virtual scene. Several sets of current 2D vehicle images are acquired, and the coordinates of defect points in each set of 2D images are obtained and mapped to the vehicle model to obtain the coordinates and normals of the defect points. Mapping defect points from 2D images to the 3D model allows for more accurate defect location and improves detection accuracy. It solves the problems of low efficiency, poor accuracy, insufficient compatibility and limited precision of existing technologies for detecting defects in automotive paint.

[0112] On the other hand, the present invention also provides a computer storage medium storing executable program code; the executable program code is used to execute any of the above-mentioned paint surface defect detection methods.

[0113] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-described paint defect detection methods.

[0114] For example, the program code can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the program code in the terminal device.

[0115] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the terminal device may also include input / output devices, network access devices, buses, etc.

[0116] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0117] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. The memory can also be an external storage device of the terminal device, such as a plug-in hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the terminal device. The memory is used to store the program code and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output.

[0118] The computer storage medium and terminal device described above are created based on the paint surface defect detection method described above. Their technical functions and beneficial effects will not be elaborated here. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for detecting paint surface defects, characterized in that, include: Several sets of calibrated vehicle surface images were acquired to obtain the intrinsic parameter matrix of each acquisition device and the first transformation matrix between each acquisition device. Create an initial virtual scene, and import the virtual acquisition devices and car models based on the intrinsic parameter matrices of each acquisition device and the first transformation matrix between each acquisition device; Based on prior engineering data, the 3D surface point cloud of the calibrated vehicle is obtained. Several 2D image points in any surface image and the corresponding 3D surface point cloud on the calibrated vehicle are selected to construct feature point pairs and obtain the relative pose between the calibrated vehicle and each acquisition device. The relative poses between the virtual acquisition devices and the car model are corrected based on the relative poses between the calibrated car and each acquisition device to obtain the final virtual scene. Acquire several sets of current 2D images of the car, obtain the coordinates of the defect points in each set of 2D images, and map them to the car model to obtain the coordinates and normals of the defect points; The steps for obtaining the intrinsic parameter matrix of each acquisition device and the first transformation matrix between the acquisition devices include: Each acquisition device is installed in a set position, and the calibration vehicle is controlled to move in a set direction. Each acquisition device scans the surface of the calibration vehicle at a preset sampling distance to obtain several sets of surface images of the calibration vehicle. Based on the surface images of all calibrated vehicles, several sets of relative poses between the calibrated vehicles and the acquisition devices are obtained. The acquisition devices are then transformed to the same coordinate system to obtain several relative acquisition devices. Image analysis was performed on the surface images of each group of calibrated vehicles to obtain the image feature points in each group of surface images; Three-dimensional reconstruction is performed on the image feature points in the coordinate systems of each relative acquisition device to obtain the first transformation matrix between the coordinate systems of each relative acquisition device and the intrinsic parameter matrix of each relative acquisition device.

2. The method according to claim 1, characterized in that, The steps for creating an initial virtual scene, and importing the virtual acquisition devices and car model based on the intrinsic parameter matrices of each acquisition device and the first transformation matrix between the acquisition devices, include: Import the data acquisition device model and the design model of the calibrated car, which are the same models as those in the actual scene, into the virtual scene to obtain virtual data acquisition device and car models. Configure the parameters of the virtual acquisition devices according to the intrinsic parameter matrix, and obtain the relative poses between the virtual acquisition devices in the virtual scene according to the first transformation matrix; Based on the location of the car in the real scene, place the corresponding car model in the virtual scene.

3. The method according to claim 1, characterized in that, The steps for obtaining the relative pose between the calibrated vehicle and each data acquisition device include: The 3D surface point cloud of the calibrated vehicle was obtained based on prior engineering data; Select several 2D image points and the corresponding 3D surface point cloud from any surface image to construct several feature point pairs; The feature point pairs are reprojected to obtain the second transformation matrix between the coordinate system of the acquisition device and the coordinate system of the calibration vehicle corresponding to the surface image, and the relative pose between the calibration vehicle and each acquisition device is obtained according to the first transformation matrix between each acquisition device.

4. The method according to claim 3, characterized in that, The steps for constructing several feature point pairs by selecting several 2D image points and corresponding 3D surface point clouds from any surface image include: The coordinates of 2D feature points in the surface image are obtained based on the image detection algorithm; The 3D surface point cloud is converted into a triangular mesh to obtain the points in the feature hole region; Fit a plane to the points in the feature hole region and extract the plane contour; The center of a circle is fitted to the planar contour to obtain 3D feature points, and several feature point pairs are constructed based on each 2D feature point and its corresponding 3D feature point.

5. The method according to claim 1, characterized in that, The steps to obtain the coordinates and normal of the defect point include: Construct and train a neural network model that takes a 2D image as input and the coordinates of the defect point as output; Acquire several sets of current 2D images of the car, and input all 2D images into a neural network model to obtain the coordinates of all defect points in each 2D image; Project the coordinates of each defect point onto the car model to obtain the corresponding 3D coordinates, which are the defect model points; Obtain the coordinates of several point clouds that are closest to each defect model point, and calculate the normal of each defect model point.

6. The method according to claim 5, characterized in that, The steps to obtain the normals of each defect model point may include: Set the neighborhood size and obtain the coordinates of all points within the set neighborhood for each defect model point to construct the covariance matrix; Eigenvalue decomposition is performed on the covariance matrix corresponding to each defect model point to obtain several eigenvalues ​​and eigenvectors corresponding to each defect model point. Obtain the minimum eigenvalue corresponding to each defect model point, and use the eigenvector corresponding to the minimum eigenvalue as the normal vector of each defect model point.

7. The method according to claim 6, characterized in that, The covariance matrix is: where C is the covariance matrix, N is the number of points within the k-neighborhood of each defect model point, p i is the coordinate of the i-th point, 0 < i ≤ N, u is the average coordinate of all points within the k-neighborhood, k is the set neighborhood size, and T is the matrix transpose symbol.

8. A computer storage medium, characterized in that, It stores executable program code; the executable program code is used to execute the paint surface defect detection method according to any one of claims 1-7.

9. A terminal device, characterized in that, It includes a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute the paint surface defect detection method according to any one of claims 1-7.

Images