Methods, apparatus, systems and readable storage media for detecting surface defects in vehicles

Abstract

This application relates to a method, apparatus, system, and readable storage medium for detecting surface defects on vehicles. The method includes: detecting the forward distance of a vehicle entering a tunnel-type inspection space; controlling multiple cameras to capture surface images of the vehicle based on the forward distance; detecting two-dimensional defect information of defects in the multiple surface images, the two-dimensional defect information including at least two-dimensional defect location information, defect type information, and defect size information; converting each two-dimensional defect location information into three-dimensional defect location information in the vehicle's corresponding coordinate system; and associating the three-dimensional defect location information, defect type information, and defect size information of each defect to obtain the vehicle surface defect detection result. This method can accurately polish vehicle surface defects.

Description

Technical Field

[0001] This application relates to the field of defect detection technology, and in particular to a method, apparatus, system and readable storage medium for detecting defects on vehicle surfaces. Background Technology

[0002] During vehicle production, imperfections such as dents and particles inevitably occur on the vehicle body surface. The quality of the surface not only affects the vehicle's aesthetics but also its wear resistance and corrosion resistance. Therefore, it is necessary to polish away these surface defects.

[0003] Currently, vehicle surface defect detection technology mostly uses robotic arms carrying camera light sources for inspection. The detection mode is generally "stop-and-go," and there is also a tunnel-type detection method where the vehicle body enters the tunnel along the production line to complete the inspection, which improves inspection efficiency to some extent. After detecting defects on the vehicle surface, a defect grinding operation can be performed, guiding the grinding device to polish and remove the defects.

[0004] However, current vehicle surface defect detection technology is not accurate enough, and a more accurate vehicle surface defect detection technology is needed to detect defects on the vehicle surface. Summary of the Invention

[0005] Therefore, it is necessary to provide an accurate method, apparatus, system, computer equipment, computer-readable storage medium, and computer program product for detecting surface defects of vehicles, in order to address the aforementioned technical problems.

[0006] In a first aspect, this application provides a method for grinding surface defects on a vehicle, applied to a tunnel-type defect detection system. The tunnel-type defect detection system includes a tunnel-type detection space and multiple cameras. The method includes:

[0007] When the vehicle to be inspected enters the tunnel-type inspection space, the distance the vehicle has traveled is detected.

[0008] Based on the forward distance, control multiple cameras to capture surface images of the vehicle to be inspected;

[0009] Two-dimensional defect information is detected in multiple surface images. The two-dimensional defect information includes at least two-dimensional defect location information, defect type information, and defect size information.

[0010] For each two-dimensional defect location information, the two-dimensional defect location information is converted into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle.

[0011] By associating the three-dimensional defect location information, defect type information, and defect size information of each defect, the vehicle surface defect detection results are obtained.

[0012] Secondly, this application also provides a vehicle surface defect grinding device, comprising:

[0013] The distance detection module is used to detect the forward distance of the vehicle under test when the vehicle under test enters the tunnel-type detection space;

[0014] The image acquisition module is used to control multiple cameras to capture surface images of the vehicle under inspection based on the forward distance.

[0015] The two-dimensional defect detection module is used to detect two-dimensional defect information of defects in multiple surface images. The two-dimensional defect information includes at least two-dimensional defect location information, defect type information and defect size information.

[0016] The 3D defect generation module is used to convert the 2D defect location information into 3D defect location information in the vehicle coordinate system corresponding to the vehicle for each 2D defect location information.

[0017] The defect result generation module is used to associate the three-dimensional defect location information, defect type information and defect size information corresponding to each two-dimensional defect location information to obtain the vehicle surface defect detection results.

[0018] Thirdly, this application also provides a tunnel-type defect detection system, which includes a controller, an encoder, a photoelectric sensor, a tunnel-type detection space, and multiple cameras;

[0019] Photoelectric sensors are used to detect whether a vehicle under inspection has entered the tunnel-type inspection space;

[0020] Multiple cameras are used to capture surface images of the vehicle to be inspected;

[0021] An encoder is used to detect the forward distance of a vehicle being inspected.

[0022] The controller is used to control the encoder to detect the forward distance of the vehicle under test when the photoelectric sensor detects that the vehicle under test has entered the tunnel-type detection space; based on the forward distance, it controls multiple cameras to capture surface images of the vehicle under test; it detects two-dimensional defect information of defects in multiple surface images, the two-dimensional defect information including at least two-dimensional defect location information, defect type information and defect size information; for each two-dimensional defect location information, it converts the two-dimensional defect location information into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle; and it associates the three-dimensional defect location information, defect type information and defect size information corresponding to each two-dimensional defect location information to obtain the vehicle surface defect detection result.

[0023] Fourthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0024] When the vehicle to be inspected enters the tunnel-type inspection space, the distance the vehicle has traveled is detected.

[0025] Based on the forward distance, control multiple cameras to capture surface images of the vehicle to be inspected;

[0026] Two-dimensional defect information is detected in multiple surface images. The two-dimensional defect information includes at least two-dimensional defect location information, defect type information, and defect size information.

[0027] For each two-dimensional defect location information, the two-dimensional defect location information is converted into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle.

[0028] By associating the three-dimensional defect location information, defect type information, and defect size information of each defect, the vehicle surface defect detection results are obtained.

[0029] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0030] When the vehicle to be inspected enters the tunnel-type inspection space, the distance the vehicle has traveled is detected.

[0031] Based on the forward distance, control multiple cameras to capture surface images of the vehicle to be inspected;

[0032] Two-dimensional defect information is detected in multiple surface images. The two-dimensional defect information includes at least two-dimensional defect location information, defect type information, and defect size information.

[0033] For each two-dimensional defect location information, the two-dimensional defect location information is converted into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle.

[0034] By associating the three-dimensional defect location information, defect type information, and defect size information of each defect, the vehicle surface defect detection results are obtained.

[0035] Sixthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0036] When the vehicle to be inspected enters the tunnel-type inspection space, the distance the vehicle has traveled is detected.

[0037] Based on the forward distance, control multiple cameras to capture surface images of the vehicle to be inspected;

[0038] Two-dimensional defect information is detected in multiple surface images. The two-dimensional defect information includes at least two-dimensional defect location information, defect type information, and defect size information.

[0039] For each two-dimensional defect location information, the two-dimensional defect location information is converted into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle.

[0040] By associating the three-dimensional defect location information, defect type information, and defect size information of each defect, the vehicle surface defect detection results are obtained.

[0041] The aforementioned vehicle surface defect detection method, apparatus, system, computer equipment, computer-readable storage medium, and computer program product, when the vehicle to be inspected enters a tunnel-type inspection space, detects the forward distance of the vehicle; based on the forward distance, controls multiple cameras to capture surface images of the vehicle; detects two-dimensional defect information of defects in the multiple surface images, the two-dimensional defect information including at least two-dimensional defect location information, defect type information, and defect size information; for each two-dimensional defect location information, converts the two-dimensional defect location information into three-dimensional defect location information in the vehicle's corresponding coordinate system; and associates the three-dimensional defect location information, defect type information, and defect size information corresponding to each two-dimensional defect location information to obtain the vehicle surface defect detection result. Throughout the process, when the vehicle to be inspected enters the tunnel-type inspection space, controlling multiple cameras to capture surface images of the vehicle based on the forward distance of the vehicle, and then performing defect detection on surface images captured by multiple cameras over multiple time periods, results in higher accuracy. This leads to more accurate vehicle surface defect detection results after associating the three-dimensional defect location information, defect type information, and defect size information corresponding to each two-dimensional defect location information. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a diagram illustrating the application environment of a vehicle surface defect detection method in one embodiment.

[0044] Figure 2 This is a flowchart illustrating a vehicle surface defect detection method in one embodiment;

[0045] Figure 3 This is a flowchart illustrating a method for polishing vehicle surface defects in another embodiment;

[0046] Figure 4 This is a schematic diagram illustrating the uniformly distributed small calibration plates affixed to the body of a standard prototype vehicle in a specific application embodiment.

[0047] Figure 5 This is a schematic diagram illustrating the visualization results of external parameter calibration in a specific application embodiment;

[0048] Figure 6 This is a schematic diagram illustrating how a ray is emitted from a simulation camera, starting with the location information of a two-dimensional defect simulation, in a specific application embodiment.

[0049] Figure 7 The placement type in one specific application embodiment includes a schematic diagram of placing the global camera directly above the polishing station;

[0050] Figure 8 This is a schematic diagram illustrating the placement of each distributed camera within the vicinity of its corresponding polishing robot in a specific application embodiment.

[0051] Figure 9 This is a structural block diagram of a vehicle surface defect grinding device in one embodiment;

[0052] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0054] Before grinding defects on the vehicle surface, it is often necessary to locate and detect these defects. Generally, vehicle surface defect detection technology often uses a robotic arm carrying a camera light source for detection, and the detection mode is usually "stop-and-go". However, the detection time is long, affecting the production line cycle. There is also a tunnel-type detection method, in which the vehicle body enters the tunnel with the production line to complete the detection. This improves the detection efficiency to some extent, but the calibration process is more cumbersome, the defect location accuracy is not high, and the compatibility with new models is poor.

[0055] Therefore, this application addresses the aforementioned accuracy issue in vehicle surface defect detection by providing an accurate vehicle surface defect detection method. When the vehicle to be inspected enters a tunnel-type inspection space, multiple cameras are controlled to capture surface images of the vehicle based on the vehicle's forward distance. Defect detection is then performed on surface images captured by multiple cameras over multiple time periods, resulting in higher accuracy. This makes the generated vehicle surface defect detection results more accurate after associating the three-dimensional defect location information, defect type information, and defect size information corresponding to each two-dimensional defect location information.

[0056] The vehicle surface defect detection method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with tunnel-type defect detection system 104 via a network. The tunnel-type defect detection system includes a controller 106, a tunnel-type detection space 108, and multiple cameras 110. The multiple cameras 110 communicate with the controller 106 via the network. A data storage system can store the data that the controller 106 needs to process. The data storage system can be integrated onto the controller 106 or placed in the cloud or on another network server. The controller 106 can be a PLC (Programmable Logic Controller) module, which is responsible for the overall hardware control and sends signals to control the hardware operation.

[0057] The user operates on terminal 102, sending a vehicle surface defect detection request to controller 106 in tunnel-type defect detection system 104. When the vehicle to be inspected 112 enters tunnel-type detection space 108, controller 106 detects the forward distance of the vehicle to be inspected 112; based on the forward distance, it controls multiple cameras 110 to capture surface images of the vehicle to be inspected 112; it detects two-dimensional defect information of defects in multiple surface images, the two-dimensional defect information including at least two-dimensional defect location information, defect type information, and defect size information; for each two-dimensional defect location information, it converts the two-dimensional defect location information into three-dimensional defect location information in the vehicle's corresponding coordinate system; it associates the three-dimensional defect location information, defect type information, and defect size information corresponding to each two-dimensional defect location information to obtain the vehicle surface defect detection result. Furthermore, controller 106 can display the vehicle surface defect detection result to terminal 102. In addition, controller 106 can also control a polishing robot to polish the vehicle surface defects based on the vehicle surface defect detection result.

[0058] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, projectors, etc. Additionally, it can be the display screen of the controller 106. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0059] In one exemplary embodiment, such as Figure 2 As shown, a method for detecting surface defects on a vehicle is provided, which can be applied to... Figure 1The tunnel-type defect detection system 104 is used as an example for illustration. The tunnel-type defect detection system includes a controller, a tunnel-type detection space, and multiple cameras, including the following S100~S500. Among them:

[0060] S100 measures the distance traveled by a vehicle when it enters a tunnel-type inspection space.

[0061] Specifically, this application includes a photoelectric sensor used to detect whether the vehicle has entered the tunnel-type inspection space. When the vehicle enters the tunnel-type inspection space, the vehicle paint defect detection program is initiated.

[0062] In addition, this application includes an encoder used to detect the forward distance of the vehicle under inspection in order to determine its position. Existing tunnel-type defect detection technologies mostly use encoders to monitor the displacement of the vehicle body, thereby updating the relative position of the vehicle body and the camera, which is not accurate enough. This method combines the calibration of the encoder and the camera to obtain the relative position of the camera and the vehicle body at multiple moments, resulting in higher accuracy.

[0063] The S200 controls multiple cameras to capture surface images of the vehicle under inspection based on the forward distance.

[0064] The camera can be replaced by other equipment used for shooting. There is more than one camera, and the number of cameras varies according to actual needs.

[0065] Specifically, whenever the vehicle to be detected moves a specified distance, the controller will control multiple cameras to take a picture. For example, if the specified distance is 10cm, when the vehicle moves 10cm, the controller will control multiple cameras to take a picture. At this time, the forward distance can be recalculated. When the vehicle moves another 10cm, the controller will control multiple cameras to take a picture again. In simple terms, when the forward distance represents the vehicle moving any multiple of the specified distance, the controller will control multiple cameras to take a picture of the surface of the vehicle. Thus, the surface images of the vehicle taken by multiple cameras at different time periods can be obtained.

[0066] In an exemplary embodiment, the surface image of the vehicle to be inspected is a structured light image captured by structured light technology. Structured light technology uses a pre-designed pattern with a special structure (such as discrete light spots, striped light, coded structured light, etc.), and then projects the pattern onto the surface of a three-dimensional object. Another camera is used to observe the distortion of the image on the three-dimensional physical surface. After the pattern is projected onto the surface of the three-dimensional object, the three-dimensional object surface image captured by the camera device is the structured light image carrying the projected pattern.

[0067] In other words, this application also includes a structured light source. When the structured light source illuminates the surface of the vehicle under inspection, it generates a structured light band. At this time, multiple cameras capture images of the vehicle's surface, which become the structured light image. It should be explained that the reason for including a structured light source is that when the structured light source illuminates the surface of the vehicle under inspection, the defective areas will exhibit more pronounced changes compared to the surrounding normal paint surface, effectively enhancing the characteristics of defects such as protrusions, depressions, particles, and pinholes. Generally, structured light sources include, but are not limited to, binary striped LED (Light Emitting Diode) sources. When the structured light source is a binary striped LED source, illuminating the surface of the vehicle under inspection with it will generate a binary-like striped light band.

[0068] In one exemplary embodiment, multiple cameras are mounted around the vehicle. The mounting of the cameras needs to ensure a continuous field of view and cover the entire cross-section of the vehicle body illuminated by the structured light source. Simultaneously, suitable camera lenses are selected to cover a wider detection area, and the spatial resolution of the cameras is maximized to detect even smaller defects.

[0069] In another exemplary embodiment, the camera and structured light source can be mounted on a fixed bracket or held by a robotic arm, allowing the camera to be relatively translated to the vehicle body. The camera then captures real-time images of the surface reflected from the vehicle body, which is moving relative to the detection system. Alternatively, a combination of bracket mounting and robotic arm holding methods can be used. For example, fixed brackets can be used on the left and right sides of the vehicle body where the height is relatively fixed, while a more flexible robotic arm holding method can be used on areas where the height changes significantly with the vehicle model, such as the roof, hood, and rear, allowing the detection points to be adjusted in real-time as the vehicle body moves. Alternatively, a bracket mounting method can be used to install the camera and structured light source, which is more suitable for scenarios where the vehicle model size changes little on the production line.

[0070] S300 detects two-dimensional defect information of defects in multiple surface images. The two-dimensional defect information includes at least two-dimensional defect location information, defect type information, and defect size information.

[0071] Specifically, the detection of two-dimensional defect information in multiple surface images is mainly achieved through deep learning algorithms for object detection. That is, defect sample data is collected in advance, labeled, and trained into an inference model. Then, the trained inference model accurately detects the two-dimensional defect information of the simulated vehicle from each surface image. Since the surface images are collected over multiple time periods, the amount of two-dimensional defect information obtained is greater and more comprehensive.

[0072] Furthermore, since the surface image is a structured light image with enhanced defect features, the surface image can be processed to determine the defect type and size information from the structured light image. For example, it can be determined whether the defect is a convex or concave shape, and whether the size of the defect is greater than a preset safety threshold. The two-dimensional defect location information, defect type information, and defect size information of the defect can be combined to generate the two-dimensional defect information of the defect.

[0073] S500 converts each two-dimensional defect location information into a three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle.

[0074] Specifically, the two-dimensional defect location information is the two-dimensional defect location information in the image coordinate system. It is also necessary to transform the two-dimensional defect location information into the three-dimensional vehicle coordinate system to obtain the three-dimensional defect location information in the vehicle coordinate system.

[0075] It should be noted that in real-world environments, due to the scale uncertainty of monocular cameras, it is impossible to directly deduce three-dimensional defect location information from two-dimensional defect location information. Therefore, this application uses ray tracing to convert two-dimensional defect location information into three-dimensional defect location information in the vehicle's coordinate system.

[0076] S500 associates the three-dimensional defect location information, defect type information and defect size information of each defect to obtain the vehicle surface defect detection results.

[0077] Specifically, the three-dimensional defect location information, defect type information and defect size information of each defect are associated to obtain the detection result of the defect. Then, based on the detection results of all defects, the detection result of vehicle surface defects is obtained.

[0078] Furthermore, the vehicle surface defect detection results are analyzed. When the vehicle surface defect detection results indicate that defect grinding is required, the grinding robot is controlled to perform defect grinding based on the three-dimensional defect location information in the vehicle surface defect detection results.

[0079] In the aforementioned vehicle surface defect detection method, when the vehicle to be inspected enters the tunnel-type inspection space, the forward distance of the vehicle is detected. Based on the forward distance, multiple cameras are controlled to capture surface images of the vehicle. Two-dimensional defect information of defects in the multiple surface images is detected, and this two-dimensional defect information includes at least two-dimensional defect location information, defect type information, and defect size information. For each two-dimensional defect location information, it is converted into three-dimensional defect location information in the vehicle's corresponding coordinate system. The three-dimensional defect location information, defect type information, and defect size information corresponding to each two-dimensional defect location information are associated to obtain the vehicle surface defect detection result. Throughout this process, when the vehicle to be inspected enters the tunnel-type inspection space, multiple cameras are controlled to capture surface images of the vehicle based on its forward distance. Defect detection is then performed on surface images captured by multiple cameras over multiple time periods, resulting in higher accuracy. Furthermore, the generated vehicle surface defect detection result is more accurate after associating the three-dimensional defect location information, defect type information, and defect size information corresponding to each two-dimensional defect location information.

[0080] In one exemplary embodiment, such as Figure 3 As shown, prior to S400, the method also includes:

[0081] S360 acquires the two-dimensional feature hole position information of the feature hole in the surface image captured by the camera for each camera in each time period, as well as the three-dimensional feature hole position information of the feature hole in the vehicle coordinate system corresponding to the vehicle to be detected. Based on the three-dimensional feature hole position information and the two-dimensional feature hole position information, it generates the first pose of the camera coordinate system relative to the vehicle coordinate system.

[0082] S370 constructs a vehicle operation simulation environment based on the preset parameter calibration information of all first poses and all cameras in different time periods.

[0083] Among them, the feature holes are typical holes on the surface of the vehicle to be inspected. Both three-dimensional and two-dimensional feature hole position information can be three-dimensional coordinate information or other information used to characterize the position.

[0084] Specifically, feature holes are distributed on the surface of the vehicle to be inspected. Therefore, the surface images captured by the camera contain feature hole information. For each camera in each time period, the two-dimensional feature hole position information is obtained from the surface images captured by the camera. In addition, the three-dimensional feature hole position information in the vehicle coordinate system corresponding to the vehicle to be inspected also needs to be obtained.

[0085] Based on the 3D and 2D feature hole position information of the same feature hole, the first pose of the camera coordinate system relative to the vehicle coordinate system is generated. In this process, the first pose of the camera coordinate system relative to the vehicle coordinate system can be obtained using the PnP (Perspective-n-Point) method. It should be understood that PnP is a geometric problem in computer vision, primarily used to estimate camera pose. Specifically, given a set of 3D (3-Dimensions) points and their corresponding 2D (2-Dimensions) point coordinates in the image, the PnP algorithm can calculate the relative relationship between the camera coordinate system and the 3D point coordinate system (vehicle coordinate system) by combining the camera's parameter calibration information.

[0086] Finally, a vehicle operation simulation environment is constructed based on the preset parameter calibration information of all first poses and all cameras in different time periods.

[0087] In an exemplary embodiment, obtaining the three-dimensional feature hole position information in the vehicle coordinate system corresponding to the vehicle to be detected includes: obtaining the hole center position information of the feature hole in the vehicle coordinate system as the three-dimensional feature hole position information of the feature hole; more specifically, obtaining the vehicle body model design and assembly information, and then directly obtaining the hole center position information of the feature hole based on the vehicle body model design and assembly information, and then using the hole center position information of the feature hole as the three-dimensional feature hole position information of the feature hole.

[0088] In an exemplary embodiment, obtaining the two-dimensional feature hole position information of a feature hole in a surface image captured by a camera includes: extracting the two-dimensional feature hole position information of the feature hole in the surface image using an ellipse fitting algorithm. In practical applications, the ellipse fitting algorithm is implemented using the least squares method or matrix decomposition method.

[0089] In an exemplary embodiment, constructing a vehicle operation simulation environment includes: importing simulation cameras corresponding to multiple cameras and simulation vehicles corresponding to vehicles to be tested into the vehicle operation simulation environment, wherein the parameter calibration information of multiple simulation cameras, the relative positions between multiple simulation cameras, and the relative positions between multiple simulation cameras and simulation vehicles to be tested need to be consistent with the actual environment.

[0090] Specifically, firstly, multiple simulated cameras are imported into the initial vehicle operation simulation environment. When importing these cameras, parameters such as focal length, image width, image height, and chip size need to be set to match those of the cameras in the actual environment to ensure consistent shooting effects between the virtual and real cameras. Then, the parameter calibration information of the simulated cameras can be updated based on the preset parameter calibration information of each camera in the actual environment.

[0091] Secondly, when importing the simulated vehicle corresponding to the vehicle to be tested into the vehicle running simulation environment, it is necessary to determine the relative position between the simulated vehicle to be tested and multiple simulated cameras. At this time, the first pose of the camera coordinate system relative to the vehicle coordinate system can be used as the pose of the simulated camera coordinate system relative to the simulated vehicle coordinate system corresponding to the simulated vehicle to be tested, so as to ensure that the positional relationship between the vehicle and the camera in the virtual and real environments can be deployed consistently.

[0092] In an exemplary embodiment, the process of generating the first pose of the camera coordinate system relative to the vehicle coordinate system based on the three-dimensional feature hole position information and the two-dimensional feature hole position information further includes: obtaining the camera's internal calibration parameters; and generating the first pose of the camera coordinate system relative to the vehicle coordinate system based on the three-dimensional feature hole position information, the two-dimensional feature hole position information, and the internal calibration parameters.

[0093] Specifically, based on the three-dimensional feature hole position information, the two-dimensional feature hole position information, and the internal calibration parameters, the first pose of the camera coordinate system relative to the vehicle coordinate system is generated, including: based on the internal calibration parameters, mapping the two-dimensional feature hole position information from the image coordinate system to the camera coordinate system to obtain the intermediate feature hole position information; and based on the intermediate feature hole position information and the three-dimensional feature hole position information, generating the first pose of the camera coordinate system relative to the vehicle coordinate system.

[0094] Specifically, the first pose of the camera coordinate system relative to the vehicle coordinate system is generated by the PnP algorithm. The PnP algorithm can be formalized as follows: given the coordinates of n three-dimensional points and their two-dimensional coordinates on the image, as well as the intrinsic parameter matrix of the camera, solve for the first pose of the camera coordinate system relative to the vehicle coordinate system.

[0095] Therefore, based on the three-dimensional feature hole position information, the two-dimensional feature hole position information, and the internal calibration parameters, the first pose of the camera coordinate system relative to the vehicle coordinate system can be generated together, so as to update the first pose to the vehicle operation simulation environment and achieve consistent deployment of the virtual and real environments.

[0096] In detail, the expression for the PnP algorithm includes: Where p is the position information of the two-dimensional feature hole on the surface image, P is the position information of the three-dimensional feature hole corresponding to the position information of the two-dimensional feature hole, K is the internal calibration parameter of the camera, R is the rotation matrix, t is the translation vector, [R | t] is the first pose of the camera coordinate system relative to the vehicle coordinate system, and s is a scaling factor.

[0097] As can be seen, if the first pose [R|t] is required, the position information of the two-dimensional feature aperture can be mapped from the image coordinate system to the camera coordinate system based on the camera's internal calibration parameters to obtain the position information of the intermediate feature aperture. Since the position information of the intermediate feature aperture is the position information in the camera coordinate system, while the position information of the three-dimensional feature aperture is the position information in the vehicle coordinate system, the first pose of the camera corresponding to the camera coordinate system relative to the vehicle coordinate system can be generated based on the position information of the intermediate feature aperture and the position information of the three-dimensional feature aperture.

[0098] In one exemplary embodiment, the vehicle operation simulation environment is established based on the first pose of all cameras and preset parameter calibration information over all time periods. Therefore, there may be situations where multiple simulation cameras or the same simulation camera captures the same defect data at different times. That is, there may be multiple observations of the same defect. The two-dimensional defect position information of the defect observed multiple times can be mapped onto the vehicle coordinate system to obtain multiple three-dimensional defect position information. An error tolerance value is set. If the Euclidean distance between any two three-dimensional defect position information is less than the error tolerance value, it is determined that the defects corresponding to these two three-dimensional defect position information are the same defect, and one of the duplicate information is removed. If the Euclidean distance between any two three-dimensional defect position information is greater than the error tolerance value, it is considered that the defects corresponding to these two three-dimensional defect position information are independent defects. In some extreme cases, if the multiple three-dimensional defect position information of the same defect exceeds the error tolerance value, the calibration accuracy in the vehicle surface defect detection process is considered to be poor.

[0099] At this point, S400 also includes: for each two-dimensional defect location information, based on the constructed vehicle operation simulation environment, converting the two-dimensional defect location information into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle.

[0100] In other words, in the constructed vehicle operation simulation environment, the ray tracing method is used to convert the two-dimensional defect location information into the three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle.

[0101] In the above embodiments, by constructing a vehicle operation simulation environment consistent with the real environment, preliminary work can be done for performing virtual-real mapping operations in the vehicle operation simulation environment, that is, for obtaining three-dimensional defect location information mapped from two-dimensional defect location information.

[0102] In an exemplary embodiment, the preset parameter calibration information for all cameras includes the internal calibration parameters of each camera and the external calibration parameters between cameras. Before constructing the vehicle operation simulation environment based on the preset parameter calibration information of all first poses and all cameras within different time periods, the following is also included:

[0103] When the calibration image is pasted onto the vehicle surface, for each camera, multiple corner points of the calibration image are detected from surface images over multiple time periods, and the internal calibration parameters of the camera are detected based on the multiple corner points; based on the surface images of all cameras in each time period, the external calibration parameters between cameras in each time period are detected.

[0104] The calibration image is actually a calibration board, which can be a QR code image, etc.

[0105] Specifically, before constructing the vehicle operation simulation environment based on the first pose and preset parameter calibration information of all cameras in all time periods, it is also necessary to obtain the preset parameter calibration information of all cameras in all time periods.

[0106] First, such as Figure 4 As shown, small calibration plates (such as AprilTag) are evenly distributed on the body of the standard prototype vehicle. Each calibration plate has its own identifier, ensuring that each calibration plate detected by the camera has a unique identifier. After the calibration plates are applied, the vehicle is sent into the tunnel-type defect detection system, where the vehicle is moved and calibration data is collected multiple times at equal intervals according to the standard procedure.

[0107] After image acquisition is completed, multiple corner information of the calibration image is detected from the surface images captured by each camera in multiple time periods. For each camera, the internal calibration parameters of each camera are calculated based on the corner information in multiple time periods and the known size information of the calibration image. In addition, the external calibration parameters between cameras in each time period can be detected based on the surface images of all cameras in each time period.

[0108] The above steps can be implemented using the open-source software colmap. The visualization results of external parameter calibration using the methods of colmap are as follows: Figure 5 As shown, Colmap is an open-source multi-view stereo vision software widely used in computer vision and 3D reconstruction. Specifically, it can recover the geometry of a 3D scene from a set of 2D images and estimate camera pose and intrinsic parameters. Furthermore, it can also be implemented using other calibration software; the implementation method is not limited.

[0109] In other embodiments, not only can AprilTag QR codes and the colmap method be used to calibrate the camera, but other types of QR codes (such as Stag, ArUco, etc.) and other open-source 3D reconstruction libraries (such as OpenMVG) can also be used to complete the calibration. Even if additional cameras are needed later, the above process can be repeated.

[0110] It is worth noting that the above process is performed while the vehicle is moving and the camera remains stationary. This can be understood as the vehicle being stationary and the camera being moving. When the camera collects calibration data at regular intervals, the above method will calculate the internal calibration parameters of each camera and the external calibration parameters between the cameras at different times. For example, if there are a total of 9 cameras and 15 sets of calibration data are collected, the internal calibration parameters of the 9 cameras and the external calibration parameters of the 9x15 cameras will be calibrated.

[0111] Finally, based on the first pose of all cameras in all time periods in the real environment, the internal calibration parameters of each camera, and the external calibration parameters between cameras, the relative positions between cameras and vehicles, the individual positions of cameras, and the relative positions between cameras in the vehicle operation simulation environment in different time periods are updated to construct a vehicle operation simulation environment consistent with the real environment.

[0112] In an exemplary embodiment, if a robotic arm is used to hold the camera, hand-eye calibration (with the eye on the hand) is also required. Specifically, this involves acquiring images of the calibration board taken by the camera and saving the pose of the robotic arm in the corresponding teach pendant. The transformation relationship between the camera coordinate system and the robotic arm end effector coordinate system, as well as the base coordinate system, is calculated to obtain the extrinsic parameters between the camera coordinate system and the reference camera coordinate system under different robotic arm postures. In particular, the extrinsic parameter relationship between the camera and the reference camera coordinate system can be calculated using the calibration results with the eye on the hand and the current posture of the robotic arm at different times.

[0113] In the above embodiments, the camera parameter calibration process is simple and easy to use. It only requires a certain number of calibration images to be affixed to the vehicle body and passing through a tunnel once. It is equally applicable when the position and number of cameras change.

[0114] In an exemplary embodiment, the vehicle operation simulation environment includes a simulated vehicle to be detected corresponding to the vehicle to be detected and a simulated camera corresponding to each camera; for each two-dimensional defect location information, the two-dimensional defect location information is converted into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle, including:

[0115] In the vehicle operation simulation environment, for each two-dimensional defect location information, a ray is emitted from the two-dimensional defect location information to the simulation camera corresponding to the two-dimensional defect location information to obtain the ray detection result; when the ray detection result indicates that the ray collides with the simulated vehicle to be detected, the collision point location information between the ray and the simulated vehicle to be detected is obtained, and the collision point location information is used as the three-dimensional defect location information of the vehicle coordinate system corresponding to the vehicle.

[0116] Specifically, such as Figure 6As shown, within multiple time periods, for a two-dimensional defect location within the i-th time period, ray tracing is used to establish a ray from the two-dimensional defect location to the corresponding simulated camera, starting from the two-dimensional defect location. The ray is then used in a vehicle simulation environment to determine whether the ray collides with the simulated vehicle, yielding a ray detection result. This result includes whether the ray collides with objects in the simulation environment. In practical applications, establishing a ray from the two-dimensional defect location to the simulated camera means establishing a ray from the optical center of the two-dimensional defect location to the simulated camera.

[0117] When the ray inspection result indicates that the ray collides with the simulated vehicle under test, the collision point between the ray and the simulated vehicle is obtained, and the collision point location information is used as the three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle. Similarly, the above-mentioned ray tracing method can also be used to obtain the three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle in the j-th or k-th time period. Furthermore, the three-dimensional defect location information in the vehicle coordinate system can be visualized in the vehicle operation simulation environment. In addition, the normal information of the defect can also be obtained through the ray tracing method.

[0118] Among them, the simulated camera corresponding to the two-dimensional defect location information refers to the simulated camera that matches the camera that captured the surface image containing the two-dimensional defect location information. To determine the simulated camera corresponding to the two-dimensional defect location information, it is first necessary to determine the surface image containing the two-dimensional defect location information, then determine the actual camera that captured the surface image, and finally determine the simulated camera that corresponds to the actual camera.

[0119] In the above embodiments, by taking the two-dimensional defect simulation location information as the starting point, ray is emitted towards the optical center of the simulation camera, and the location information of the collision point between the ray and the simulated vehicle to be detected is obtained, thereby accurately realizing the coordinate system transformation from two-dimensional defect location information to three-dimensional defect location information.

[0120] In an exemplary embodiment, after associating the three-dimensional defect location information, defect type information, and defect size information of each defect to obtain the vehicle surface defect detection result, the method further includes:

[0121] The three-dimensional defect location information is mapped from the vehicle coordinate system to the preset grinding robot coordinate system to obtain the grinding defect location information; based on the grinding defect location information, the grinding robot is controlled to perform the grinding operation of the vehicle surface defects.

[0122] Specifically, the three-dimensional defect location information is the location information in the vehicle coordinate system. When the grinding robot wants to grind the defect, it needs to obtain the location information in the base coordinate system where the grinding robot is located. Therefore, it is necessary to map the three-dimensional defect location information from the vehicle coordinate system to the grinding robot coordinate system to obtain the grinding defect location information, so as to control the grinding robot to perform the vehicle surface defect grinding operation based on the grinding defect location information, that is, to control the grinding robot to perform the vehicle surface defect grinding operation at the grinding defect location of the vehicle to be inspected.

[0123] In an exemplary embodiment, when the grinding robot is performing actual grinding, if the three-dimensional defect position information error or coordinate transformation error of the defect is large, resulting in poor positioning accuracy and thus poor grinding effect, it is advisable to equip the grinding robot with an additional precision positioning device at the end of the grinding robot. Using the above-mentioned defect positioning result as the initial value, the precision positioning device is guided to perform secondary precision positioning, thereby achieving precision-guided grinding of the defect.

[0124] In the above embodiments, by performing a virtual-real mapping operation in the vehicle operation simulation environment to obtain accurate three-dimensional defect location information, the defect grinding process is more accurate.

[0125] In an exemplary embodiment, the vehicle operation simulation environment includes a simulated vehicle to be inspected corresponding to the vehicle to be inspected; mapping the three-dimensional defect location information from the vehicle coordinate system to a preset grinding robot coordinate system includes:

[0126] In the case of using a grinding auxiliary camera to assist in the grinding operation of vehicle surface defects, the placement type of the grinding auxiliary camera and the simulated vehicle point cloud information of the vehicle to be tested in the vehicle operation simulation environment are obtained; based on the placement type, the actual vehicle point cloud information captured by the grinding auxiliary camera and the second pose between the grinding auxiliary camera coordinate system and the grinding robot coordinate system are obtained; based on the point cloud registration result generated by the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second pose, the three-dimensional defect location information is mapped from the vehicle coordinate system to the grinding robot coordinate system.

[0127] Specifically, multiple polishing robots can be set up, and the number of polishing robots is determined according to actual needs. Generally, considering the polishing cycle time, four polishing robots are usually set up, distributed around the vehicle, each responsible for a quarter of the vehicle body area.

[0128] The placement types of polishing auxiliary cameras include, but are not limited to: global placement and distributed placement. Global placement refers to hanging a polishing auxiliary camera directly above the polishing station, while distributed placement refers to placing a polishing auxiliary camera near each polishing robot.

[0129] Because the placement types of the grinding assistance cameras differ, their positions also differ, resulting in different actual vehicle point cloud information captured by the cameras and different second poses between the grinding assistance camera coordinate system and the grinding robot coordinate system. In other words, it is necessary to obtain the actual vehicle point cloud information captured by the grinding assistance cameras under different placement types, as well as the second pose between the grinding assistance camera coordinate system and the grinding robot coordinate system. The second pose between the grinding assistance camera coordinate system and the grinding robot coordinate system is obtained through hand-eye calibration technology. Furthermore, it is also necessary to obtain the simulated vehicle point cloud information of the vehicle to be tested in the vehicle operation simulation environment.

[0130] Furthermore, based on the point cloud registration results generated from the actual vehicle point cloud information and the simulated vehicle point cloud information, as well as the second pose, the target defect location information is mapped from the vehicle coordinate system to the grinding robot coordinate system to obtain the grinding defect location information.

[0131] In other words, based on the actual vehicle point cloud information and the simulated vehicle point cloud information, the point cloud registration result between the actual vehicle point cloud information in the actual environment and the simulated vehicle point cloud information in the simulated environment is obtained during the polishing process. Among them, the point cloud registration technology is the process of finding the mapping relationship between different point clouds from different perspectives and using a certain algorithm to transform different point clouds of the same target scene into the same coordinate system to form a more complete point cloud.

[0132] Finally, based on the point cloud registration results of the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second pose obtained based on the hand-eye calibration results of the grinding robot and the grinding auxiliary camera, the three-dimensional defect simulation position information is mapped from the simulated vehicle coordinate system to the grinding robot coordinate system.

[0133] Furthermore, based on the point cloud registration results of the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second pose obtained based on the hand-eye calibration results of the grinding robot and the grinding auxiliary camera, the three-dimensional defect location information is mapped from the vehicle coordinate system to the grinding robot coordinate system. This includes: generating point cloud registration results based on the actual vehicle point cloud information and the simulated vehicle point cloud information; mapping the three-dimensional defect location information from the vehicle coordinate system to the grinding auxiliary camera coordinate system based on the point cloud registration results; and mapping the three-dimensional defect location information from the grinding auxiliary camera coordinate system to the grinding robot coordinate system based on the second pose to obtain the grinding defect location information.

[0134] In the above embodiments, by using different placement types of the grinding auxiliary camera, the actual vehicle point cloud information and the second pose between the grinding auxiliary camera coordinate system and the grinding robot coordinate system under different placement conditions are obtained. Then, based on the point cloud registration results of the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second pose obtained based on the hand-eye calibration results of the grinding robot and the grinding auxiliary camera, the three-dimensional defect location information is accurately mapped from the grinding auxiliary camera coordinate system to the grinding robot coordinate system to obtain the grinding defect location information.

[0135] In an exemplary embodiment, the polishing auxiliary camera includes a global camera or multiple distributed cameras. The polishing auxiliary camera coordinate system includes a global coordinate system corresponding to the global camera and a distributed coordinate system corresponding to each distributed camera. The placement type includes placing the global camera directly above the polishing station or placing each distributed camera within the vicinity of its corresponding polishing robot. Based on the placement type, the actual vehicle point cloud information captured by the polishing auxiliary camera and the second pose between the polishing auxiliary camera coordinate system and the polishing robot coordinate system are obtained, including:

[0136] When the placement type includes placing the global camera directly above the polishing station, the actual vehicle point cloud information captured by the global camera and the second pose between the global coordinate system and the polishing robot coordinate system are obtained; when the placement type includes placing each distributed camera within the vicinity of the corresponding polishing robot, for each distributed camera, the actual vehicle point cloud information captured by the distributed camera and the second pose between the distributed coordinate system and the polishing robot coordinate system are obtained.

[0137] Specifically, when the polishing auxiliary camera includes a global camera, the placement type is to place the global camera directly above the polishing station; when the polishing auxiliary camera includes multiple distributed cameras, the placement type is to place each distributed camera within the vicinity of the corresponding polishing robot.

[0138] When the placement type includes placing the global camera directly above the polishing station, the global camera captures point cloud information of the top of the vehicle body by taking a top-down view, and uses this as the actual vehicle point cloud information. Then, multiple polishing robots are calibrated with the global camera using hand-eye calibration. Based on the hand-eye calibration results, the second pose between the global coordinate system corresponding to the global camera and the coordinate system of the polishing robot is obtained. For example... Figure 7As shown, taking a global camera including a 3D camera as an example, there are three robotic arms of a grinding robot: 1, 2, 3, and 4. During the process of mapping the three-dimensional defect location information from the vehicle coordinate system to the grinding robot coordinate system, the three-dimensional defect simulation location information in the vehicle coordinate system + the point cloud configuration result of the actual vehicle point cloud information captured by the 3D camera and the simulated vehicle point cloud information + the second pose between the 3D camera coordinate system and the grinding robot 1 / 2 / 3 / 4 coordinate system = the grinding defect location information in the grinding robot 1 / 2 / 3 / 4 coordinate system.

[0139] In cases where the placement type includes placing each distributed camera within the vicinity of its corresponding polishing robot, one distributed camera is configured for each polishing robot. Each distributed camera captures the actual vehicle point cloud information. Subsequently, multiple polishing robots perform hand-eye calibration with their corresponding distributed cameras to obtain the second pose between the distributed coordinate system of the distributed camera and the coordinate system of the polishing robot. Figure 8 As shown, taking a distributed camera system including 3D camera 1, 3D camera 1, 3D camera 1 and 3D camera 1 as an example, a grinding robot robotic arm 1, a grinding robot robotic arm 2, a grinding robot robotic arm 3 and a grinding robot robotic arm 4 are set up. In the process of mapping the three-dimensional defect position information from the vehicle coordinate system to the grinding robot coordinate system, the three-dimensional defect position information in the vehicle coordinate system + the point cloud configuration result of the actual vehicle point cloud information and the simulated vehicle point cloud information corresponding to 3D cameras 1 / 2 / 3 / 4 + the second pose between 3D cameras 1 / 2 / 3 / 4 and the corresponding grinding robot 1 / 2 / 3 / 4 coordinate systems respectively = the grinding defect position information in the grinding robot 1 / 2 / 3 / 4 coordinate system.

[0140] In the above embodiments, the grinding auxiliary cameras are placed in different layouts to accurately map all three-dimensional defect location information from the vehicle coordinate system corresponding to the simulated vehicle to the preset grinding robot coordinate system, so as to obtain accurate defect grinding positions and thus achieve accurate defect grinding.

[0141] In an exemplary embodiment, the specific process of the vehicle surface defect detection method of this application is as follows:

[0142] 1. System control components:

[0143] ① The controller, usually a PLC module, is mainly responsible for the overall hardware control and sends signals to control the hardware operation.

[0144] ② Photoelectric sensor: Used to detect whether the vehicle has arrived at the inspection area. When the vehicle is in the inspection area, the defect detection program is started.

[0145] ③ Encoder: Used to detect the forward distance of the vehicle body and determine its position; whenever the vehicle body moves a specified distance, it will control the camera to take a picture.

[0146] ④ Vision module (camera): A binary striped LED light source is used to illuminate the surface of the vehicle body to generate a binary striped light stripe. At the same time, the camera captures the image of the striped light stripe reflected on the vehicle body. Specifically, the camera and the light source are mounted on a fixed bracket or held by a robotic arm, so that the camera is relatively translated to the vehicle body. The camera captures the image of the striped light stripe reflected on the vehicle body, which is moving relative to the detection system in real time.

[0147] ⑤ Processing and Calculation Unit: Located within the controller, this unit processes the acquired images, identifies and locates defects on the vehicle body, determines the defect type, calculates the defect size, and then combines the vehicle body CAD 3D model, the vehicle body reference position during photography, and camera calibration parameters to determine the specific location of the defect on the vehicle body. This location is then mapped onto the vehicle body 3D model and further displayed on the software interface to show the defect location information on various parts of the vehicle body.

[0148] 2. System calibration process:

[0149] Calibration process: Small calibration plates (such as AprilTags) are evenly distributed on the body of a standard prototype vehicle. Each calibration plate has its own ID, ensuring that each calibration plate detected by a camera has a unique identifier. After the calibration plates are applied, the vehicle body is sent into a tunnel-type defect detection system. Following a standard procedure, the vehicle body moves and calibration data is collected multiple times at equal intervals. In other words, when the vehicle is detected to have moved a preset distance, multiple cameras are controlled to capture surface images of feature holes on the vehicle surface.

[0150] After image acquisition, the AprilTag corner points are extracted using a colmap-based method. Based on the known size information, the intrinsic parameters of each camera and the extrinsic parameters between cameras can be calculated. Furthermore, since the vehicle is in motion, the above method calculates the intrinsic parameters of each camera and the extrinsic parameters between cameras at different times as the cameras collect calibration data at regular intervals.

[0151] 3. Simulation environment setup:

[0152] ① Determining the positions of each camera in the simulation environment: After importing the camera model into the simulation environment, set the camera's focal length, image width, image height, chip size, and other parameters to be consistent with the actual camera to achieve consistent shooting effects between the virtual and real cameras. Subsequently, based on the extrinsic parameter relationships between the cameras obtained from the above colmap, update the relative positions of the cameras in the simulation environment.

[0153] ② Determining the position of the camera and the vehicle body in the simulation environment: After importing the vehicle model into the simulation environment, it is necessary to determine the relative position between the vehicle model and the camera. This application uses a PnP-based method to obtain the relative relationship between the vehicle model coordinate system and the camera coordinate system, thereby ensuring that the positional relationship between the vehicle model and the camera in the virtual and real environments can be deployed consistently.

[0154] Based on the known vehicle body digital model design and assembly information, the 3D hole center coordinates of the feature holes in the vehicle coordinate system are directly obtained. Based on the vehicle body image information actually collected by the camera, such as surface images, the 2D pixel coordinates corresponding to the above 3D hole centers are extracted using an ellipse fitting algorithm.

[0155] Based on known camera intrinsic parameters and 3D-2D point pairs, the PnP algorithm is used to solve the pose of the camera coordinate system relative to the vehicle coordinate system, and the pose is updated in the simulation environment to achieve consistency between the virtual and real environments.

[0156] 4. Defect detection process:

[0157] Based on deep learning algorithms for object detection, defect sample data is pre-collected, labeled, and used to train the inference model, accurately extracting two-dimensional defect location information from surface images. Furthermore, defect type and size information can also be obtained from the surface images.

[0158] 5. Three-dimensional mapping of defects:

[0159] Since the camera captures data at the same time as the calibration, the camera's pose relative to the vehicle body at a given moment can be determined by the calibration results described above. A ray tracing method can be used: a ray is established from the 2D defect location information to the camera's optical center, and the collision point between the ray and the vehicle body model is calculated in a simulation environment. This yields the 3D defect location information corresponding to the defect in the vehicle coordinate system, which is then visualized in the software interface. Finally, the 3D defect location information, defect type information, and defect size information of each defect are correlated to obtain the vehicle surface defect detection results.

[0160] 6. Defect removal:

[0161] In practical applications, there may be situations where multiple cameras or the same camera captures the same defect data at different times. That is, there may be multiple observations of the same defect. The results of multiple observations can be mapped onto the vehicle coordinate system, and an error tolerance value can be set. If the Euclidean distance between defects is less than the error tolerance value, it is judged as a duplicate point.

[0162] After completing the vehicle surface defect detection, the polishing robot is controlled to perform the surface defect polishing operation. The specific process is as follows:

[0163] 1. Defect coordinate transfer:

[0164] Through the above-mentioned 3D defect mapping, the 3D defect position information and normal information of the defect in the vehicle coordinate system can be obtained. Now it is necessary to transfer it to the base coordinate system of the grinding robot to guide the grinding.

[0165] Grinding station layout: You can use either a "global" or "distributed" layout.

[0166] The similarities between the two are as follows:

[0167] There are generally four polishing robots (considering the polishing cycle), distributed around the vehicle body, each responsible for a quarter of the vehicle body area.

[0168] The differences between the two are as follows:

[0169] ① A global layout involves suspending a 3D camera directly above the polishing station to capture point cloud information of the vehicle's top surface. Subsequently, four polishing robots perform hand-eye calibration with this 3D camera. Using the registration results of the actual vehicle point cloud and the simulated vehicle point cloud, as well as the hand-eye calibration results, the defect information can be transformed from the vehicle coordinate system to the polishing robot coordinate system (e.g., defect A in the vehicle coordinate system + point cloud registration result + hand-eye calibration of polishing robots 1 / 2 / 3 / 4 = defect A in the base coordinate system of polishing robots 1 / 2 / 3 / 4).

[0170] ② In a distributed layout, each polishing robot is equipped with a 3D camera, installed near its respective robot, to capture point cloud information of the vehicle body. Subsequently, the four polishing robots perform hand-eye calibration with their respective 3D cameras. Using the registration results of the actual vehicle point cloud captured by a certain 3D camera and the simulated vehicle point cloud in the simulation, as well as the hand-eye calibration results of the polishing robot associated with that 3D camera, the defect information can be transformed from the vehicle coordinate system to the coordinate system of that polishing robot (e.g., defect A in the vehicle coordinate system + point cloud registration results of 3D cameras 1 / 2 / 3 / 4 + hand-eye calibration of polishing robots 1 / 2 / 3 / 4 = defect A in the base coordinate system of polishing robots 1 / 2 / 3 / 4).

[0171] 2. Defect-guided grinding and precision positioning system:

[0172] Using the methods described above, the coordinates and normals of the defect in the grinding robot's base coordinate system have been obtained, allowing the grinding equipment to be directly guided for grinding operations. During actual grinding, if the three-dimensional mapping error or coordinate transformation error of the defect is large, resulting in poor defect positioning accuracy and grinding effect, it is advisable to equip the grinding robot with an additional precision positioning device at its end effector. Using the aforementioned defect positioning results as initial values, the precision positioning device can be guided to perform secondary precision positioning, thereby achieving precision-guided grinding of the defect.

[0173] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0174] Based on the same inventive concept, this application also provides a vehicle surface defect detection device for implementing the vehicle surface defect detection method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more vehicle surface defect detection device embodiments provided below can be found in the limitations of the vehicle surface defect detection method described above, and will not be repeated here.

[0175] In one exemplary embodiment, this is applied to a tunnel-type defect detection system, which includes a tunnel-type detection space and multiple cameras, such as... Figure 9 As shown, a vehicle surface defect detection device is provided, including: a distance detection module 100, an image acquisition module 200, a two-dimensional defect detection module 300, a three-dimensional defect generation module 400, and a defect result generation module 500, wherein:

[0176] The distance detection module 100 is used to detect the forward distance of the vehicle to be detected when the vehicle to be detected enters the tunnel-type detection space.

[0177] Image acquisition module 200 is used to control multiple cameras to capture surface images of the vehicle to be inspected based on the forward distance;

[0178] The two-dimensional defect detection module 300 is used to detect two-dimensional defect information of defects in multiple surface images. The two-dimensional defect information includes at least two-dimensional defect location information, defect type information and defect size information.

[0179] The 3D defect generation module 400 is used to convert the 2D defect location information into 3D defect location information in the vehicle coordinate system corresponding to the vehicle for each 2D defect location information.

[0180] The defect result generation module 500 is used to associate the three-dimensional defect location information, defect type information and defect size information corresponding to each two-dimensional defect location information to obtain the vehicle surface defect detection results.

[0181] In one embodiment, the vehicle surface defect detection device further includes a simulation environment construction module. The simulation environment construction module is used to acquire, for each camera in each time period, two-dimensional feature hole position information of feature holes in the surface image captured by the camera, and three-dimensional feature hole position information of feature holes in the vehicle coordinate system corresponding to the vehicle to be detected. Based on the three-dimensional feature hole position information and the two-dimensional feature hole position information, it generates the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system. Based on all first poses in different time periods and the preset parameter calibration information of all cameras, it constructs a vehicle operation simulation environment. The three-dimensional defect generation module 400 is also used to convert the two-dimensional defect position information into three-dimensional defect position information in the vehicle coordinate system corresponding to the vehicle based on the constructed vehicle operation simulation environment.

[0182] In one embodiment, the preset parameter calibration information of all cameras includes the internal calibration parameters of each camera and the external calibration parameters between cameras. The vehicle surface defect detection device also includes a camera parameter calibration module. The camera parameter calibration module is used to detect multiple corner point information of the calibration image from surface images of multiple time periods for each camera when the calibration image is pasted onto the vehicle surface, and to detect the internal calibration parameters of the camera based on the multiple corner point information; and to detect the external calibration parameters between cameras in each time period based on the surface images of all cameras in each time period.

[0183] In one embodiment, the vehicle operation simulation environment includes a simulated vehicle to be detected corresponding to the vehicle to be detected and a simulated camera corresponding to each camera; the three-dimensional defect generation module 400 is further configured to, in the vehicle operation simulation environment, for each two-dimensional defect location information, emit rays from the two-dimensional defect location information as the starting point to the simulated camera corresponding to the two-dimensional defect location information to obtain ray detection results; when the ray detection results indicate that the ray collides with the simulated vehicle to be detected, obtain the collision point location information between the ray and the simulated vehicle to be detected, and use the collision point location information as the three-dimensional defect location information of the vehicle coordinate system corresponding to the vehicle.

[0184] In one embodiment, the vehicle surface defect detection device further includes a defect grinding module, which is used to map the three-dimensional defect location information from the vehicle coordinate system to a preset grinding robot coordinate system to obtain grinding defect location information; based on the grinding defect location information, the grinding robot is controlled to perform vehicle surface defect grinding operations.

[0185] In one embodiment, the vehicle operation simulation environment includes a simulated vehicle to be tested corresponding to the vehicle to be tested; the defect polishing module is further configured to, when the surface defect polishing operation is assisted by a polishing auxiliary camera, acquire the placement type of the polishing auxiliary camera and the simulated vehicle point cloud information of the simulated vehicle to be tested in the vehicle operation simulation environment; based on the placement type, acquire the actual vehicle point cloud information of the vehicle captured by the polishing auxiliary camera, and the second pose between the coordinate system of the polishing auxiliary camera and the coordinate system of the polishing robot; based on the point cloud registration result generated by the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second pose, map the three-dimensional defect location information from the vehicle coordinate system to the polishing robot coordinate system.

[0186] In one embodiment, the polishing auxiliary camera includes a global camera or multiple distributed cameras. The coordinate system of the polishing auxiliary camera includes the global coordinate system corresponding to the global camera and the distributed coordinate system corresponding to each distributed camera. The placement type includes placing the global camera directly above the polishing station or placing each distributed camera in the vicinity of the corresponding polishing robot. The defect polishing module is further used to acquire the actual vehicle point cloud information of the vehicle captured by the global camera and the second pose between the global coordinate system and the polishing robot coordinate system when the placement type includes placing the global camera directly above the polishing station; and to acquire the actual vehicle point cloud information of the vehicle captured by the distributed camera and the second pose between the distributed coordinate system and the polishing robot coordinate system for each distributed camera when the placement type includes placing each distributed camera in the vicinity of the corresponding polishing robot.

[0187] Each module in the aforementioned vehicle surface defect detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0188] In one exemplary embodiment, a vehicle surface defect detection system is also provided, the system comprising:

[0189] Photoelectric sensors are used to detect whether a vehicle under inspection has entered the tunnel-type inspection space;

[0190] Multiple cameras are used to capture surface images of the vehicle to be inspected;

[0191] An encoder is used to detect the forward distance of a vehicle being inspected.

[0192] The controller is used to control the encoder to detect the forward distance of the vehicle under inspection when the vehicle under inspection enters the tunnel-type inspection space; based on the forward distance, control multiple cameras to capture surface images of the vehicle under inspection; detect two-dimensional defect information of defects in multiple surface images, the two-dimensional defect information including at least two-dimensional defect location information, defect type information and defect size information; for each two-dimensional defect location information, convert the two-dimensional defect location information into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle; associate the three-dimensional defect location information, defect type information and defect size information corresponding to each two-dimensional defect location information to obtain the vehicle surface defect detection result.

[0193] In addition, the vehicle surface defect detection system also includes: a light source illumination device: emitting a light source onto the vehicle body surface.

[0194] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as surface images. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for detecting surface defects on vehicles.

[0195] Those skilled in the art will understand that Figure 10 The structure shown is a block diagram of a partial structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0196] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0197] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0198] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0199] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0200] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0201] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting surface defects on a vehicle, characterized in that, The method is applied to a tunnel-type defect detection system, which includes a tunnel-type detection space and multiple cameras. When the vehicle to be tested enters the tunnel-type testing space, the forward distance of the vehicle to be tested is detected; Based on the forward distance, the multiple cameras are controlled to capture surface images of the vehicle to be detected, thereby obtaining surface images of the vehicle captured by the multiple cameras at different time periods; Two-dimensional defect information is detected in multiple surface images, wherein the two-dimensional defect information includes at least two-dimensional defect location information, defect type information, and defect size information; For each of the two-dimensional defect location information, the two-dimensional defect location information is converted into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle. By associating the three-dimensional defect location information, defect type information, and defect size information of each defect, the vehicle surface defect detection results are obtained. Before converting the two-dimensional defect location information into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle, the method further includes: for each camera in each time period, acquiring two-dimensional feature hole location information of feature holes in the surface image captured by the camera, and three-dimensional feature hole location information of the feature holes in the vehicle coordinate system corresponding to the vehicle to be detected; generating a first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system based on the three-dimensional feature hole location information and the two-dimensional feature hole location information; constructing a vehicle operation simulation environment based on all first poses and preset parameter calibration information of all cameras in different time periods; and then converting the two-dimensional defect location information into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle based on the constructed vehicle operation simulation environment.

2. The method according to claim 1, characterized in that, The preset parameter calibration information for all the cameras includes the internal calibration parameters of each camera and the external calibration parameters between the cameras. Before constructing the vehicle operation simulation environment based on the preset parameter calibration information of all the first poses and all the cameras in different time periods, it also includes: When the calibration image is pasted onto the vehicle surface, for each camera, multiple corner information of the calibration image is detected from surface images over multiple time periods, and the internal calibration parameters of the camera are detected based on the multiple corner information. Based on the surface images of all the cameras in each time period, detect the external calibration parameters between the cameras in each time period.

3. The method according to claim 1, characterized in that, The vehicle operation simulation environment includes the simulated vehicle to be tested corresponding to the vehicle to be tested and the simulated camera corresponding to each of the cameras. The step of converting each two-dimensional defect location information into three-dimensional defect location information in the vehicle coordinate system includes: In the vehicle operation simulation environment, for each of the two-dimensional defect location information, a ray is emitted from the two-dimensional defect location information as the starting point to the simulation camera corresponding to the two-dimensional defect location information to obtain the ray detection result; When the ray detection result indicates that the ray collides with the simulated vehicle to be detected, the collision point location information between the ray and the simulated vehicle to be detected is obtained, and the collision point location information is used as the three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle.

4. The method according to claim 1, characterized in that, After associating the three-dimensional defect location information, defect type information, and defect size information of each defect to obtain the vehicle surface defect detection results, the method further includes: The three-dimensional defect location information is mapped from the vehicle coordinate system to the preset grinding robot coordinate system to obtain the grinding defect location information; Based on the location information of the grinding defects, the grinding robot is controlled to perform grinding operations on the surface defects of the vehicle.

5. The method according to claim 4, characterized in that, The vehicle operation simulation environment includes the simulated vehicle to be tested corresponding to the vehicle to be tested; The step of mapping the three-dimensional defect location information from the vehicle coordinate system to the preset grinding robot coordinate system includes: In the case of using a grinding auxiliary camera to assist in the grinding operation of vehicle surface defects, the placement type of the grinding auxiliary camera and the point cloud information of the simulated vehicle to be detected in the vehicle operation simulation environment are obtained. Based on the placement type, the actual vehicle point cloud information captured by the polishing auxiliary camera and the second pose between the polishing auxiliary camera coordinate system and the polishing robot coordinate system are obtained. Based on the point cloud registration result generated from the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second pose, the three-dimensional defect location information is mapped from the vehicle coordinate system to the grinding robot coordinate system.

6. The method according to claim 5, characterized in that, The polishing auxiliary camera includes a global camera or multiple distributed cameras. The polishing auxiliary camera coordinate system includes the global coordinate system corresponding to the global camera and the distributed coordinate system corresponding to each of the distributed cameras. The placement type includes placing the global camera directly above the polishing station or placing each distributed camera within the vicinity of its corresponding polishing robot. Based on the placement type, acquiring the actual vehicle point cloud information captured by the polishing auxiliary camera, and the second pose between the polishing auxiliary camera coordinate system and the polishing robot coordinate system, includes: When the placement type includes placing the global camera directly above the polishing station, the actual vehicle point cloud information captured by the global camera and the second pose between the global coordinate system and the polishing robot coordinate system are obtained. When the placement type includes placing each distributed camera in the vicinity of the corresponding polishing robot, for each of the distributed cameras, the actual vehicle point cloud information of the vehicle captured by the distributed camera, and the second pose between the distributed coordinate system and the polishing robot coordinate system are obtained.

7. A vehicle surface defect detection device, characterized in that, An apparatus for use in tunnel-type defect detection systems, the tunnel-type defect detection system comprising a tunnel-type detection space and multiple cameras, the apparatus comprising: The distance detection module is used to detect the forward distance of the vehicle under test when the vehicle under test enters the tunnel-type detection space; An image capturing module is used to control the multiple cameras to capture surface images of the vehicle to be detected based on the forward distance, thereby obtaining surface images of the vehicle captured by the multiple cameras at different time periods. A two-dimensional defect detection module is used to detect two-dimensional defect information of defects in multiple surface images. The two-dimensional defect information includes at least two-dimensional defect location information, defect type information, and defect size information. The three-dimensional defect generation module is used to convert the two-dimensional defect location information into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle for each of the two-dimensional defect location information. The defect result generation module is used to associate the three-dimensional defect location information, defect type information and defect size information corresponding to each of the two-dimensional defect location information to obtain the vehicle surface defect detection results. Before converting the two-dimensional defect location information into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle, for each camera within each time period, the two-dimensional feature hole location information of the feature hole in the surface image captured by the camera, and the three-dimensional feature hole location information of the feature hole in the vehicle coordinate system corresponding to the vehicle to be detected are acquired. Based on the three-dimensional feature hole location information and the two-dimensional feature hole location information, the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system is generated. Based on all the first poses in different time periods and the preset parameter calibration information of all the cameras, a vehicle operation simulation environment is constructed. Then, based on the constructed vehicle operation simulation environment, the two-dimensional defect location information is converted into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle.

8. The apparatus according to claim 7, characterized in that, The device also includes a defect polishing module, which is used to map the three-dimensional defect location information from the vehicle coordinate system to a preset polishing robot coordinate system to obtain polishing defect location information; based on the polishing defect location information, the polishing robot is controlled to perform vehicle surface defect polishing operation.

9. A tunnel-type defect detection system, characterized in that, The system includes a controller, an encoder, photoelectric sensors, a tunnel-type detection space, and multiple cameras; The photoelectric sensor is used to detect whether the vehicle to be detected has entered the tunnel-type detection space; The plurality of cameras are used to capture surface images of the vehicle to be inspected; The encoder is used to detect the forward distance of the vehicle to be detected; The controller is configured to, when the photoelectric sensor detects that the vehicle to be inspected has entered the tunnel-type inspection space, control the encoder to detect the forward distance of the vehicle to be inspected, and obtain surface images of the vehicle captured by multiple cameras within different time periods; based on the forward distance, control the multiple cameras to capture surface images of the vehicle to be inspected; and detect two-dimensional defect information of defects in the multiple surface images, wherein the two-dimensional defect information includes at least two-dimensional defect location information, defect type information, and defect size information. For each of the two-dimensional defect location information, the two-dimensional defect location information is converted into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle; the three-dimensional defect location information, defect type information and defect size information corresponding to each of the two-dimensional defect location information are associated to obtain the vehicle surface defect detection result; Before converting the two-dimensional defect location information into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle, for each camera in each time period, the two-dimensional feature hole location information of the feature hole in the surface image captured by the camera, and the three-dimensional feature hole location information of the feature hole in the vehicle coordinate system corresponding to the vehicle to be detected are obtained. Based on the three-dimensional feature hole location information and the two-dimensional feature hole location information, the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system is generated. Based on all the first poses in different time periods and the preset parameter calibration information of all the cameras, a vehicle operation simulation environment is constructed. Then, based on the constructed vehicle operation simulation environment, the two-dimensional defect location information is converted into three-dimensional defect location information in the vehicle coordinate system corresponding to the vehicle.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images