Vehicle surface defect detection method, device and system and readable storage medium

Abstract

The invention relates to a vehicle surface defect detection method, device and system and a readable storage medium. The method comprises the following steps: detecting the advancing distance of a to-be-detected vehicle under the condition that the to-be-detected vehicle enters a tunnel type detection space; based on the forward distance, multiple cameras are controlled to shoot surface images of the to-be-detected vehicle; detecting two-dimensional defect information of defects in the plurality of surface images, wherein the two-dimensional defect information at least comprises two-dimensional defect position information, defect type information and defect size information; for each piece of two-dimensional defect position information, converting the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle; and associating the three-dimensional defect position information, the defect type information and the defect size information of each defect to obtain a vehicle surface defect detection result. By adopting the method, the vehicle surface defects can be accurately polished.

Description

Technical Field

[0001] The present application relates to the field of defect detection technology, and in particular to a vehicle surface defect detection method, device, system and readable storage medium. Background Art

[0002] During the vehicle production process, the surface of the vehicle body will inevitably produce defects such as pits and particles. The quality of the surface not only affects the aesthetics of the vehicle body, but also affects the wear resistance and corrosion resistance of the vehicle body. Therefore, it is necessary to polish the defects on the vehicle surface.

[0003] At present, vehicle surface defect detection technology mostly uses a robot arm carrying a camera light source for detection. Its detection mode is generally "stop-and-go". There is also a tunnel-type detection method, where the vehicle body can be inspected as it enters the tunnel along the production line, which improves the detection efficiency to a certain extent. After the defects on the vehicle surface are detected, the defect grinding operation can be performed to guide the grinding device to grind and polish the defects.

[0004] However, the current vehicle surface defect detection technology is not accurate enough, and a more accurate vehicle surface defect detection technology is needed to detect defects on the vehicle surface. Summary of the invention

[0005] Based on this, it is necessary to provide an accurate vehicle surface defect detection method, device, system, computer equipment, computer readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a vehicle surface defect polishing method, which is applied to a tunnel defect detection system, wherein the tunnel defect detection system includes a tunnel detection space and multiple cameras, and the method includes:

[0007] When the vehicle to be detected enters the tunnel-type detection space, detecting the forward distance of the vehicle to be detected;

[0008] Based on the advancing distance, a plurality of cameras are controlled to capture a surface image of the vehicle to be detected;

[0009] Detecting two-dimensional defect information of defects in a plurality of surface images, the two-dimensional defect information at least including two-dimensional defect position information, defect type information and defect size information;

[0010] For each two-dimensional defect position information, convert the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle;

[0011] The three-dimensional defect position information, defect type information and defect size information of each defect are associated to obtain the vehicle surface defect detection result.

[0012] In a second aspect, the present application also provides a vehicle surface defect grinding device, comprising:

[0013] A distance detection module is used to detect the forward distance of the vehicle to be detected when the vehicle to be detected enters the tunnel-type detection space;

[0014] An image capturing module, used for controlling a plurality of cameras to capture a surface image of a vehicle to be detected based on a forward distance;

[0015] A two-dimensional defect detection module, used to detect two-dimensional defect information of defects in multiple surface images, where the two-dimensional defect information at least includes two-dimensional defect position information, defect type information and defect size information;

[0016] A three-dimensional defect generation module, for each two-dimensional defect position information, converting the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle;

[0017] The defect result generation module is used to associate the three-dimensional defect position information, defect type information and defect size information corresponding to each two-dimensional defect position information to obtain the vehicle surface defect detection result.

[0018] In a third aspect, the present application also provides a tunnel-type defect detection system, the system comprising a controller, an encoder, a photoelectric sensor, a tunnel-type detection space and a plurality of cameras;

[0019] Photoelectric sensor, used to detect whether the vehicle to be detected enters the tunnel detection space;

[0020] A plurality of cameras for capturing surface images of the vehicle to be detected;

[0021] An encoder, used to detect the forward distance of the vehicle to be detected;

[0022] The controller is used to control the encoder to detect the forward distance of the vehicle to be detected when the photoelectric sensor detects that the vehicle to be detected enters the tunnel-type detection space; based on the forward distance, control multiple cameras to capture the surface image of the vehicle to be detected; detect two-dimensional defect information of defects in multiple surface images, the two-dimensional defect information at least including two-dimensional defect position information, defect type information and defect size information; for each two-dimensional defect position information, convert the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle; associate the three-dimensional defect position information, defect type information and defect size information corresponding to each two-dimensional defect position information to obtain a vehicle surface defect detection result.

[0023] In a fourth aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0024] When the vehicle to be detected enters the tunnel-type detection space, detecting the forward distance of the vehicle to be detected;

[0025] Based on the advancing distance, a plurality of cameras are controlled to capture a surface image of the vehicle to be detected;

[0026] Detecting two-dimensional defect information of defects in a plurality of surface images, the two-dimensional defect information at least including two-dimensional defect position information, defect type information and defect size information;

[0027] For each two-dimensional defect position information, convert the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle;

[0028] The three-dimensional defect position information, defect type information and defect size information of each defect are associated to obtain the vehicle surface defect detection result.

[0029] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0030] When the vehicle to be detected enters the tunnel-type detection space, detecting the forward distance of the vehicle to be detected;

[0031] Based on the advancing distance, a plurality of cameras are controlled to capture a surface image of the vehicle to be detected;

[0032] Detecting two-dimensional defect information of defects in a plurality of surface images, the two-dimensional defect information at least including two-dimensional defect position information, defect type information and defect size information;

[0033] For each two-dimensional defect position information, convert the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle;

[0034] The three-dimensional defect position information, defect type information and defect size information of each defect are associated to obtain the vehicle surface defect detection result.

[0035] In a sixth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0036] When the vehicle to be detected enters the tunnel-type detection space, detecting the forward distance of the vehicle to be detected;

[0037] Based on the advancing distance, a plurality of cameras are controlled to capture a surface image of the vehicle to be detected;

[0038] Detecting two-dimensional defect information of defects in a plurality of surface images, the two-dimensional defect information at least including two-dimensional defect position information, defect type information and defect size information;

[0039] For each two-dimensional defect position information, convert the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle;

[0040] The three-dimensional defect position information, defect type information and defect size information of each defect are associated to obtain the vehicle surface defect detection result.

[0041] The above-mentioned vehicle surface defect detection method, device, system, computer equipment, computer-readable storage medium and computer program product detect the forward distance of the vehicle to be detected when the vehicle to be detected enters the tunnel-type detection space; based on the forward distance, control multiple cameras to capture the surface image of the vehicle to be detected; detect the two-dimensional defect information of the defects in the multiple surface images, and the two-dimensional defect information at least includes two-dimensional defect position information, defect type information and defect size information; for each two-dimensional defect position information, convert the two-dimensional defect position information into three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle; associate the three-dimensional defect position information, defect type information and defect size information corresponding to each two-dimensional defect position information to obtain the vehicle surface defect detection result. In the whole process, when the vehicle to be detected enters the tunnel-type detection space, control multiple cameras to capture the surface image of the vehicle to be detected according to the forward distance of the vehicle to be detected, and then perform defect detection on the surface images captured by multiple cameras in multiple time periods, with higher accuracy, so that after associating the three-dimensional defect position information, defect type information and defect size information corresponding to each two-dimensional defect position information, the generated vehicle surface defect detection result is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A diagram showing an application environment of a vehicle surface defect detection method in an embodiment;

[0044] Figure 2 A schematic diagram of a process of a vehicle surface defect detection method in one embodiment;

[0045] Figure 3 A schematic flow chart of a method for polishing vehicle surface defects in another embodiment;

[0046] Figure 4 It is a schematic diagram of pasting evenly distributed small calibration plates on the body of a standard sample vehicle in a specific application embodiment;

[0047] Figure 5 A schematic diagram of visualization results of external parameter calibration in a specific application embodiment;

[0048] Figure 6 A schematic diagram of emitting rays to a simulation camera using two-dimensional defect simulation position information as a starting point in a specific application embodiment;

[0049] Figure 7 A schematic diagram of a placement type in a specific application embodiment including placing a global camera directly above a grinding station;

[0050] Figure 8 It is a schematic diagram of placing each distributed camera within the vicinity of a corresponding polishing robot in a specific application embodiment;

[0051] Fig. 9 is a structural block diagram of a vehicle surface defect polishing device in one embodiment;

[0052] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are used to explain the present application and are not used to limit the present application.

[0054] Before polishing the surface defects of the vehicle, it is often necessary to locate and detect the defects on the vehicle surface. Generally, the vehicle surface defect detection technology mostly uses a robot arm carrying a camera light source for detection. Its detection mode is generally "stop-and-go", but the detection time is long, which affects the production line rhythm. There are also tunnel-type detection methods. The body can be inspected as soon as it enters the tunnel along the production line, which improves the detection efficiency to a certain extent, but the calibration process is more cumbersome, the defect positioning accuracy is not high, and the compatibility with new models is poor.

[0055] Therefore, in response to the above-mentioned problem of accuracy in vehicle surface defect detection, the present application provides an accurate vehicle surface defect detection method. When the vehicle to be inspected enters a tunnel-type inspection space, multiple cameras are controlled to capture surface images of the vehicle to be inspected according to the forward distance of the vehicle to be inspected, and then defect detection is performed on the surface images captured by multiple cameras in multiple time periods with higher accuracy. After associating the three-dimensional defect position information, defect type information and defect size information corresponding to each two-dimensional defect position information, the generated vehicle surface defect detection result is more accurate.

[0056] The vehicle surface defect detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the tunnel defect detection system 104 through the network. The tunnel defect detection system includes a controller 106, a tunnel detection space 108 and multiple cameras 110. The multiple cameras 110 communicate with the controller 106 through the network. The data storage system can store the data that the controller 106 needs to process. The data storage system can be integrated on the controller 106, or it can be placed on the cloud or other network servers. The controller 106 can be a PLC (Programmable Logic Controller) module. The PLC module is responsible for the control of the overall hardware and sends signals to control the hardware work.

[0057] The user operates on the terminal 102 and sends a vehicle surface defect detection request to the controller 106 in the tunnel defect detection system 104. When the vehicle 112 to be detected enters the tunnel detection space 108, the controller 106 detects the forward distance of the vehicle 112 to be detected; based on the forward distance, the controller 106 controls multiple cameras 110 to capture the surface image of the vehicle 112 to be detected; detects the two-dimensional defect information of the defects in the multiple surface images, and the two-dimensional defect information at least includes two-dimensional defect position information, defect type information and defect size information; for each two-dimensional defect position information, the two-dimensional defect position information is converted into three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle; the three-dimensional defect position information, defect type information and defect size information corresponding to each two-dimensional defect position information are associated to obtain the vehicle surface defect detection result. Then, the controller 106 can display the vehicle surface defect detection result to the terminal 102. In addition, the controller 106 can also control the grinding robot to grind the vehicle surface defects according to the vehicle surface defect detection result.

[0058] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, projection devices, etc. In addition, the terminal 102 may be a display screen of the controller 106. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc.

[0059] In an exemplary embodiment, Figure 2 As shown, a vehicle surface defect detection method is provided, and the method is applied to Figure 1Taking the tunnel-type defect detection system 104 in the example as an example, the tunnel-type defect detection system includes a controller, a tunnel-type detection space and a plurality of cameras, including the following S100 to S500. Among them:

[0060] S100: When a vehicle to be detected enters a tunnel-type detection space, a forward distance of the vehicle to be detected is detected.

[0061] Specifically, a photoelectric sensor is provided in the present application, and the photoelectric sensor is used to detect whether the vehicle has reached the tunnel-type detection space. When the vehicle enters the tunnel-type detection space, the vehicle paint defect detection program is started.

[0062] In addition, the present application is also provided with an encoder, which is used to detect the forward distance of the vehicle to be detected to determine the position of the vehicle to be detected. Existing tunnel defect detection technologies mostly use encoders to monitor the displacement of the vehicle body, thereby updating the relative position of the vehicle body and the camera, but the accuracy is not high enough; this method combines the encoder with the calibration of the camera to obtain the relative position of the camera and the vehicle body at multiple moments, with high accuracy.

[0063] S200 , based on the forward distance, controlling multiple cameras to capture a surface image of the vehicle to be detected.

[0064] Among them, the camera can be replaced by other devices for shooting. There is more than one camera, and the number of cameras varies according to actual needs.

[0065] Specifically, whenever the vehicle to be detected moves a specified distance, the controller will control multiple cameras to complete a photo respectively. For example, taking the specified distance of 10 cm as an example, when the vehicle to be detected moves 10 cm, the controller will control multiple cameras to complete a photo respectively. At this time, the forward distance can be recalculated. When the vehicle to be detected moves 10 cm again, the controller will control multiple cameras to complete a photo respectively. In layman's terms, when the forward distance represents that the vehicle to be detected has moved any multiple of the specified distance, multiple cameras are controlled to capture the surface image of the vehicle to be detected. Thus, the vehicle surface images captured by multiple cameras in different time periods can be obtained.

[0066] In an exemplary embodiment, the surface image of the vehicle to be detected is a structured light image captured by structured light technology, wherein the structured light technology uses a pre-designed pattern with a special structure (such as discrete light spots, striped light, coded structured light, etc.), and then projects the pattern onto the surface of a three-dimensional object, and uses another camera to observe the distortion of the imaging on the three-dimensional physical surface. After the pattern is projected onto the surface of the three-dimensional object, the surface image of the three-dimensional object captured by the camera device is a structured light image carrying the projection pattern.

[0067] That is to say, a structured light light source is also provided in the present application, and the structured light source is irradiated on the surface of the vehicle to be inspected to generate a structured light band. At this time, multiple cameras capture the surface image of the vehicle to be inspected, and the surface image is the structured light image. It should be explained that the reason for setting the structured light light source is that when the structured light source is irradiated on the surface of the vehicle to be inspected, the defective area will undergo more obvious changes relative to the surrounding normal paint surface, which can effectively enhance the characteristics of defects such as protrusions, depressions, particles, and shrinkage holes. Generally, structured light sources include but are not limited to light sources such as binary stripe LED (Light Emitting Diode, light emitting diode) light sources. When the structured light source is a binary stripe LED light source, irradiating the binary stripe LED light source on the surface of the vehicle to be inspected will generate a quasi-binary stripe light band.

[0068] In an exemplary embodiment, multiple cameras are installed around the vehicle, and the installation of multiple cameras needs to ensure that the field of view is continuous and can cover the cross-section of the entire vehicle body illuminated by the structured light source. At the same time, the appropriate camera lens is selected to cover a wider inspection area, and the spatial resolution of the camera is improved as much as possible so that smaller defects can be detected.

[0069] In another exemplary embodiment, the camera and the structured light source can be mounted on a fixed bracket or held by a robot arm, so that the camera and the vehicle body can be relatively translated, and the camera can collect in real time the surface image reflected on the vehicle body that is moving in real time relative to the detection system. In addition, the bracket fixing or robot arm holding method can also be used in combination, such as using a fixed bracket method for the left and right sides of the vehicle body whose height positions are relatively fixed, and using a relatively flexible robot arm holding method for areas whose height positions change greatly with the vehicle model, such as the roof, front cover, and rear, so that the detection point can be adjusted in real time as the vehicle body moves; the camera and structured light source can also be installed by bracket fixing, which is more suitable for scenes where the size of the vehicle model on the production line changes slightly.

[0070] S300, detecting two-dimensional defect information of defects in a plurality of surface images, where the two-dimensional defect information at least includes two-dimensional defect position information, defect type information, and defect size information.

[0071] Specifically, the detection of two-dimensional defect information of defects in multiple surface images is mainly achieved through target detection deep learning algorithms. That is to say, defect sample data is collected in advance for annotation and training of the inference model, and then the two-dimensional defect information of the simulated vehicle is accurately detected from each surface image through the trained inference model. Moreover, since the surface images are surface images collected in multiple time periods, the amount of two-dimensional defect information obtained is larger and more comprehensive.

[0072] Furthermore, since the surface image is a structured light image that enhances the defect features, the surface image can also be processed to determine the defect type information and defect size information of the defect from the structured light image, for example, to determine whether the defect type is a protrusion or a depression, to determine whether the size of the defect is greater than a preset safety threshold, etc., and to combine the two-dimensional defect position information, defect type information and defect size information of the defect to generate two-dimensional defect information of the defect.

[0073] S500, for each two-dimensional defect position information, convert the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle.

[0074] Specifically, the two-dimensional defect position information is the two-dimensional defect position information in the image coordinate system, and the two-dimensional defect position information also needs to be converted into the three-dimensional vehicle coordinate system to obtain the three-dimensional defect position information in the vehicle coordinate system.

[0075] It should be noted that in actual environments, due to the scale uncertainty of the monocular camera, it is impossible to directly infer the three-dimensional defect location information from the two-dimensional defect location information. Therefore, this application will use the ray tracing method to convert the two-dimensional defect location information into the three-dimensional defect location information of the vehicle coordinate system corresponding to the vehicle.

[0076] S500, associating the three-dimensional defect position information, defect type information and defect size information of each defect to obtain a vehicle surface defect detection result.

[0077] Specifically, the three-dimensional defect position information, defect type information and defect size information of each defect are associated to obtain the detection result of the defect, and then based on the detection results of all defects, the vehicle surface defect detection result is obtained.

[0078] Furthermore, the vehicle surface defect detection results are analyzed, and when the vehicle surface defect detection results indicate that defect polishing is required, the polishing robot is controlled to perform defect polishing according to the three-dimensional defect position information in the vehicle surface defect detection results.

[0079] In the above vehicle surface defect detection method, when the vehicle to be detected enters the tunnel-type detection space, the forward distance of the vehicle to be detected is detected; based on the forward distance, multiple cameras are controlled to capture the surface image of the vehicle to be detected; two-dimensional defect information of defects in multiple surface images is detected, and the two-dimensional defect information at least includes two-dimensional defect position information, defect type information and defect size information; for each two-dimensional defect position information, the two-dimensional defect position information is converted into three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle; the three-dimensional defect position information, defect type information and defect size information corresponding to each two-dimensional defect position information are associated to obtain the vehicle surface defect detection result. In the whole process, when the vehicle to be detected enters the tunnel-type detection space, multiple cameras are controlled to capture the surface image of the vehicle to be detected according to the forward distance of the vehicle to be detected, and then defect detection is performed on the surface images captured by multiple cameras in multiple time periods, with higher accuracy, so that after associating the three-dimensional defect position information, defect type information and defect size information corresponding to each two-dimensional defect position information, the generated vehicle surface defect detection result is more accurate.

[0080] In an exemplary embodiment, Figure 3 As shown, before S400, the method further includes:

[0081] S360, for each camera in each time period, obtain the two-dimensional feature hole position information of the feature hole in the surface image taken by the camera, and the three-dimensional feature hole position information of the feature hole in the vehicle coordinate system corresponding to the vehicle to be detected, and based on the three-dimensional feature hole position information and the two-dimensional feature hole position information, generate the first position pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system.

[0082] S370, constructing a vehicle operation simulation environment based on all first poses and preset parameter calibration information of all cameras in different time periods.

[0083] The characteristic hole is a typical hole on the surface of the vehicle to be detected. The three-dimensional characteristic hole position information and the two-dimensional characteristic hole position information can be three-dimensional coordinate information or other information used to characterize the position.

[0084] Specifically, there are characteristic holes distributed on the surface of the vehicle to be inspected. Therefore, the surface image taken by the camera contains the characteristic hole information. For each camera in each time period, the two-dimensional characteristic hole position information of the characteristic holes is obtained from the surface image taken by the camera. In addition, it is also necessary to obtain the three-dimensional characteristic hole position information of the characteristic holes in the vehicle coordinate system corresponding to the vehicle to be inspected.

[0085] Based on the three-dimensional feature hole position information and the two-dimensional feature hole position information of the same feature hole, the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system is generated. In this process, the first pose of the camera coordinate system relative to the vehicle coordinate system can be obtained by the PnP (Perspective-n-Point, multi-point perspective imaging) method. It should be understood that PnP is a geometric problem in computer vision, which is mainly used to estimate the pose of the camera. Specifically, given a set of 3D (3-Dimensions, three-dimensional) points and their corresponding 2D (2-Dimensions, two-dimensional) point coordinates in the image, the PnP algorithm can calculate the relative relationship between the camera coordinate system and the 3D point coordinate system (vehicle coordinate system) in combination with the camera parameter calibration information.

[0086] Finally, the vehicle operation simulation environment is constructed based on the preset parameter calibration information of all first poses and all cameras in different time periods.

[0087] In an exemplary embodiment, the three-dimensional feature hole position information of the feature hole in the vehicle coordinate system corresponding to the vehicle to be detected is obtained, including: in the vehicle coordinate system, obtaining the hole center position information of the feature hole as the three-dimensional feature hole position information of the feature hole, more specifically, obtaining the vehicle body digital model design and assembly information, and then directly obtaining the hole center position information of the feature hole based on the vehicle body digital model design and assembly information, and then using the hole center position information of the feature hole as the three-dimensional feature hole position information of the feature hole.

[0088] In an exemplary embodiment, obtaining two-dimensional feature hole position information of feature holes in a surface image taken by a camera includes: extracting the two-dimensional feature hole position information of the feature holes in the surface image using an ellipse fitting algorithm. In practical applications, the ellipse fitting algorithm is implemented using a least squares method or a matrix decomposition method.

[0089] In an exemplary embodiment, constructing a vehicle operation simulation environment includes: importing simulation cameras corresponding to multiple cameras and simulated vehicles to be detected corresponding to the vehicles to be detected into the vehicle operation simulation environment, and the parameter calibration information of the multiple simulation cameras, the relative positions between the multiple simulation cameras, and the relative positions between the multiple simulation cameras and the simulated vehicles to be detected need to be consistent with the actual environment.

[0090] Specifically, first, multiple simulation cameras are imported into the initial vehicle operation simulation environment. When importing multiple simulation cameras, the focal length, image width, image height, chip size and other parameters of the simulation cameras need to be set to be consistent with the cameras in the actual environment, so as to achieve consistent shooting effects of virtual and real cameras. At this time, the parameter calibration information of the simulation camera can also be updated according to the preset parameter calibration information of each camera in the actual environment.

[0091] Secondly, when importing the simulated vehicle to be detected corresponding to the vehicle to be detected into the vehicle operation simulation environment, it is necessary to determine the relative position between the simulated vehicle to be detected and multiple simulated cameras. At this time, the first pose of the camera coordinate system relative to the vehicle coordinate system can be used as the pose of the simulated camera coordinate system corresponding to the simulated camera relative to the simulated vehicle coordinate system corresponding to the simulated vehicle to be detected, so as to ensure that the positional relationship between the vehicle and the camera in the virtual and real environments can be deployed consistently.

[0092] In an exemplary embodiment, based on the three-dimensional feature hole position information and the two-dimensional feature hole position information, the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system is generated, which also includes: obtaining the internal calibration parameters of the camera; based on the three-dimensional feature hole position information, the two-dimensional feature hole position information and the internal calibration parameters, the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system is generated.

[0093] Among them, based on the three-dimensional feature hole position information, the two-dimensional feature hole position information and the internal calibration parameters, the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system is generated, including: based on the internal calibration parameters, the two-dimensional feature hole position information is mapped from the image coordinate system to the camera coordinate system to obtain the intermediate feature hole position information; based on the intermediate feature hole position information and the three-dimensional feature hole position information, the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system is generated.

[0094] Specifically, generating the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system is achieved through the PnP algorithm. The PnP algorithm can be formalized as follows: given the coordinates of n three-dimensional space points and their two-dimensional coordinates on the image, as well as the camera's intrinsic parameter matrix, solve the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system.

[0095] Therefore, based on the three-dimensional feature hole position information, the two-dimensional feature hole position information and the internal calibration parameters, the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system can be jointly generated to update the first pose to the vehicle operation simulation environment and achieve consistent deployment of the virtual and real environments.

[0096] In detail, the expression of the PnP algorithm includes: . Among them, p is the two-dimensional feature hole position information on the surface image, P is the three-dimensional feature hole position information corresponding to the two-dimensional feature hole position information, K is the internal calibration parameter of the camera, R is the rotation matrix, t is the translation vector, [R | t] is the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system, and s is a scaling factor.

[0097] It can be seen that if the first pose [R | t] needs to be obtained, the two-dimensional feature hole position information can be first mapped from the image coordinate system to the camera coordinate system based on the internal calibration parameters of the camera to obtain the intermediate feature hole position information. Since the intermediate feature hole position information is the position information in the camera coordinate system, and the three-dimensional feature hole position information is the position information in the vehicle coordinate system, the first pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system can be generated based on the intermediate feature hole position information and the three-dimensional feature hole position information.

[0098] In an exemplary embodiment, the vehicle operation simulation environment is established based on the first pose of all cameras in all time periods and the preset parameter calibration information. Therefore, there may be multiple simulation cameras or the same simulation camera capturing the same defect data at different times, that is, there may be multiple observations of the same defect. The two-dimensional defect position information of the defect observed multiple times can be mapped to the vehicle coordinate system to obtain multiple three-dimensional defect position information, and the error tolerance value is set. If the Euclidean distance between any two three-dimensional defect position information is less than the error tolerance value, the defects corresponding to the two three-dimensional defect position information are judged to be the same defect, and one of the duplicate information is removed; if the Euclidean distance between any two three-dimensional defect position information is greater than the error tolerance value, the defects corresponding to the two three-dimensional defect position information are considered to be independent defects. In some extreme cases, if multiple three-dimensional defect position information of the same defect exceeds the error tolerance value, it is considered that the calibration accuracy in the vehicle surface defect detection process is poor.

[0099] At this time, S400 also includes: for each two-dimensional defect position information, based on the constructed vehicle operation simulation environment, converting the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle.

[0100] That is to say, in the constructed vehicle operation simulation environment, each two-dimensional defect position information is converted into three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle through the ray tracing method.

[0101] In the above embodiment, by constructing a vehicle operation simulation environment consistent with the real environment, preliminary preparations can be made for the subsequent virtual-reality mapping operation in the vehicle operation simulation environment, that is, for obtaining three-dimensional defect position information mapped from two-dimensional defect position information.

[0102] In an exemplary embodiment, the preset parameter calibration information of all cameras includes internal calibration parameters of each camera and external calibration parameters between cameras. Before building a vehicle operation simulation environment based on all first poses and preset parameter calibration information of all cameras in different time periods, it also includes:

[0103] When the calibration image is pasted onto the vehicle surface, for each camera, multiple corner point information of the calibration image is detected from surface images of multiple time periods, and based on the multiple corner point information, the internal calibration parameters of the camera are detected; based on the surface images of all cameras in each time period, the external calibration parameters between the cameras in each time period are detected.

[0104] The calibration image is actually a calibration plate, and the calibration plate can be a QR code image or the like.

[0105] Specifically, before constructing the vehicle operation simulation environment based on the first poses and preset parameter calibration information of all cameras in all time periods, it is also necessary to obtain the preset parameter calibration information of all cameras in all time periods.

[0106] First, if Figure 4 As shown in the figure, small calibration plates (such as AprilTag) are evenly distributed on the body of the standard sample vehicle. Each small calibration plate has its own identification to ensure that each calibration plate detected by the camera has a unique identification. After the calibration plates are attached, the vehicle is sent to the tunnel defect detection system. According to the standard process, the vehicle moves and multiple calibration data are collected at equal intervals.

[0107] After image acquisition is completed, multiple corner point information of the calibration image is detected from the surface images taken by each camera in multiple time periods, and for each camera, the internal calibration parameters of each camera are calculated based on the corner point information of multiple time periods and the known size information of the calibration image. In addition, the external calibration parameters between cameras in each time period can be detected based on the surface images of all cameras in each time period.

[0108] The above steps can be implemented by the open source software colmap. The visualization results of the external parameter calibration using the open source software colmap are as follows: Figure 5 As shown in Figure 1, colmap is an open source multi-view stereo vision software widely used in computer vision and 3D reconstruction. Specifically, it can recover the geometric structure of a 3D scene from a set of 2D images and estimate the camera pose and camera intrinsic parameters. In addition, it can also be implemented by other calibration software, and its implementation method is not limited.

[0109] In other embodiments, not only can the AprilTag QR code be used to calibrate the camera using the colmap method, but other types of QR codes (such as Stag, ArUco, etc.) and other open source 3D reconstruction libraries (such as openmvg) can also be used to complete the calibration work. Even if additional cameras need to be installed later, the above process can be repeated to complete it.

[0110] It is worth noting that in the above process, the vehicle is moving while the camera remains stationary. It can be relatively understood that the vehicle is stationary while the camera is moving. When the camera collects calibration data at a certain distance, the above method will calculate the internal calibration parameters of each camera and the external calibration parameters between cameras at different times. For example, if there are 9 cameras in total and 15 sets of calibration data are collected, the internal calibration parameters of the 9 cameras and the external calibration parameters of 9x15 cameras will be calibrated.

[0111] Finally, based on the first poses of all cameras in all time periods in the real environment, the internal calibration parameters of each camera, and the external calibration parameters between cameras, the relative position between the camera and the vehicle, the position of each camera, and the relative position between cameras in the vehicle operation simulation environment in different time periods are updated to construct a vehicle operation simulation environment consistent with the real environment.

[0112] In an exemplary embodiment, if a robot arm holds a camera, hand-eye calibration (eye on hand) is also required. Specifically, it is necessary to collect the calibration plate image taken by the camera and save the position of the robot arm in the corresponding teaching pendant. The transformation relationship between the camera coordinate system and the robot arm end coordinate system and the base coordinate system is obtained by calculation, and finally the external parameters between the camera coordinate system and the reference camera coordinate system under different robot arm postures are obtained. Specifically, the external parameter relationship between cameras at different times can be calculated through the calibration results of the eye on the hand + the current posture of the robot arm at different times.

[0113] In the above embodiment, the camera parameter calibration process is simple and easy to use, and only a certain number of calibration images need to be posted on the vehicle body and the vehicle needs to pass through a tunnel once; the same is applicable when the position and number of cameras change.

[0114] In an exemplary embodiment, the vehicle operation simulation environment includes a simulated vehicle to be detected corresponding to the vehicle to be detected and a simulated camera corresponding to each camera; for each two-dimensional defect position information, the two-dimensional defect position information is converted into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle, including:

[0115] In the vehicle operation simulation environment, for each two-dimensional defect position information, rays are emitted to the simulation camera corresponding to the two-dimensional defect position information with the two-dimensional defect position information as the starting point to obtain the ray detection result; when the ray detection result indicates that the ray collides with the simulated vehicle to be detected, the collision point position information between the ray and the simulated vehicle to be detected is obtained, and the collision point position information is used as the three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle.

[0116] Specifically, Figure 6As shown, in multiple time periods, for a two-dimensional defect position information in the i-th time period, a ray is emitted to the simulation camera corresponding to the two-dimensional defect position information by using the ray tracing method, starting from the two-dimensional defect position information, to establish a ray from the two-dimensional defect simulation position information to the simulation camera, and to determine whether the ray collides with the simulation vehicle in the vehicle operation simulation environment, and to obtain a ray detection result, which includes whether the ray collides with an object in the simulation environment. In practical applications, establishing a ray from the two-dimensional defect simulation position information to the simulation camera is to establish a ray from the two-dimensional defect simulation position information to the optical center of the simulation camera.

[0117] When the ray detection result indicates that the ray collides with the simulated vehicle to be detected, the collision point between the ray and the simulated vehicle to be detected is obtained, and the collision point position information of the collision point is used as the three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle. Similarly, the above-mentioned ray tracing method can also be used in the jth time period or the kth time period to obtain the three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle. Furthermore, the three-dimensional defect position information in the vehicle coordinate system can also be visualized in the vehicle operation simulation environment. In addition, the normal information of the defect can also be obtained by the ray tracing method.

[0118] Among them, the simulation camera corresponding to the two-dimensional defect position information refers to a simulation camera that matches the camera that took the surface image containing the two-dimensional defect position information. To determine the simulation camera corresponding to the two-dimensional defect position information, it is first necessary to determine the surface image containing the two-dimensional defect position information, and then determine the actual shooting camera of the surface image, and then determine the simulation camera corresponding to the actual shooting camera.

[0119] In the above embodiment, by taking the two-dimensional defect simulation position information as the starting point, a ray is emitted to the optical center of the simulation camera, and the position information of the collision point between the ray and the simulated vehicle to be detected is obtained, thereby accurately realizing the coordinate system transformation of the two-dimensional defect position information to the three-dimensional defect position information.

[0120] In an exemplary embodiment, after associating the three-dimensional defect position information, defect type information and defect size information of each defect to obtain the vehicle surface defect detection result, the method further includes:

[0121] The three-dimensional defect position information is mapped from the vehicle coordinate system to the preset polishing robot coordinate system to obtain the polishing defect position information; based on the polishing defect position information, the polishing robot is controlled to perform the vehicle surface defect polishing operation.

[0122] Specifically, the three-dimensional defect position information is the position information in the vehicle coordinate system. When the grinding robot wants to grind the defect, it needs to obtain the position information in the base coordinate system where the grinding robot is located. Therefore, the three-dimensional defect position information needs to be mapped from the vehicle coordinate system to the grinding robot coordinate system to obtain the grinding defect position information to control the grinding robot to perform the vehicle surface defect grinding operation based on the grinding defect position information, that is, to control the grinding robot to perform the vehicle surface defect grinding operation at the grinding defect position of the vehicle to be inspected.

[0123] In an exemplary embodiment, when the grinding robot performs actual grinding, if the three-dimensional defect position information error of the defect is large or the coordinate conversion error is large, resulting in poor positioning accuracy and poor grinding effect, it is possible to consider additionally installing a precision positioning device at the end of the grinding robot, using the above-mentioned defect positioning result as the initial value, and guiding the precision positioning device to perform secondary precision positioning, thereby achieving precision guided grinding of the defect.

[0124] In the above embodiment, by performing a virtual-to-real mapping operation in a vehicle operation simulation environment, accurate three-dimensional defect position information of the defect is obtained, and the defect polishing process is performed more accurately.

[0125] In an exemplary embodiment, the vehicle operation simulation environment includes a simulated vehicle to be inspected corresponding to the vehicle to be inspected; mapping the three-dimensional defect position information from the vehicle coordinate system to the preset polishing robot coordinate system includes:

[0126] In the case of using a grinding auxiliary camera to assist in the vehicle surface defect grinding operation, obtain the placement type of the grinding auxiliary camera and the simulated vehicle point cloud information of the simulated vehicle to be detected in the vehicle operation simulation environment; based on the placement type, obtain the actual vehicle point cloud information of the vehicle photographed by the grinding auxiliary camera and the second pose between the grinding auxiliary camera coordinate system corresponding to the grinding auxiliary camera and the grinding robot coordinate system; based on the point cloud alignment result generated by the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second pose, map the three-dimensional defect position information from the vehicle coordinate system to the grinding robot coordinate system.

[0127] Specifically, multiple polishing robots can be provided, and the number of polishing robots is determined according to actual needs. Generally, considering the polishing rhythm, four polishing robots are generally provided, distributed around the vehicle, and each is responsible for a quarter of the vehicle body.

[0128] The placement types of grinding auxiliary cameras include but are not limited to: global placement and distributed placement. Global placement refers to hanging a grinding auxiliary camera directly above the grinding station, and distributed placement refers to placing a grinding auxiliary camera near each grinding robot.

[0129] Due to the different placement types of the polishing auxiliary camera, the placement position of the polishing auxiliary camera is also different, and thus, the actual vehicle point cloud information of the vehicle photographed by the polishing auxiliary camera and the second pose between the polishing auxiliary camera coordinate system corresponding to the polishing auxiliary camera and the polishing robot coordinate system are also different. In other words, it is necessary to obtain the actual vehicle point cloud information of the vehicle photographed by the polishing auxiliary camera under different placement types, as well as the second pose between the polishing auxiliary camera coordinate system corresponding to the polishing auxiliary camera and the polishing robot coordinate system, wherein the second pose between the polishing auxiliary camera coordinate system corresponding to the polishing auxiliary camera and the polishing robot coordinate system is obtained through hand-eye calibration technology. In addition, it is also necessary to obtain the simulated vehicle point cloud information of the simulated vehicle to be detected in the vehicle operation simulation environment.

[0130] Furthermore, based on the point cloud registration results generated by the actual vehicle point cloud information and the simulated vehicle point cloud information, as well as the second pose, the target defect position information is mapped from the vehicle coordinate system to the polishing robot coordinate system to obtain the polishing defect position information.

[0131] That is to say, based on the actual vehicle point cloud information and the simulated vehicle point cloud information, the point cloud registration result between the actual vehicle point cloud information of the vehicle in the actual environment during the polishing process and the simulated vehicle point cloud information in the simulation environment is obtained. Among them, the point cloud registration technology is to find the mapping relationship between different point clouds under different perspectives, and use a certain algorithm to convert different point clouds of the same target scene into the same coordinate system to form a more complete point cloud.

[0132] Finally, based on the point cloud registration results of the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second pose obtained based on the hand-eye calibration results of the polishing robot and the polishing auxiliary camera, the three-dimensional defect simulation position information is mapped from the simulated vehicle coordinate system to the polishing robot coordinate system.

[0133] Furthermore, based on the point cloud registration results of the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second posture obtained based on the hand-eye calibration results of the polishing robot and the polishing auxiliary camera, the three-dimensional defect position information is mapped from the vehicle coordinate system to the polishing robot coordinate system, including: generating point cloud registration results based on the actual vehicle point cloud information and the simulated vehicle point cloud information; mapping the three-dimensional defect position information from the vehicle coordinate system to the polishing auxiliary camera coordinate system based on the point cloud registration results; mapping the three-dimensional defect position information from the polishing auxiliary camera coordinate system to the polishing robot coordinate system based on the second posture to obtain the polishing defect position information.

[0134] In the above embodiment, through different placement types of polishing auxiliary cameras, the actual vehicle point cloud information under different placement conditions and the second pose between the polishing auxiliary camera coordinate system corresponding to the polishing auxiliary camera and the polishing robot coordinate system are obtained, and then based on the point cloud alignment results of the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second pose obtained based on the hand-eye calibration results of the polishing robot and the polishing auxiliary camera, the three-dimensional defect position information is accurately mapped from the polishing auxiliary camera coordinate system to the polishing robot coordinate system to obtain the polishing defect position information.

[0135] In an exemplary embodiment, the grinding auxiliary camera includes a global camera or a plurality of distributed cameras, the grinding auxiliary camera coordinate system includes a global coordinate system corresponding to the global camera and a distributed coordinate system corresponding to each distributed camera, the placement type includes placing the global camera directly above the grinding station or placing each distributed camera within the vicinity of the corresponding grinding robot, and based on the placement type, obtaining actual vehicle point cloud information of the vehicle photographed by the grinding auxiliary camera, and a second pose between the grinding auxiliary camera coordinate system corresponding to the grinding auxiliary camera and the grinding robot coordinate system, including:

[0136] When the placement type includes placing the global camera directly above the polishing station, the actual vehicle point cloud information of the vehicle photographed by the global camera and the second pose between the global coordinate system and the polishing robot coordinate system are obtained; when the placement type includes placing each distributed camera within the vicinity of the corresponding polishing robot, for each distributed camera, the actual vehicle point cloud information of the vehicle photographed by the distributed camera and the second pose between the distributed coordinate system and the polishing robot coordinate system are obtained.

[0137] Specifically, when the polishing auxiliary camera includes a global camera, the placement type is to place the global camera directly above the polishing station; when the polishing auxiliary camera includes multiple distributed cameras, the placement type is to place each distributed camera within the vicinity of the corresponding polishing robot.

[0138] When the placement type includes placing the global camera directly above the grinding station, the global camera captures the point cloud information of the top of the vehicle body by looking down, and uses it as the actual vehicle point cloud information. Then, multiple grinding robots are calibrated with the global camera by hand-eye calibration, and the second pose between the global coordinate system corresponding to the global camera and the coordinate system of the grinding robot is obtained based on the hand-eye calibration results. Figure 7As shown, taking the global camera including a 3D camera as an example, there are provided a grinding robot's robotic arm 1, a grinding robot's robotic arm 2, a grinding robot's robotic arm 3 and a grinding robot's robotic arm 4. In the process of mapping the three-dimensional defect position information from the vehicle coordinate system to the grinding robot coordinate system, the three-dimensional defect simulation position information in the vehicle coordinate system + the point cloud configuration result of the actual vehicle point cloud information taken by the 3D camera and the simulated vehicle point cloud information + the second pose between the 3D camera coordinate system and the grinding robot 1 / 2 / 3 / 4 coordinate system = the grinding defect position information in the grinding robot 1 / 2 / 3 / 4 coordinate system.

[0139] When the placement type includes placing each distributed camera within the vicinity of the corresponding polishing robot, a distributed camera is configured for each polishing robot, and each distributed camera captures the actual vehicle point cloud information of the vehicle. Subsequently, multiple polishing robots perform hand-eye calibration with their corresponding distributed cameras to obtain the second pose between the distributed coordinate system corresponding to the distributed camera and the coordinate system of the polishing robot. Figure 8 As shown, taking the distributed cameras including 3D camera 1, 3D camera 1, 3D camera 1 and 3D camera 1 as an example, there are provided a grinding robot's robotic arm 1, a grinding robot's robotic arm 2, a grinding robot's robotic arm 3 and a grinding robot's robotic arm 4. In the process of mapping the three-dimensional defect position information from the vehicle coordinate system to the grinding robot coordinate system, the three-dimensional defect position information in the vehicle coordinate system + the point cloud configuration result of the actual vehicle point cloud information corresponding to 3D cameras 1 / 2 / 3 / 4 and the simulated vehicle point cloud information + the second pose between 3D cameras 1 / 2 / 3 / 4 and the corresponding grinding robot 1 / 2 / 3 / 4 coordinate systems = the grinding defect position information in the grinding robot 1 / 2 / 3 / 4 coordinate system.

[0140] In the above embodiments, the grinding auxiliary cameras are placed in different layouts to accurately map all three-dimensional defect position information from the vehicle coordinate system corresponding to the simulated vehicle to the preset grinding robot coordinate system to obtain the accurate defect grinding position, thereby achieving accurate defect grinding.

[0141] In an exemplary embodiment, the specific process of the vehicle surface defect detection method of the present application is as follows:

[0142] 1. System control components:

[0143] ① The controller, usually a PLC module, is mainly responsible for the overall hardware control and sends signals to control the hardware operation.

[0144] ② Photoelectric sensor: used to detect whether the vehicle has reached the detection area. When the vehicle is in the detection area, the defect detection program is started.

[0145] ③Encoder: used to detect the forward distance of the vehicle body and determine the position of the vehicle body; every time the vehicle body moves a specified distance, it will control the camera to complete a photo.

[0146] ④ Visual module (camera): A binary stripe LED light source is used to illuminate the surface of the vehicle body to generate a quasi-binary stripe light band. At the same time, the camera collects the stripe light band image reflected on the vehicle body. Specifically, the camera and the light source are fixed on a bracket or held by a robot arm, so that the camera and the vehicle body are relatively translated. The camera collects the stripe light band image reflected on the vehicle body that moves in real time relative to the detection system.

[0147] ⑤ Processing and calculation unit: It is set in the controller and is used to process the collected images, identify and locate the defect position on the car body, determine the defect type, and calculate the defect size. Subsequently, the specific position of the defect on the car body can be determined by combining the car body CAD three-dimensional model, the car body reference position when taking pictures, and the camera calibration parameters, and mapped to the car body three-dimensional model. The defect position information on each part of the car body can be further displayed on the software interface.

[0148] 2. System calibration process:

[0149] Calibration process: small calibration plates (such as AprilTag) are evenly distributed on the body of the standard sample car. Each small calibration plate has its own ID, which ensures that each calibration plate detected by the camera has a unique corresponding identification. After the calibration plates are attached, the body is sent to the tunnel defect detection system. According to the standard process, the body moves and multiple calibration data are collected at equal intervals. That is, when it is detected that the vehicle moves a preset distance, multiple cameras are controlled to capture the surface image of the characteristic holes on the vehicle surface.

[0150] After the image acquisition is completed, the AprilTag corner points are extracted using a colmap-based method, and the intrinsic parameters of each camera and the extrinsic parameters between cameras can be calculated based on the known size information. And because the vehicle is moving, when the camera collects calibration data at a certain distance, the above method will calculate the intrinsic parameters of each camera and the extrinsic parameters between cameras at different times.

[0151] 3. Simulation environment construction:

[0152] ① Determine the position of each camera in the simulation environment: After importing the camera model in the simulation environment, set the camera's focal length, image width, image height, chip size and other parameters to be consistent with the actual camera to achieve consistent shooting effects between the virtual and real cameras. Then, based on the external reference relationship between cameras obtained by the colmap above, update the relative positions of the cameras in the simulation environment.

[0153] ② Determine the position of the camera and the car body in the simulation environment: After the car model is imported into the simulation environment, the relative position between the car model and the camera needs to be determined. This application uses a PnP-based method to obtain the relative relationship between the car model coordinate system and the camera coordinate system, thereby ensuring that the position relationship between the car model and the camera in the virtual and real environments can be deployed consistently.

[0154] Based on the known vehicle body digital model design and assembly information, the 3D hole center coordinates of the characteristic hole in the vehicle coordinate system are directly obtained, and based on the vehicle body image information actually collected by the camera, such as the surface image, the 2D pixel coordinates corresponding to the above 3D hole center are extracted using an ellipse fitting algorithm.

[0155] Based on the known camera intrinsic parameters and 3D-2D point pairs, the PnP algorithm is used to solve the posture of the camera coordinate system relative to the vehicle coordinate system, and the posture is updated to the simulation environment to achieve consistent deployment of the virtual and real environments.

[0156] 4. Defect detection process:

[0157] Based on the target detection deep learning algorithm, the defect sample data is collected in advance for annotation and training of the inference model, and the two-dimensional defect location information is accurately extracted from the surface image. In addition, the defect type information and defect size information of the defect can also be obtained through the surface image.

[0158] 5. Defect 3D mapping:

[0159] Since the camera captures data at the same time as the camera calibration, the camera's relative position to the vehicle body at a certain moment can be determined by the calibration results. The "ray tracing method" can be used, that is, to establish a ray from the two-dimensional defect position information to the camera's optical center, and calculate the collision point between the ray and the vehicle body digital model in the simulation environment, so as to obtain the three-dimensional defect position information corresponding to the defect in the vehicle coordinate system, and visualize it in the software interface. Finally, the three-dimensional defect position information, defect type information and defect size information of each defect are associated to obtain the vehicle surface defect detection results.

[0160] 6. Defect deduplication:

[0161] In actual applications, there may be multiple cameras or the same camera capturing the same defect data at different times. That is, there may be multiple observations of the same defect. The results of multiple observations can be mapped to the vehicle coordinate system and the error tolerance can be set. If the Euclidean distance between defects is less than the error tolerance, they are judged as duplicate points.

[0162] After completing the vehicle surface defect detection, control the polishing robot to perform the vehicle surface defect polishing operation. The specific process is as follows:

[0163] 1. Defect coordinate transfer:

[0164] Through the above-mentioned three-dimensional defect mapping, the three-dimensional defect position information and normal information of the defect in the vehicle coordinate system can be obtained, which now needs to be transferred to the base coordinate system of the polishing robot for guiding polishing.

[0165] Grinding station layout: You can use "global" or "distributed" layout.

[0166] The similarities between the two are as follows:

[0167] There are usually four grinding robots (taking into account the grinding rhythm), distributed around the car body, each responsible for one quarter of the car body.

[0168] The differences between the two are as follows:

[0169] ① Global layout: A 3D camera is hung directly above the polishing station to capture the point cloud information of the top of the car body from above, and then the four polishing robots are calibrated with the 3D camera. Using the registration results of the vehicle point cloud and the vehicle simulation point cloud in the simulation and the hand-eye calibration results, the defect information can be converted from the vehicle coordinate system to the polishing robot coordinate system (for example: defect A in the vehicle coordinate system + point cloud registration results + hand-eye calibration of polishing robots 1 / 2 / 3 / 4 = defect A in the base coordinate system of polishing robots 1 / 2 / 3 / 4).

[0170] ② Distributed layout: Each polishing robot is equipped with a 3D camera, which is installed near each robot to capture the point cloud information of the car body. Then, the four polishing robots perform hand-eye calibration with their respective 3D cameras. By using the registration results of the vehicle point cloud of a certain 3D camera and the vehicle simulation point cloud in the simulation and the hand-eye calibration results of the polishing robot associated with the 3D camera, the defect information can be converted from the vehicle coordinate system to the polishing robot coordinate system (for example: defect A in the vehicle coordinate system + point cloud registration results of 3D cameras 1 / 2 / 3 / 4 + hand-eye calibration of polishing robots 1 / 2 / 3 / 4 = defect A in the base coordinate system of polishing robots 1 / 2 / 3 / 4).

[0171] 2. Defect-guided grinding and precision positioning system:

[0172] Through the above method, the coordinates and normal information of the defect in the base coordinate system of the grinding robot have been obtained, which can directly guide the grinding equipment to perform grinding operations. During actual grinding, if the three-dimensional mapping error and coordinate conversion error of the defect are large, resulting in poor positioning accuracy and poor grinding effect of the defect, it can be considered to assemble an additional precision positioning device at the end of the grinding robot, using the above defect positioning result as the initial value, and guiding the precision positioning device to perform secondary precision positioning, thereby achieving precision guided grinding of the defect.

[0173] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0174] Based on the same inventive concept, the embodiment of the present application also provides a vehicle surface defect detection device for implementing the vehicle surface defect detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more vehicle surface defect detection device embodiments provided below can refer to the limitations of the vehicle surface defect detection method above, and will not be repeated here.

[0175] In an exemplary embodiment, the invention is applied to a tunnel-type defect detection system, which includes a tunnel-type detection space and a plurality of cameras, such as Fig. 9 As shown, a vehicle surface defect detection device is provided, comprising: a distance detection module 100, an image shooting module 200, a two-dimensional defect detection module 300, a three-dimensional defect generation module 400 and a defect result generation module 500, wherein:

[0176] The distance detection module 100 is used to detect the forward distance of the vehicle to be detected when the vehicle to be detected enters the tunnel-type detection space;

[0177] An image capturing module 200 is used to control a plurality of cameras to capture a surface image of a vehicle to be detected based on the advancing distance;

[0178] A two-dimensional defect detection module 300 is used to detect two-dimensional defect information of defects in multiple surface images, where the two-dimensional defect information at least includes two-dimensional defect position information, defect type information and defect size information;

[0179] A three-dimensional defect generating module 400 is used to convert the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle for each two-dimensional defect position information;

[0180] The defect result generating module 500 is used to associate the three-dimensional defect position information, defect type information and defect size information corresponding to each two-dimensional defect position information to obtain the vehicle surface defect detection result.

[0181] In one embodiment, the vehicle surface defect detection device also includes a simulation environment construction module, which is used to obtain, for each camera in each time period, the two-dimensional feature hole position information of the feature holes in the surface image taken by the camera, and the three-dimensional feature hole position information of the feature holes in the vehicle coordinate system corresponding to the vehicle to be detected, and generate the first position pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system based on the three-dimensional feature hole position information and the two-dimensional feature hole position information; construct a vehicle operation simulation environment based on all the first position poses in different time periods and the preset parameter calibration information of all cameras; the three-dimensional defect generation module 400 is also used to convert the two-dimensional defect position information into the three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle based on the constructed vehicle operation simulation environment.

[0182] In one embodiment, the preset parameter calibration information of all cameras includes internal calibration parameters of each camera and external calibration parameters between cameras. The vehicle surface defect detection device also includes a camera parameter calibration module. The camera parameter calibration module is used to detect multiple corner point information of the calibration image from surface images of multiple time periods for each camera when the calibration image is pasted to the vehicle surface, and detect the internal calibration parameters of the camera based on the multiple corner point information; based on the surface images of all cameras in each time period, detect the external calibration parameters between cameras in each time period.

[0183] In one embodiment, the vehicle operation simulation environment includes a simulated vehicle to be detected corresponding to the vehicle to be detected and a simulated camera corresponding to each camera; the three-dimensional defect generation module 400 is also used to, in the vehicle operation simulation environment, for each two-dimensional defect position information, use the two-dimensional defect position information as a starting point to emit rays to the simulated camera corresponding to the two-dimensional defect position information to obtain ray detection results; when the ray detection result indicates that the ray collides with the simulated vehicle to be detected, the collision point position information between the ray and the simulated vehicle to be detected is obtained, and the collision point position information is used as the three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle.

[0184] In one embodiment, the vehicle surface defect detection device also includes a defect polishing module, which is used to map the three-dimensional defect position information from the vehicle coordinate system to the preset polishing robot coordinate system to obtain polishing defect position information; based on the polishing defect position information, control the polishing robot to perform the vehicle surface defect polishing operation.

[0185] In one embodiment, the vehicle operation simulation environment includes a simulated vehicle to be inspected corresponding to the vehicle to be inspected; the defect grinding module is also used to obtain the placement type of the grinding auxiliary camera and the simulated vehicle point cloud information of the simulated vehicle to be inspected in the vehicle operation simulation environment when the vehicle surface defect grinding operation is assisted by the grinding auxiliary camera; based on the placement type, the actual vehicle point cloud information of the vehicle photographed by the grinding auxiliary camera and the second pose between the grinding auxiliary camera coordinate system corresponding to the grinding auxiliary camera and the grinding robot coordinate system are obtained; based on the point cloud alignment result generated by the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second pose, the three-dimensional defect position information is mapped from the vehicle coordinate system to the grinding robot coordinate system.

[0186] In one embodiment, the polishing auxiliary camera includes a global camera or multiple distributed cameras, the polishing auxiliary camera coordinate system includes a global coordinate system corresponding to the global camera and a distributed coordinate system corresponding to each distributed camera, the placement type includes placing the global camera directly above the polishing station or placing each distributed camera in the vicinity of a corresponding polishing robot, and the defect polishing module is also used to obtain actual vehicle point cloud information of the vehicle photographed by the global camera and a second pose between the global coordinate system and the polishing robot coordinate system when the placement type includes placing the global camera directly above the polishing station; when the placement type includes placing each distributed camera in the vicinity of a corresponding polishing robot, for each distributed camera, obtain actual vehicle point cloud information of the vehicle photographed by the distributed camera and a second pose between the distributed coordinate system and the polishing robot coordinate system.

[0187] Each module in the above-mentioned vehicle surface defect detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0188] In an exemplary embodiment, a vehicle surface defect detection system is also provided, the system comprising:

[0189] Photoelectric sensor, used to detect whether the vehicle to be detected enters the tunnel detection space;

[0190] A plurality of cameras for capturing surface images of the vehicle to be detected;

[0191] An encoder, used to detect the forward distance of the vehicle to be detected;

[0192] The controller is used to control an encoder to detect the forward distance of the vehicle to be detected when the vehicle to be detected enters a tunnel-type detection space; based on the forward distance, control multiple cameras to capture the surface image of the vehicle to be detected; detect two-dimensional defect information of defects in multiple surface images, the two-dimensional defect information at least including two-dimensional defect position information, defect type information and defect size information; for each two-dimensional defect position information, convert the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle; associate the three-dimensional defect position information, defect type information and defect size information corresponding to each two-dimensional defect position information to obtain a vehicle surface defect detection result.

[0193] In addition, the vehicle surface defect detection system also includes: a light source irradiation device: emitting light source onto the surface of the vehicle body.

[0194] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as surface images. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a vehicle surface defect detection method is implemented.

[0195] Those skilled in the art will understand that Fig.10 The structure shown in the figure is a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0196] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0197] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0198] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0199] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0200] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0201] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A vehicle surface defect detection method, characterized in that: Applied to a tunnel-type defect detection system, the tunnel-type defect detection system includes a tunnel-type detection space and a plurality of cameras, the method includes: When the vehicle to be detected enters the tunnel-type detection space, detecting the forward distance of the vehicle to be detected; Based on the advancing distance, controlling the multiple cameras to capture a surface image of the vehicle to be detected; Detecting two-dimensional defect information of defects in a plurality of the surface images, wherein the two-dimensional defect information at least includes two-dimensional defect position information, defect type information and defect size information; For each of the two-dimensional defect position information, convert the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle; The three-dimensional defect position information, defect type information and defect size information of each defect are associated to obtain a vehicle surface defect detection result.

2. The method according to claim 1, characterized in that Before converting the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle, the method further includes: For each camera in each time period, obtain the two-dimensional feature hole position information of the feature hole in the surface image taken by the camera, and the three-dimensional feature hole position information of the feature hole in the vehicle coordinate system corresponding to the vehicle to be detected, and generate the first position pose of the camera coordinate system corresponding to the camera relative to the vehicle coordinate system based on the three-dimensional feature hole position information and the two-dimensional feature hole position information; Building a vehicle operation simulation environment based on the preset parameter calibration information of all the first postures and all the cameras in different time periods; The converting the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle comprises: Based on the constructed vehicle operation simulation environment, the two-dimensional defect position information is converted into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle.

3. The method according to claim 2, characterized in that The preset parameter calibration information of all the cameras includes internal calibration parameters of each camera and external calibration parameters between the cameras. Before constructing the vehicle operation simulation environment based on all the first postures and the preset parameter calibration information of all the cameras in different time periods, the following is also included: When the calibration image is pasted onto the vehicle surface, for each camera, multiple corner point information of the calibration image is detected from surface images of multiple time periods, and internal calibration parameters of the camera are detected based on the multiple corner point information; Based on the surface images of all the cameras in each time period, external calibration parameters between the cameras in each time period are detected.

4. The method according to claim 2, characterized in that: The vehicle operation simulation environment includes a simulated vehicle to be detected corresponding to the vehicle to be detected and a simulated camera corresponding to each of the cameras; The converting, for each of the two-dimensional defect position information, the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle includes: In the vehicle operation simulation environment, for each of the two-dimensional defect position information, taking the two-dimensional defect position information as a starting point, emitting rays to a simulation camera corresponding to the two-dimensional defect position information to obtain a ray detection result; When the ray detection result indicates that the ray collides with the simulated vehicle to be detected, the collision point position information of the ray and the simulated vehicle to be detected is obtained, and the collision point position information is used as the three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle.

5. The method according to claim 2, characterized in that: After associating the three-dimensional defect position information, defect type information and defect size information of each defect to obtain the vehicle surface defect detection result, the method further includes: Mapping the three-dimensional defect position information from the vehicle coordinate system to a preset polishing robot coordinate system to obtain polishing defect position information; Based on the grinding defect position information, the grinding robot is controlled to perform a vehicle surface defect grinding operation.

6. The method according to claim 5, characterized in that The vehicle operation simulation environment includes a simulated vehicle to be detected corresponding to the vehicle to be detected; The mapping of the three-dimensional defect position information from the vehicle coordinate system to a preset grinding robot coordinate system includes: In the case where a grinding auxiliary camera is used to assist a vehicle surface defect grinding operation, obtaining a placement type of the grinding auxiliary camera and simulated vehicle point cloud information of a simulated vehicle to be detected in the vehicle operation simulation environment; Based on the placement type, actual vehicle point cloud information of the vehicle photographed by the polishing auxiliary camera and a second pose between a polishing auxiliary camera coordinate system corresponding to the polishing auxiliary camera and a polishing robot coordinate system are acquired; Based on the point cloud registration result generated by the actual vehicle point cloud information and the simulated vehicle point cloud information, and the second posture, the three-dimensional defect position information is mapped from the vehicle coordinate system to the polishing robot coordinate system.

7. The method according to claim 6, characterized in that The polishing auxiliary camera includes a global camera or a plurality of distributed cameras, the polishing auxiliary camera coordinate system includes a global coordinate system corresponding to the global camera and a distributed coordinate system corresponding to each of the distributed cameras, the placement type includes placing the global camera directly above the polishing station or placing each distributed camera within the vicinity of the corresponding polishing robot, and based on the placement type, obtaining actual vehicle point cloud information of the vehicle photographed by the polishing auxiliary camera, and a second posture between the polishing auxiliary camera coordinate system corresponding to the polishing auxiliary camera and the polishing robot coordinate system, including: When the placement type includes placing the global camera directly above the grinding station, obtaining actual vehicle point cloud information of the vehicle photographed by the global camera and a second pose between the global coordinate system and the grinding robot coordinate system; When the placement type includes placing each distributed camera within the vicinity of a corresponding polishing robot, for each distributed camera, actual vehicle point cloud information of the vehicle photographed by the distributed camera and a second pose between the distributed coordinate system and the polishing robot coordinate system are obtained.

8. A vehicle surface defect detection device, characterized in that: Applied to a tunnel-type defect detection system, the tunnel-type defect detection system includes a tunnel-type detection space and a plurality of cameras, and the device includes: A distance detection module, used to detect the forward distance of the vehicle to be detected when the vehicle to be detected enters the tunnel-type detection space; An image capturing module, configured to control the plurality of cameras to capture a surface image of the vehicle to be detected based on the advancing distance; A two-dimensional defect detection module, used to detect two-dimensional defect information of defects in the plurality of surface images, wherein the two-dimensional defect information at least includes two-dimensional defect position information, defect type information and defect size information; A three-dimensional defect generating module, for each of the two-dimensional defect position information, converting the two-dimensional defect position information into three-dimensional defect position information of a vehicle coordinate system corresponding to the vehicle; The defect result generation module is used to associate the three-dimensional defect position information, defect type information and defect size information corresponding to each of the two-dimensional defect position information to obtain a vehicle surface defect detection result.

9. A tunnel type defect detection system, characterized in that: The system includes a controller, an encoder, a photoelectric sensor, a tunnel-type detection space and a plurality of cameras; The photoelectric sensor is used to detect whether the vehicle to be detected enters the tunnel-type detection space; The multiple cameras are used to capture the surface image of the vehicle to be detected; The encoder is used to detect the forward distance of the vehicle to be detected; The controller is used to control the encoder to detect the forward distance of the vehicle to be detected when the photoelectric sensor detects that the vehicle to be detected enters the tunnel-type detection space; based on the forward distance, control the multiple cameras to capture the surface image of the vehicle to be detected; detect two-dimensional defect information of defects in the multiple surface images, wherein the two-dimensional defect information at least includes two-dimensional defect position information, defect type information and defect size information; For each of the two-dimensional defect position information, the two-dimensional defect position information is converted into three-dimensional defect position information of the vehicle coordinate system corresponding to the vehicle; the three-dimensional defect position information, defect type information and defect size information corresponding to each of the two-dimensional defect position information are associated to obtain a vehicle surface defect detection result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images