Body welding spot missing welding detection method and system based on three-dimensional vision and global matching

Abstract

This invention proposes a method and system for detecting missing weld points on vehicle bodies based on 3D vision and global matching, belonging to the field of image recognition technology. The method includes: determining the transformation relationship between the camera coordinate system and the vehicle body coordinate system through calibration; at each observation position, a 3D camera acquires an image I containing a frame of 2D texture and the corresponding 3D data; inputting the acquired image I into a pre-trained deep learning object detection model, outputting the bounding box information (u, v, w, h) for each weld point; for each detected weld point, calculating the set S of the weld point's 3D coordinates in the vehicle body coordinate system. det For the theoretical solder joint library S cad Each theoretical solder joint P in cad,i In set S det The invention employs a "detect first, match later" strategy, which does not rely on the precise projection of the theoretical position, thus reducing the requirements for the positioning accuracy of the production line tooling fixtures and conveying system.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and system for detecting missing welds at automotive body weld points based on three-dimensional vision and global matching. Background Technology

[0002] The manufacturing quality of a car body is crucial to ensuring the safety performance and lifespan of the entire vehicle. Resistance spot welding, as a primary joining process, is paramount, and the integrity of the weld points is of utmost importance. Missing welds—weld points that are required to be welded in the design but are not actually welded—severely weaken the structural strength of the car body, posing a significant safety hazard.

[0003] Current mainstream weld inspection methods, such as manual hammering and visual inspection, are not only inefficient and costly, but also suffer from high rates of missed detections and false positives due to the influence of inspector experience and fatigue. While traditional non-destructive testing technologies such as ultrasonic and X-ray inspections offer high precision, the equipment is expensive and the testing speed is slow, making it difficult to integrate into high-speed automated production lines for comprehensive inspection. They are typically used only for laboratory analysis or spot checks of critical weld points. With the development of machine vision technology, image processing-based automated inspection solutions have emerged. 3D vision technology, using 3D cameras, can acquire point cloud data of the vehicle body surface, thereby obtaining the three-dimensional coordinates and morphology of weld points, enabling more accurate location and analysis.

[0004] Existing 3D inspection solutions typically involve first projecting the three-dimensional coordinates of the theoretical weld point onto the captured image, and then performing feature analysis within a small area near the projected point to determine the presence of the weld point. This type of method heavily relies on the positioning accuracy of the vehicle body at the workstation. When there is a translational or rotational error between the actual and theoretical positions of the vehicle body, the projected position will deviate from the actual weld point location, resulting in inaccurate image areas and consequently, misjudgments. Summary of the Invention

[0005] In view of this, the present invention proposes a method and system for detecting missing welds at vehicle body weld points based on three-dimensional vision and global matching, in order to solve the technical problems of the existing three-dimensional vision detection schemes mentioned in the background art, which have excessively high requirements for vehicle body positioning accuracy and poor robustness.

[0006] The technical solution of this invention is implemented as follows: In a first aspect, the present invention proposes a method for detecting missing welds at automotive body weld points based on three-dimensional vision and global matching, including: The transformation relationship between the camera coordinate system and the vehicle coordinate system is determined through calibration; At each observation location, the 3D camera acquires an image I containing a frame of 2D texture and the corresponding 3D data, which includes 3D point cloud and depth map. The acquired image I is input into a pre-trained deep learning object detection model, which outputs the bounding box information (u, v, w, h) for each weld point, where (u, v) are the pixel coordinates of the center point of the bounding box, and w and h are the width and height of the bounding box, respectively. For each detected solder joint, the pixel coordinates (u) of the bounding box center point are used. c , v c The three-dimensional coordinates P of the solder joint in the camera coordinate system were calculated using the camera's intrinsic parameters. C By utilizing the transformation relationship between the camera coordinate system and the vehicle coordinate system, the three-dimensional coordinates P in the vehicle coordinate system are calculated. W This yields a set S containing the three-dimensional coordinates of all detected weld points that have been transformed into the vehicle body coordinate system. det ; For the theoretical solder joint library S cad Each theoretical solder joint P in cad,i In set S det A global match is performed. If a match is successful, the theoretical solder joint P is set. cad,i Marked as "Matched", the theoretical solder joint library S cad Solder joints not marked as "matched" are identified as defective solder joints P during the inspection. leak ; For all identified solder joint defects P leak The results are summarized and an inspection report is generated, listing the solder joints, missing solder joints, and the corresponding labeled images of the missing solder joints.

[0007] In some optional implementations, preferably, determining the transformation relationship between the camera coordinate system and the vehicle coordinate system through calibration includes: The camera's intrinsic parameter matrix K is obtained by calibrating the camera; The relative pose of the camera and the robot end effector is determined by hand-eye calibration, and the transformation matrix from the tool coordinate system F to the camera coordinate system C is obtained. F T C ; The positional relationship between the robot and the vehicle body is calibrated to obtain the transformation matrix of the vehicle body coordinate system W relative to the robot base coordinate system B. B T W ; By transforming the matrix B T W pose of the robot's end effector B T F Transformation matrix F T C and the three-dimensional coordinates P in the camera coordinate system C The coordinates P of the weld point in the vehicle body coordinate system were calculated. w .

[0008] In some optional implementations, preferably, the calibration of the camera to obtain the camera's intrinsic parameter matrix K includes: The coordinates of the principal point (u0, v0) are obtained by photographing the calibration board; By tilt factor s, and the camera's focal length f in the x and y directions x and f y Given the principal point coordinates (u0, v0), the camera's intrinsic parameter matrix K is calculated.

[0009] In some optional implementations, preferably, the step of determining the relative pose relationship between the camera and the robot end effector through hand-eye relationship calibration yields a transformation matrix from the tool coordinate system F to the camera coordinate system C. F T C ,include: A robot carrying a camera observes a fixed calibration object from different poses, based on a series of robot end-effector poses. B T F,i and the pose of the calibration object observed by the camera C T obj,i The transformation matrix is ​​obtained by solving. F T C .

[0010] In some optional implementations, preferably, the calibration of the positional relationship between the robot and the vehicle body yields the transformation matrix of the vehicle body coordinate system W relative to the robot base coordinate system B. B T W ,include: By selecting at least three fixed and easily identifiable reference points on the vehicle body, measuring the coordinates of these reference points in the vehicle body coordinate system W, and using a robot-guided camera to accurately measure the coordinates of these reference points in the robot base coordinate system B, the transformation matrix can be calculated.

[0011] In some alternative implementations, preferably, the pre-trained deep learning object detection model is trained using the following method: The training is conducted in a large number of image samples containing different types of solder joints, with more than 50,000 samples and more than 300 training cycles. The training samples include various lighting conditions, background environments, and interference samples such as reflections, oil stains, and scratches. The interference samples form an interference sample pool. In the subsequent model iteration training phase, a forced sampling mechanism is set up so that each training batch must extract a set proportion of training samples from the interference sample pool.

[0012] In some alternative implementations, preferably, the use of pixel coordinates (u) of the bounding box center point... c , vc The three-dimensional coordinates P of the solder joint in the camera coordinate system were calculated using the camera's intrinsic parameters. C ,include: Query in the depth map (u c , v c The depth value d of the location; Using the depth value d, the camera's intrinsic parameter matrix K, and the pixel coordinates of the bounding box center point (u c , v c The three-dimensional coordinates P of the solder joint in the camera coordinate system were calculated. C .

[0013] In some alternative implementations, preferably, the theoretical solder joint library S cad Each theoretical solder joint P in cad,i In set S det A global match is performed. If a match is successful, the theoretical solder joint P is set. cad,i Marked as "matched", including: For the theoretical solder joint library S cad Each theoretical solder joint p in cad,i In the detected set S of solder joints det Find the point p that is closest to it. det,j ; Calculate point p det,j With the corresponding theoretical solder point p cad,i The Euclidean distance d between them ij ; Set a reasonable distance threshold ε, if d ij If ε < , then the theoretical solder joint P is determined. cad,i With the detected solder joint p det,j Match successful.

[0014] In some alternative implementations, preferably, setting a reasonable distance threshold ε includes: The distance threshold ε is calculated using camera error σ1, robot error σ2, calibration error σ3, and manufacturing error σ4.

[0015] Secondly, this invention proposes a vehicle body weld point missing weld detection system based on three-dimensional vision and global matching, including a robot, a 3D camera, and an industrial control computer, wherein: The robot is used to move to key observation positions in various areas of the vehicle body to be inspected, following a pre-planned path. The 3D camera is mounted on the flange at the end of the robot and is used to take a picture at each key observation position, acquiring an image I containing a frame of 2D texture and the corresponding 3D data, including 3D point cloud and depth map. The industrial control computer is electrically connected to the 3D camera and is used to output the bounding box information (u, v, w, h) of each weld point from the image I input into the pre-trained deep learning object detection model, where (u, v) are the pixel coordinates of the center point of the bounding box, and w and h are the width and height of the bounding box, respectively; it is also used to calculate the three-dimensional coordinates P of the weld point in the camera coordinate system. C By utilizing the transformation relationship between the camera coordinate system and the vehicle coordinate system, the three-dimensional coordinates P in the vehicle coordinate system are calculated. W This yields a set S containing the three-dimensional coordinates of all detected weld points that have been transformed into the vehicle body coordinate system. det It is also used for the theoretical solder joint library S cad Each theoretical solder joint P in cad,i In set S det A global match is performed. If a match is successful, the theoretical solder joint P is set. cad,i Marked as "Matched", the theoretical solder joint library S cad Solder joints not marked as "matched" are identified as defective solder joints P during the inspection. leak And for all identified solder joint defects P leak The results are summarized and a test report is generated, listing the solder joints, the missing solder joints, and the corresponding labeled images of the missing solder joints.

[0016] The vehicle body weld point missing weld detection method and system based on three-dimensional vision and global matching of the present invention has the following advantages over the prior art: (1) For each detected solder joint, calculate the three-dimensional coordinates P of the solder joint in the camera coordinate system. C The three-dimensional coordinates P in the vehicle body coordinate system are obtained through coordinate transformation. W This yields a set S containing the three-dimensional coordinates of all detected weld points that have been transformed into the vehicle body coordinate system. det For the theoretical solder joint library S cad Each theoretical solder joint P in cad,i In set S det A global match is performed. If a match is successful, the theoretical solder joint P is set. cad,i Marked as "Matched", the theoretical solder joint library S cad Solder joints not marked as "matched" are identified as defective solder joints P during the inspection. leakThe "detect first, match later" strategy adopted in this invention does not rely on the precise projection of the theoretical position. Even if the car body has a translation of a few millimeters or a slight rotation, as long as the weld point is still within the camera's field of view, the deep learning model can detect it. The subsequent global matching algorithm can automatically correct this systematic deviation based on the distribution of the overall point cloud and complete the matching correctly. This greatly reduces the requirements for the positioning accuracy of the production line tooling fixtures and conveying system, and is more suitable for the actual industrial production environment. (2) The camera is calibrated to obtain the camera's intrinsic parameter matrix K; the relative pose relationship between the camera and the robot's end effector is determined through hand-eye relationship calibration, and the transformation matrix from the tool coordinate system F to the camera coordinate system C is obtained. F T C The positional relationship between the robot and the vehicle body is calibrated to obtain the transformation matrix of the vehicle body coordinate system W relative to the robot base coordinate system B. B T W ; By transforming the matrix B T W pose of the robot's end effector B T F Transformation matrix F T C and the three-dimensional coordinates P in the camera coordinate system C The coordinates P of the weld point in the vehicle body coordinate system were calculated. w The above method enables the conversion from the camera coordinate system to the robot coordinate system, and the positioning accuracy of the detected structure does not depend on the tooling fixtures and conveying system of the production line, resulting in higher accuracy. (3) By artificially increasing the proportion of high-difficulty interference samples, while ensuring that the overall distribution of the number of samples of various interference conditions such as imprints, scratches, and oil stains is balanced, the interference of pseudo solder joint features is completely suppressed from the bottom feature space, thereby improving the accuracy of the detection results. (4) By using the theoretical solder joint library S cad Each theoretical solder joint p in cad,i In the detected set S of solder joints det Find the point p that is closest to it. det,j ; Calculate point p det,j With the corresponding theoretical solder point p cad,i The Euclidean distance d between them ij Set a reasonable distance threshold ε, if d ij If ε < , then the theoretical solder joint P is determined. cad,i With the detected solder joint p det,jSuccessful matching means that multiple theoretical points may match the same detection point, but only the nearest theoretical point is matched. This "many-to-one" matching strategy can effectively handle the overall offset caused by positioning errors. Even if the vehicle body has a slight overall translation or rotation, as long as ε is set reasonably, the weld point can still be matched correctly, improving the accuracy of matching. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the vehicle body weld point missing weld detection method based on three-dimensional vision and global matching in an embodiment of the present invention. Figure 2 This is a schematic diagram of the coordinate system transformation relationship in an embodiment of the present invention; Figure 3 This is a schematic diagram of the solder joint detection and three-dimensional positioning process in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating an example of the visualization results of solder joint defects in an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of the vehicle body weld point missing weld detection system based on three-dimensional vision and global matching in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0021] In the description of the embodiments of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention.

[0022] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0023] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0024] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. Additionally, examples of various specific processes and materials are provided in this invention; however, those skilled in the art will recognize the applicability of other processes and / or the use of other materials.

[0025] The technical solution will now be explained in detail: Reference Figures 1-4 As shown, a first aspect of the present invention proposes a method for detecting missing weld points on vehicle bodies based on three-dimensional vision and global matching, comprising: Step S1: Determine the transformation relationship between the camera coordinate system and the vehicle coordinate system through calibration; Step S1 specifically includes: Step S11: Calibrate the camera to obtain the camera's intrinsic parameter matrix K; specifically including: The coordinates of the principal point (u0, v0) are obtained by photographing the calibration board; By tilt factor s, and the camera's focal length f in the x and y directions x and f y Given the principal point coordinates (u0, v0), the camera's intrinsic parameter matrix K is calculated. The formula for calculating the camera's intrinsic parameter matrix K is as follows: (1); In equation (1), f x and f y , where are the focal lengths of the camera in the x and y directions, respectively; s is the tilt factor; and (u0, v0) are the coordinates of the principal point. Step S12: Determine the relative pose relationship between the camera and the robot end effector through hand-eye calibration, and obtain the transformation matrix from the tool coordinate system F to the camera coordinate system C. F T C Specifically, it includes: A robot carrying a camera observes a fixed calibration object from different poses, based on a series of robot end-effector poses. B T F,i and the pose of the calibration object observed by the camera C T obj,i The transformation matrix is ​​obtained by solving. F T C Solve the famous equation AX = ZB, where X is the equation we are looking for. F T C The calibration process is completed using an "eye-on-hand" method. Step S13: Calibrate the positional relationship between the robot and the vehicle body to obtain the transformation matrix of the vehicle body coordinate system W relative to the robot base coordinate system B. B T W Specifically, it includes: By selecting at least three fixed and easily identifiable reference points on the vehicle body, measuring the coordinates of the reference points in the vehicle body coordinate system W, and using a robot-guided camera to accurately measure the coordinates of these reference points in the robot base coordinate system B, the transformation matrix is ​​calculated. Step S14: By transforming the matrix B T W pose of the robot's end effector B T F Transformation matrix F T C and the three-dimensional coordinates P in the camera coordinate system C The coordinates P of the weld point in the vehicle body coordinate system were calculated. w After calibration, any 3D point P measured in the camera coordinate system... C All can be converted to coordinates P in the vehicle body coordinate system using the following formula. W The coordinates P of the weld point in the vehicle body coordinate systemw The calculation formula is as follows: (2); In equation (2), B T F This represents the pose of the robot's end effector.

[0026] Step S2: At each observation location, the 3D camera acquires an image I containing a frame of 2D texture and the corresponding 3D data, which includes 3D point cloud and depth map; In step S2, the robot moves to the key observation positions of each area to be detected on the vehicle body according to the pre-planned path. At each observation position, the robot pauses and triggers the 3D camera to take a picture.

[0027] Step S3: Input the acquired image I into a pre-trained deep learning object detection model and output the bounding box information (u, v, w, h) for each weld point, where (u, v) are the pixel coordinates of the center point of the bounding box, and w and h are the width and height of the bounding box, respectively. In step S3, the deep learning object detection model can be, for example, YOLOv5, Faster R-CNN, etc. This model is trained on a large number of image samples containing various solder joints under different lighting and background conditions, and can accurately detect the location of solder joints in the image and output the bounding box information (u, v, w, h) of each solder joint. The pre-trained deep learning object detection model in step S3 is trained using the following method: The training is conducted in a large number of image samples containing different types of solder joints, with more than 50,000 samples and more than 300 training cycles. The training samples include various lighting conditions, background environments, and interference samples such as reflections, oil stains, and scratches. The interference samples form an interference sample pool. In subsequent model iteration training phases, a forced sampling mechanism is implemented, requiring each training batch to extract a predetermined proportion of training samples from the interference sample pool. After training, the deep learning object detection model can distinguish between "real solder joints" and "background interference."

[0028] Traditional two-dimensional vision methods only analyze surface features such as grayscale, shape, and texture, making them easily misled by scratches or reflections. The main feature of the deep learning model in this invention is "deep-level nonlinear visual features." Through a multi-layered neural convolutional network (CNN), the model can extract semantic information of solder joints at the pixel level and beyond. Furthermore, this model can incorporate spatial context, not only observing a single point but also learning the relative relationship between the solder joint and its surrounding material environment. Even if local interference obscures some details, the model can still make judgments based on the surrounding texture information. The model ultimately outputs the bounding box information (u, v, w, h) for each solder joint. This ability to extract local targets from the global image enables it to accurately locate solder joints that are severely affected by interference.

[0029] To improve accuracy, the average or median of all 3D points within a small neighborhood (e.g., 3 × 3 pixels) near the center of the bounding box can be taken to suppress noise and improve accuracy.

[0030] Step S4: For each detected solder joint, use the pixel coordinates (u) of the bounding box center point. c , v c The three-dimensional coordinates P of the solder joint in the camera coordinate system were calculated using the camera's intrinsic parameters. C By utilizing the transformation relationship between the camera coordinate system and the vehicle coordinate system, the three-dimensional coordinates P in the vehicle coordinate system are calculated. W This yields a set S containing the three-dimensional coordinates of all detected weld points that have been transformed into the vehicle body coordinate system. det ; In step S4, the pixel coordinates (u) of the center point of the bounding box are used. c , v c The three-dimensional coordinates P of the solder joint in the camera coordinate system were calculated using the camera's intrinsic parameters. C ,include: Query in the depth map (u c , v c The depth value d of the location; Using the depth value d, the camera's intrinsic parameter matrix K, and the pixel coordinates of the bounding box center point (u c , v c The three-dimensional coordinates P of the solder joint in the camera coordinate system were calculated. C The three-dimensional coordinates P of the solder joint in the camera coordinate system. C The calculation formula is as follows: (3); In equation (3), d is (u c , v c The depth value corresponding to the position, where K is the camera's intrinsic parameter matrix, (u c , vc ) represents the pixel coordinates of the center point of the bounding box.

[0031] Step S5: For the theoretical solder joint library S cad Each theoretical solder joint P in cad,i In set S det A global match is performed. If a match is successful, the theoretical solder joint P is set. cad,i Marked as "Matched", the theoretical solder joint library S cad Solder joints not marked as "matched" are identified as defective solder joints P during the inspection. leak ; In step S5, the system stores a theoretical coordinate database extracted from the CAD 3D design model of the vehicle model, denoted as S. cad ={p cad,1 , p cad,2 , ..., p cad,N}, where N is the theoretical total number of solder joints; the specific methods for global matching include: For the theoretical solder joint library S cad Each theoretical solder joint p in cad,i In the detected set S of solder joints det Find the point p that is closest to it. det,j This can be accelerated using data structures such as kd-trees. Calculate point p det,j With the corresponding theoretical solder point p cad,i The Euclidean distance d between them ij Euclidean distance d ij The calculation formula is as follows: (4); Set a reasonable distance threshold ε, if d ij If ε < , then the theoretical solder joint P is determined. cad,i With the detected solder joint p det,j Match successful. The distance threshold ε is typically set using a combination of theoretical calculations and empirical adjustments. 1. Theoretical calculations are based on the fact that error synthesis is not a single formula, but rather on the envelope of the overall system error. The calculation formula is as follows: (5); In equation (5), σ1 is the camera error, σ2 is the robot error, σ3 is the calibration error, and σ4 is the manufacturing error; camera and robot errors are usually in the range of 0.1mm - 0.5mm; manufacturing and positioning tolerances are usually in the range of 1mm - 2mm for body stamping deformation and workstation docking deviation. 2. Experience and process constraints for physical error prevention: The spacing between adjacent weld points is mostly more than 15mm. Setting the threshold to 3mm can ensure that the measured value of point A will not be mistakenly associated with the theoretical coordinates of point B; Industry standard: In body-in-white inspection, 3mm is a general balance point that takes into account both "tolerating reasonable deformation" and "capturing position deviation". 3. In summary, the distance threshold ε is determined by the system accuracy at its lower limit and by the solder joint spacing at its upper limit, and is ultimately determined by fine-tuning based on actual measurement data from the production line.

[0032] After traversing all theoretical solder joints, in S cad All weld points that are not yet marked as "matched" are the missed weld points found in this inspection. This "many-to-one" matching strategy, where multiple theoretical points may match the same inspection point, but only the nearest one is matched for each theoretical point, can effectively handle overall offsets caused by positioning errors. Even if the vehicle body has slight overall translation or rotation, as long as... With proper settings, most solder joints can still be correctly matched.

[0033] Step S6: For all identified solder joint defects P leak The results are summarized and an inspection report is generated, listing the solder joints, missing solder joints, and the corresponding labeled images of the missing solder joints.

[0034] In step S6, for each identified solder joint P... leak The system needs to provide intuitive visualizations. The system can perform the following operations: 1. In the theoretical solder joint database, based on the coordinates P of the missing solder joint... leak Theoretically, a reverse search should be performed to determine which frame or frames of images captured by the camera should provide the clearest view of this location. 2. Determine the three-dimensional coordinates P of the weld defect. leak (In the vehicle coordinate system), through the inverse process of coordinate transformation, it is projected back onto the two-dimensional pixel plane of the target image; 3. On the image, draw a conspicuous mark (such as a red circle or cross) at the projection location, and attach the number, coordinates and other information of the solder joint; 4. Generate a detailed inspection report from all inspection results, including the list of matched solder joints, the list of missing solder joints, and their corresponding labeled images, for quality traceability, manual review, and subsequent adjustment of process parameters.

[0035] Based on the same concept, a second aspect of the present invention, combined with... Figure 5 As shown, a vehicle body weld spot defect detection system based on 3D vision and global matching is proposed, including a robot, a 3D camera, and an industrial control computer, wherein: The robot is used to move to key observation positions in various areas of the vehicle body to be inspected, following a pre-planned path. The 3D camera is mounted on the flange at the end of the robot and is used to take a picture at each key observation position, acquiring an image I containing a frame of 2D texture and the corresponding 3D data, including 3D point cloud and depth map. The industrial control computer is electrically connected to the 3D camera and is used to output the bounding box information (u, v, w, h) of each weld point from the image I input into the pre-trained deep learning object detection model, where (u, v) are the pixel coordinates of the center point of the bounding box, and w and h are the width and height of the bounding box, respectively; it is also used to calculate the three-dimensional coordinates P of the weld point in the camera coordinate system. C By utilizing the transformation relationship between the camera coordinate system and the vehicle coordinate system, the three-dimensional coordinates P in the vehicle coordinate system are calculated. W This yields a set S containing the three-dimensional coordinates of all detected weld points that have been transformed into the vehicle body coordinate system. det It is also used for the theoretical solder joint library S cad Each theoretical solder joint P in cad,i In set S det A global match is performed. If a match is successful, the theoretical solder joint P is set. cad,i Marked as "Matched", the theoretical solder joint library S cad Solder joints not marked as "matched" are identified as defective solder joints P during the inspection. leak And for all identified solder joint defects P leak The results are summarized and a test report is generated, listing the solder joints, the missing solder joints, and the corresponding labeled images of the missing solder joints.

[0036] Taking the detection of missing welds in the side panel assembly of a certain brand SUV's body-in-white as an example, this embodiment of the invention will be specifically described. The system configuration used in this embodiment is: one KUKA KR 210 R2700 prime industrial robot, and one LMIGOcator 3D intelligent snapshot sensor (model G3-3210), which can simultaneously output 1920x1200 pixel 2D color images and point cloud data of the same scale. The control core is an Advantech industrial computer equipped with an NVIDIA RTX 3080 GPU. The specific method process is as follows: Step 1: System Calibration: In the calibration environment set up next to the production line, a 9x12 checkerboard calibration board was first used to take 20 images at different distances and angles to calculate the intrinsic parameters of the Gocator camera. Then, a calibration rod with three standard balls was fixed in the work area. The robot, carrying a camera, observed this calibration rod from 15 different poses. Hand-eye calibration was completed by solving AX=ZB, obtaining the hand-eye transformation matrix. F T CFinally, on a standard white car body, three stable feature points, such as the B-pillar and sill beam, were selected as references. Their coordinates in the vehicle body design coordinate system were measured using a CMM coordinate measuring machine. Then, a robot-guided camera was precisely aligned with these three points to collect their coordinates in the robot's base system, thereby calculating the transformation matrix between the car body and the robot base. B T W ; Step 2: Data Acquisition and Weld Joint Detection: This side panel assembly has a total of 188 theoretical weld joints. The offline programming software planned 28 robot shooting poses to ensure that all weld joints are clearly covered by at least one frame of image. On the production line, after a side panel assembly is positioned by the conveyor chain, the robot starts the automatic detection program. In the 10th shooting pose, the camera captures the rear C-pillar area of ​​the side panel. The 2D image is fed into the YOLOv5s model that has been trained on the server. The model has been trained for 300 cycles on a dataset containing more than 50,000 weld joint samples. The model completes inference within 25 milliseconds and outputs the bounding boxes of 12 weld joints in the image. Step 3: 3D Coordinate Extraction and Transformation: Taking the center pixel (854, 632) of one of the detected bounding boxes as an example, query the corresponding point cloud data to obtain its coordinates in the camera coordinate system as P. C = (150.2, -88.6, 750.1) mm; at this time, the end effector pose fed back by the robot controller is B T F ; using the already calibrated F T C and B T W Using the coordinate transformation formula, the coordinates of the weld point in the vehicle body coordinate system are calculated to be P. W = (1852.5, 950.3, 621.8) mm; repeat this process for all detected solder joints; Step 4: Global Matching and Missing Solder Identification: After capturing images in all 28 poses, the system detected and located 187 valid solder joints, forming a detection set S. det ; Combine this set with S, which contains the coordinates of 188 theoretical solder joints cad Perform matching; set the matching distance threshold ε = 3.5 mm; after the matching algorithm runs, it is found that S cad The theoretical solder joint numbered "SW-C1-07" has coordinates of (1988.4, 951.1, 834.2) mm. Within its 3.5 mm radius, no points from Sdet were found; therefore, the system determines this solder joint to be a missing solder joint. Step 5: Result Visualization: The system queries the database and finds that the optimal observation pose for the theoretical solder joint "SW-C1-07" is the 15th shooting pose. The system retrieves the 15th frame image and projects the 3D coordinates of the missing solder joint onto the image, obtaining the pixel coordinates (1024, 768). The system draws a red circle at this location and labels it "Missing Solder: SW-C1-07". The final generated report shows "Total inspections: 188, 187 qualified, 1 missing solder joint", along with all labeled images. The operator can then immediately review and process the results.

[0037] Compared with the prior art, the present invention has the following significant technical advantages and beneficial effects: (1) High precision and high robustness: It combines the powerful feature extraction capability of deep learning in 2D image recognition with the precise spatial positioning capability of 3D vision; the deep learning model can effectively overcome interference such as changes in lighting, surface reflection, and oil stains, while the 3D data ensures the accuracy of positioning. (2) Strong tolerance to positioning errors: The "detect first, match later" strategy adopted in this invention does not rely on the precise projection of the theoretical position; even if the car body has a translation of a few millimeters or a small angle of rotation, as long as the welding point is still within the camera's field of view, the deep learning model can detect it, and the subsequent global matching algorithm can automatically correct this systematic deviation according to the distribution of the overall point cloud and correctly complete the matching; this greatly reduces the requirements for the positioning accuracy of the production line tooling fixtures and conveying system, and is more suitable for the actual industrial production environment; (3) High efficiency and full automation: The entire inspection process is automatically executed by the robot without human intervention. The inspection cycle can be controlled within 45 seconds, which fully meets the requirements of the automobile production line. It can achieve 100% full inspection of every car body that comes off the line, fundamentally eliminating the problem caused by missing welds.

[0038] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting missing welds at vehicle body weld points based on 3D vision and global matching, characterized in that, include: The transformation relationship between the camera coordinate system and the vehicle coordinate system is determined through calibration; At each observation location, the 3D camera acquires an image I containing a frame of 2D texture and the corresponding 3D data, which includes 3D point cloud and depth map. The acquired image I is input into a pre-trained deep learning object detection model, which outputs the bounding box information (u, v, w, h) for each weld point, where (u, v) are the pixel coordinates of the center point of the bounding box, and w and h are the width and height of the bounding box, respectively. For each detected solder joint, the pixel coordinates (u) of the bounding box center point are used. c , v c The three-dimensional coordinates P of the solder joint in the camera coordinate system were calculated using the camera's intrinsic parameters. C By utilizing the transformation relationship between the camera coordinate system and the vehicle coordinate system, the three-dimensional coordinates P in the vehicle coordinate system are calculated. W This yields a set S containing the three-dimensional coordinates of all detected weld points that have been transformed into the vehicle body coordinate system. det ; For the theoretical solder joint library S cad Each theoretical solder joint P in cad,i In set S det A global match is performed. If a match is successful, the theoretical solder joint P is set. cad,i Marked as "Matched", the theoretical solder joint library S cad Solder joints not marked as "matched" are identified as defective solder joints P during the inspection. leak ; For all identified solder joint defects P leak The results are summarized and an inspection report is generated, listing the solder joints, missing solder joints, and the corresponding labeled images of the missing solder joints.

2. The method for detecting missing welds at vehicle body weld points based on three-dimensional vision and global matching as described in claim 1, characterized in that, The process of determining the transformation relationship between the camera coordinate system and the vehicle coordinate system through calibration includes: The camera's intrinsic parameter matrix K is obtained by calibrating the camera; The relative pose of the camera and the robot end effector is determined by hand-eye calibration, and the transformation matrix from the tool coordinate system F to the camera coordinate system C is obtained. F T C ; The positional relationship between the robot and the vehicle body is calibrated to obtain the transformation matrix of the vehicle body coordinate system W relative to the robot base coordinate system B. B T W ; By transforming the matrix B T W pose of the robot's end effector B T F Transformation matrix F T C and the three-dimensional coordinates P in the camera coordinate system C The coordinates P of the weld point in the vehicle body coordinate system were calculated. w .

3. The method for detecting missing welds at vehicle body weld points based on three-dimensional vision and global matching as described in claim 2, characterized in that, The process of calibrating the camera to obtain the camera's intrinsic parameter matrix K includes: The coordinates of the principal point (u0, v0) are obtained by photographing the calibration board; By tilt factor s, and the camera's focal length f in the x and y directions x and f y Given the principal point coordinates (u0, v0), the camera's intrinsic parameter matrix K is calculated.

4. The method for detecting missing welds at vehicle body weld points based on three-dimensional vision and global matching as described in claim 2, characterized in that, The relative pose relationship between the camera and the robot end effector is determined through hand-eye relationship calibration, resulting in a transformation matrix from the tool coordinate system F to the camera coordinate system C. F T C ,include: A robot carrying a camera observes a fixed calibration object from different poses, based on a series of robot end-effector poses. B T F,i and the pose of the calibration object observed by the camera C T obj,i The transformation matrix is ​​obtained by solving. F T C .

5. The method for detecting missing welds at vehicle body weld points based on three-dimensional vision and global matching as described in claim 2, characterized in that, The positional relationship between the robot and the vehicle body is calibrated to obtain the transformation matrix of the vehicle body coordinate system W relative to the robot base coordinate system B. B T W ,include: By selecting at least three fixed and easily identifiable reference points on the vehicle body, measuring the coordinates of these reference points in the vehicle body coordinate system W, and using a robot-guided camera to accurately measure the coordinates of these reference points in the robot base coordinate system B, the transformation matrix can be calculated.

6. The method for detecting missing weld points on vehicle bodies based on three-dimensional vision and global matching as described in claim 1, characterized in that, The pre-trained deep learning object detection model is trained using the following method: The training is conducted in a large number of image samples containing different types of solder joints, with more than 50,000 samples and more than 300 training cycles. The training samples include various lighting conditions, background environments, and interference samples such as reflections, oil stains, and scratches. The interference samples form an interference sample pool. In the subsequent model iteration training phase, a forced sampling mechanism is set up so that each training batch must extract a set proportion of training samples from the interference sample pool.

7. The method for detecting missing welds at vehicle body weld points based on three-dimensional vision and global matching as described in claim 1, characterized in that, The use of pixel coordinates (u) of the center point of the bounding box c , v c The three-dimensional coordinates P of the solder joint in the camera coordinate system were calculated using the camera's intrinsic parameters. C ,include: Query in the depth map (u c , v c The depth value d of the location; Using the depth value d, the camera's intrinsic parameter matrix K, and the pixel coordinates of the bounding box center point (u c , v c The three-dimensional coordinates P of the solder joint in the camera coordinate system were calculated. C .

8. The method for detecting missing welds at vehicle body weld points based on three-dimensional vision and global matching as described in claim 1, characterized in that, The theoretical solder joint library S cad Each theoretical solder joint P in cad,i In set S det A global match is performed. If a match is successful, the theoretical solder joint P is set. cad,i Marked as "matched", including: For the theoretical solder joint library S cad Each theoretical solder joint p in cad,i In the detected set S of solder joints det Find the point p that is closest to it. det,j ; Calculate point p det,j With the corresponding theoretical solder point p cad,i The Euclidean distance d between them ij ; Set a reasonable distance threshold ε, if d ij If ε < , then the theoretical solder joint P is determined. cad,i With the detected solder joint p det,j Match successful.

9. The method for detecting missing welds at vehicle body weld points based on three-dimensional vision and global matching as described in claim 8, characterized in that, The setting of a reasonable distance threshold ε includes: The distance threshold ε is calculated using camera error σ1, robot error σ2, calibration error σ3, and manufacturing error σ4.

10. A vehicle body weld spot missing weld detection system based on 3D vision and global matching, characterized in that, This includes robots, 3D cameras, and industrial control computers, among which: The robot is used to move to key observation positions in various areas of the vehicle body to be inspected, following a pre-planned path. The 3D camera is mounted on the flange at the end of the robot and is used to take a picture at each key observation position, acquiring an image I containing a frame of 2D texture and the corresponding 3D data, including 3D point cloud and depth map. The industrial control computer is electrically connected to the 3D camera and is used to output the bounding box information (u, v, w, h) of each weld point from the image I input into the pre-trained deep learning object detection model, where (u, v) are the pixel coordinates of the center point of the bounding box, and w and h are the width and height of the bounding box, respectively; it is also used to calculate the three-dimensional coordinates P of the weld point in the camera coordinate system. C By utilizing the transformation relationship between the camera coordinate system and the vehicle coordinate system, the three-dimensional coordinates P in the vehicle coordinate system are calculated. W This yields a set S containing the three-dimensional coordinates of all detected weld points that have been transformed into the vehicle body coordinate system. det It is also used for the theoretical solder joint library S cad Each theoretical solder joint P in cad,i In set S det A global match is performed. If a match is successful, the theoretical solder joint P is set. cad,i Marked as "Matched", the theoretical solder joint library S cad Solder joints not marked as "matched" are identified as defective solder joints P during the inspection. leak And for all identified solder joint defects P leak The results are summarized and a test report is generated, listing the solder joints, the missing solder joints, and the corresponding labeled images of the missing solder joints.

Images