Method and device for detecting and positioning weld surface defects based on pseudo binocular vision

Abstract

The invention belongs to the technical field of nondestructive testing, and particularly relates to a method and a device for detecting and positioning weld surface defects based on pseudo binocular vision, and the method comprises the following steps: preprocessing a data set acquired by a CCD (Charge Coupled Device) camera; constructing a pseudo binocular stereoscopic vision defect detection and positioning model based on a single CCD camera; identifying and classifying defects on the surface of the welding seam; calibrating the coordinates of the defect center point by using an algorithm; according to the method, a pseudo binocular vision model based on a single camera is innovatively provided, compared with a traditional binocular vision system, the complexity of camera synchronization and calibration is reduced, the position and the angle of the camera can be adjusted according to needs, and the accuracy of calibration is improved. And therefore, different application scenes and requirements can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of non-destructive testing for weld surface defects, and particularly relates to a method for detecting and locating weld surface defects based on pseudo-binocular vision, and also relates to a device for detecting and locating weld surface defects. Background Art

[0002] Weld surface defects refer to various defects that occur on the surface of the weld during the welding process. These defects not only affect the appearance of the weld, but may also reduce the mechanical properties of the weld and the safety of the structure. Weld surface defects mainly include pores, inclusions, weld beads, undercuts, depressions, surface cracks, etc. In order to ensure the quality and qualification rate of products, the detection of weld surface defects is an essential process in the production process of related products.

[0003] Due to factors such as the physical condition and professional level of workers, traditional manual inspection is very likely to produce inconsistent results when judging defects, resulting in insufficient data accuracy and low detection efficiency, and it can no longer meet the requirements of modern equipment. Binocular vision-based detection requires the establishment of an accurate binocular vision system, which has a high cost, complex equipment configuration, and high computational complexity and amount of calculation. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for detecting and locating weld surface defects based on pseudo-binocular vision, which can quickly detect and locate weld surface defects without being restricted by materials, and perform real-time and accurate intelligent detection of weld surface defects.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for detecting and locating weld surface defects based on pseudo-binocular vision includes the following steps:

[0007] Step 1: Use a single CCD camera to take pictures of the weld surface on the left and right sides of the weld once respectively to obtain two left and right images;

[0008] Step 2: Input the left and right images into the YOLOv4 network to identify and classify weld surface defects, and extract the coordinates of the defect center points on the left and right images;

[0009] Step 3: Use the NCC stereo matching algorithm to find the corresponding points on the right image for the defect center points on the left image; use the NCC stereo matching algorithm to find the corresponding points on the left image for the defect center points on the right image;

[0010] Step 4: Take the midpoint between the defect center point on the left image and the corresponding point on the right image as the calibration point for the defect center point on the left image; take the midpoint between the defect center point on the right image and the corresponding point on the left image as the calibration point for the defect center point on the right image;

[0011] Step 5: Substitute the coordinates of the defect center points on the calibrated left and right images into the set coordinate conversion relationship to obtain the three-dimensional coordinates of the defect center points in the robot base coordinate system.

[0012] As a more optimal technical solution of the present invention, the input data set of the YOLOv4 network is an image obtained by using a single camera to photograph the weld surface; the backbone part of the YOLOv4 network uses the Darknet53 network framework as the network extraction backbone for feature extraction, and an attention mechanism is added to the residual blocks of the Darknet53 network framework; the spatial pyramid pooling module SPP is used to increase the reception range of the backbone features, and the top-down FPN feature pyramid is used to improve the feature extraction ability.

[0013] As a more optimal technical solution of the present invention, the calibration of the defect center point in Step 3 is specifically as follows:

[0014] Find the corresponding point Q on the right image for the defect center point Q1 on the left image through the NCC stereo matching algorithm 12 , find the corresponding point Q on the left image for the defect center point Q2 on the right image through the NCC stereo matching algorithm 21 , based on the NCC-based stereo matching method, its expression is as follows:

[0015]

[0016] In the formula: S(s,t) is a two-dimensional isomorphic region of size M×N, and T(s,t) is a template region of size M×N, is the average pixel value within S(t, t), is the average pixel value of T(s, t);

[0017] Find the midpoint between Q1 and Q 21 to be Q 10 Find the midpoint between Q2 and Q 12 to be Q 20 , obtain the coordinates of point Q 10 and the coordinates of point Q 20 , and the calculation formula is as follows:

[0018]

[0019] As a more optimal technical solution of the present invention, the coordinate conversion algorithm in Step 5 is specifically as follows:

[0020] Substitute the pixel coordinates (u1, v1) of the calibrated center point on the left image and the pixel coordinates (u2, v2) of the right image into the following formula to obtain the three-dimensional coordinates (x B , y B , zB );

[0021]

[0022] Another object of the present invention is to provide a detection and positioning system for weld surface defects based on pseudo-binocular vision, including

[0023] Image acquisition module: Take pictures of the weld surface once on each side of the weld to obtain two left and right images;

[0024] Target recognition and positioning module: Transmit the two captured images to the trained YOLOv4 network, and output the coordinates of the defect center points on the extracted left and right images;

[0025] Target calibration module: Take the midpoint between the defect center point on the left image and the corresponding point on the right image as the calibration point of the defect center point on the left image, and take the midpoint between the defect center point on the right image and the corresponding point on the left image as the calibration point of the defect center point on the right image;

[0026] Target calculation module: Substitute the calibrated defect center point coordinates into the set coordinate conversion relationship to obtain the three-dimensional coordinates of the defect center point in the robot base coordinate system, and realize the spatial positioning of the defect.

[0027] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, it realizes the steps of a method for detecting and positioning weld surface defects based on pseudo-binocular vision.

[0028] The beneficial effects are as follows:

[0029] The present invention constructs a deep learning weld surface defect detection algorithm based on a pseudo-binocular vision model using improved YOLOv4, and realizes the rapid recognition, detection and positioning of weld surface defects. Since the traditional binocular vision system is restricted by its lack of flexibility in practical applications, the present invention proposes the use of pseudo-binocular vision with a single camera, which reduces the complexity of camera synchronization and calibration, and can also adjust the position and angle of the camera according to different application scenarios and requirements. The design is simplified and it is easier to integrate with other systems or devices, and the cost is low. The present invention calibrates the pseudo-binocular stereo vision defect detection and positioning model using the set solution algorithm, substitutes the coordinates of the defect center points on the two images obtained by the calibrated pseudo-binocular vision into the pseudo-binocular vision model, and performs coordinate conversion based on the set coordinate conversion algorithm to improve the robustness and accuracy of the matching, and realizes the spatial positioning of the defect. Description of the Drawings

[0030] The accompanying drawings described herein are used to provide a further understanding of the present invention and form a part of this application. The schematic examples and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0031] Figure 1 It is a schematic structural diagram of the pseudo-binocular vision model of the present invention.

[0032] Figure 2 It is a step block diagram of a method for detecting and positioning weld surface defects based on pseudo-binocular vision of the present invention.

[0033] Figure 3 It is the defect recognition effect of the present invention Figure 1 , Figure A is the left figure, Figure B is the right figure, C is a pit, and D is a weld tumor.

[0034] Figure 4 It is the defect recognition effect of the present invention Figure 2 , Figure A is the left figure, Figure B is the right figure, and D is a weld tumor.

[0035] Figure 5 It is a schematic diagram of defect center point calibration on two images of the present invention. Figure A is the left figure, Figure B is the right figure, and the defects are within the green frame lines.

[0036] Figure 6 It is a schematic diagram of spatial point coordinate transformation of the present invention.

[0037] In the figure: 1. Rail; 2. Robot mounting base; 3. Robot; 4. CCD camera; 5. Computer; 6. Industrial control computer; 7. Weld; 8. Defect. Detailed implementation manners

[0038] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0039] See Figures 1 to 6As shown in the figure, the method for detecting and locating weld surface defects based on pseudo-binocular vision provided by the present invention first prepares for quality inspection, uses a single CCD camera to take pictures, preprocesses the data set collected by the camera, and makes the processed weld surface defect images into a new data set; then constructs a pseudo-binocular stereo vision defect detection and location model based on a single CCD camera; then classifies and identifies image defects, and completes the identification, classification, location and extraction of the pixel coordinates of the defect center point on the camera image through the improved YOLOv4 algorithm task; then calibrates the defect center points on the two images, and calibrates the defect center point coordinates on the two images obtained by pseudo-binocular vision by using the improved template matching algorithm; then performs three-dimensional coordinate transformation, calibrates the pseudo-binocular stereo vision defect detection and location model by using the set solution algorithm, brings the calibrated defect center point coordinates on the two images obtained by pseudo-binocular vision into the pseudo-binocular stereo vision defect detection and location model, and performs coordinate transformation based on the set coordinate transformation algorithm to achieve the spatial location of the defect; finally, the quality inspection is completed and the robot returns to the origin.

[0040] The above-mentioned preprocessing is to transform the image brightness and contrast, so that the gray value of the weld pixel points tends to 255, and the gray value of the background pixel points tends to 0, so as to increase the difference between the gray value of the weld pixel points and the gray value of the background pixel points within the ROI. In addition, perform image morphological processing on the image, first erode the image and then dilate the image.

[0041] See Figure 1 As shown in the figure, the pseudo-binocular vision model of the present invention mainly includes a track 1, a robot mounting base 2, a robot 3, a CCD camera 4, a computer 5 and an industrial control computer 6. The robot mounting base 2 is slidably connected to the track 1, and the robot 3 fixed on the robot mounting base 2 holds the CCD camera 4 to scan and inspect the weld surface. The CCD camera 4 transmits the quality inspection information to the computer 5 in real time through a data cable, and the computer 5 transmits the scanning information to the industrial control computer 6 to analyze the weld surface state in real time and detect and locate defects such as pores and weld beads on the weld surface.

[0042] See Figures 2 to 6 As shown in the figure, the method for detecting and locating weld surface defects based on pseudo-binocular vision provided by the present invention first prepares for quality inspection, then constructs a pseudo-binocular stereo vision defect detection and location model based on a single CCD camera 4, then classifies and identifies image defects, then calibrates the defect center point coordinates on the two images based on the set template matching algorithm, and finally performs coordinate transformation based on the set coordinate transformation algorithm to achieve the spatial location of the defect. Specifically as follows:

[0043] S1. Quality inspection preparation: Use the robotic arm of robot 3 to clamp the CCD camera 4 above the weld surface, turn on the CCD camera 4, and prepare for quality inspection.

[0044] S2. Build a pseudo-binocular stereo vision defect detection and localization model based on a single CCD camera 4:

[0045] Take two shots along the 45° inclined direction of the weld by a single CCD camera 4 to obtain two stereo images, and transmit them to the detection platform, and then establish a pseudo-binocular vision model. The hardware composition of the pseudo-binocular vision model mainly includes a CCD camera 4, a track 1, a computer 5, a robot mount 2, a robot 3, and an industrial control computer 6.

[0046] S2.1. Calibrate the pseudo-binocular stereo vision defect detection and localization model using the overdetermined equation solving algorithm based on Chebyshev solution; the following coordinate transformation process is the calibration process.

[0047] S2.2. Substitute the coordinates of the defect center points on the two images obtained by the calibrated pseudo-binocular vision into the model, and perform coordinate transformation based on the set coordinate transformation algorithm. Specifically:

[0048] S2.2.1. Denote O1U1V1, O C1 X C1 Y C1 Z C1 , O H1 X H1 Y H1 Z H1 to represent the left image pixel coordinate system, the camera coordinate system, and the world coordinate system respectively. Denote O2U2V2, O C2 X C2 Y C2 Z C2 , O H2 X H2 Y H2 Z H2 to represent the right image pixel coordinate system, the camera coordinate system, and the world coordinate system respectively. O B X B Y B Z B is the robot base coordinate system.

[0049] S2.2.2. Taking the left image as the research object, the coordinate relationship between any pixel in the image in the image pixel coordinate system and the image physical coordinate system can be expressed as:

[0050]

[0051] According to the central perspective projection rule and the similar triangle ratio transformation, it can be obtained that:

[0052]

[0053] where d x , d y , f, u0, v0 are the internal parameters of the camera.

[0054] In actual situations, the non - parallelism between the lens itself and the camera imaging plane and the lens shape will cause tangential distortion and radial distortion. Taking (x E1 , y E1 ) and (x s1 , y s1 ) as examples, the distortion formulas are as follows:

[0055]

[0056] where k1, k2, k3, t1, t2 are five distortion parameters.

[0057]

[0058] Through rigid - body transformation, the conversion relationship between the world coordinate system and the camera coordinate system is:

[0059]

[0060] O B1 X B1 Y B1 Z B1 The relationship between the coordinates of point E in O H1 X H1 Y H1 Z H1 and the coordinates in O

[0061]

[0062] O W X W Y W Z W is: B2 X B2 Y B2 Z B2 is:

[0063]

[0064] Similarly, it can be obtained that:

[0065]

[0066] S2.2.3. Substitute the pixel coordinates (u1, v1) of point E in the left image and the pixel coordinates (u2, v2) of point E in the right image into the following formula:

[0067]

[0068] Obtain the three-dimensional coordinates (x B , y B , z B ) of point E in the robot base coordinate system.

[0069] The transformation of the coordinates of a point in space is realized through the above formula.

[0070] S3. Classify and identify image defects:

[0071] S3.1. Define the targets for classification and identification detection, including surface pores, undercut, overlap, and pits;

[0072] S3.2. Improve the YOLOv4 algorithm, and propose an automatic detection scheme for weld surface defects based on deep learning of YOLOv4. Through the improved YOLOv4 algorithm, the recognition and classification of weld surface defects on the camera image are completed, and the pixel coordinates of the defect center point are located and extracted. Specifically:

[0073] S3.2.1. Use the residual structure on the backbone network, improve DarkNet53, add a channel attention mechanism to its residual block, modify the activation function and BN layer, and add an SPP structure to each branch of FPN;

[0074] S3.2.2. Feature fusion of the FPN feature pyramid network, integrate deep features into shallow features, the top-level features are fused with low-level features through upsampling, and each layer is independently predicted to retain small target information;

[0075] S3.2.3. By setting relevant parameters, modify the anchor points to predicted bounding boxes, and remove all unnecessary bounding boxes through methods such as setting thresholds and NMS;

[0076] S3.2.4. Shoot with a single CCD camera, preprocess the dataset collected by the camera, make the processed weld surface defect images into a new dataset, and input it for training.

[0077] Through the improved YOLOv4 algorithm, the recognition, classification of weld surface defects on the camera image are completed, and the pixel coordinates of the defect center point are located and extracted.

[0078] S4. Calibrate the defect center point coordinates on two images based on the set template matching algorithm:

[0079] S4.1. Identify the left and right images through the improved YOLOv4 algorithm. The center point identified in the left image is Q1, and the center point identified in the right image is Q2;

[0080] S4.2. Establish the calibration principle for the defect center point, specifically as follows:

[0081] S4.2.1. For the defect center point Q1 recognized by the improved YOLOv4 algorithm on the left image, find the corresponding point Q on the right image through the NCC stereo matching algorithm 12 . For the defect center point Q2 recognized by the improved YOLOv4 algorithm on the right image, find the corresponding point Q on the left image through the NCC stereo matching algorithm 21 . Based on the NCC-based stereo matching method, its expression is as follows:

[0082]

[0083] In the formula: S(s, t) is a two-dimensional isomorphic region of size M×N, and T(s, t) is a template region of size M×N, is the average pixel value within S(s, t), is the average pixel value of T(s, t).

[0084] S4.2.2. Calculate the center point between Q1 and Q 21 as Q 10 . Calculate the center point between Q2 and Q 12 as Q 20 . The calculation formula is as follows:

[0085]

[0086] The coordinates of point Q 10 and the coordinates of point Q 20 can be obtained through the above formula.

[0087] S5. Perform coordinate transformation based on the set coordinate transformation algorithm to achieve the spatial positioning of the defect:

[0088] Take Q 10 as the calibration point of Q1, and take Q 20 as the calibration point of Q2, and substitute them into step S2.2.3. Then, the coordinate values in the world coordinate system can be obtained, thereby achieving the spatial positioning of the defect.

[0089] S6. The quality inspection is completed.

[0090] Turn off the camera, and the robot returns to the origin, waiting for the next quality inspection.

[0091] The present invention also provides a detection and positioning system for weld surface defects based on pseudo-binocular vision, including the following:

[0092] Image acquisition module: Take a picture of the weld surface on both the left and right sides of the weld once to obtain two left and right images;

[0093] Target recognition and positioning module: Transmit the two captured images to the trained YOLOv4 network, and output the coordinates of the defect center points on the left and right images extracted;

[0094] Target calibration module: Take the midpoint of the defect center point on the left image and the corresponding point on the right image as the calibration point of the defect center point on the left image, and take the midpoint of the defect center point on the right image and the corresponding point on the left image as the calibration point of the defect center point on the right image;

[0095] Target calculation module: Substitute the calibrated coordinates of the defect center point into the set coordinate conversion relationship to obtain the three-dimensional coordinates of the defect center point in the robot base coordinate system, and realize the spatial positioning of the defect.

[0096] The system provided by the present invention can realize the detection and positioning of defects such as pores, pits, undercuts, and overlaps on the weld surface, and is applicable to materials such as steel, iron, and magnesium alloy. The present invention innovatively proposes a pseudo-binocular vision model based on a single camera. Compared with the traditional binocular vision system, it reduces the complexity of camera synchronization and calibration, and can adjust the position and angle of the camera according to needs to adapt to different application scenarios and requirements.

[0097] Those skilled in the art know that in addition to implementing the system and its various devices, modules, and units provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system and its various devices, modules, and units provided by the present invention to be in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers to achieve the same function. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be regarded as the structure within the hardware component; the devices, modules, and units for implementing various functions can also be regarded as both software modules for implementing the method and the structure within the hardware component.

[0098] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for detecting and locating weld surface defects based on pseudo binocular vision, characterized in that: The steps include: Step 1: Use a single CCD camera to photograph the weld surface once on both sides of the weld to obtain two left and right images; Step 2: Input the left and right images into the YOLOv4 network to identify and classify weld surface defects, and extract the coordinates of the defect center points on the left and right images; Step 3: The center point of the defect on the left image is used to find the corresponding point on the right image through the NCC stereo matching algorithm; the center point of the defect on the right image is used to find the corresponding point on the left image through the NCC stereo matching algorithm; Step 4: The midpoint between the defect center point on the left image and the corresponding point on the right image is used as the calibration point for the defect center point on the left image; the midpoint between the defect center point on the right image and the corresponding point on the left image is used as the calibration point for the defect center point on the right image; Step 5: Substitute the coordinates of the defect center point on the calibrated left and right images into the set coordinate transformation relationship to obtain the three-dimensional coordinates of the defect center point in the robot base coordinate system.

2. The method for detecting and locating weld surface defects based on pseudo binocular vision according to claim 1, characterized in that: The YOLOv4 network has an input data set of images taken of the weld surface by a single camera; the YOLOv4 network backbone uses the Darknet53 network framework as the network extraction backbone for feature extraction, and an attention mechanism is added to the residual block of the Darknet53 network framework; the spatial pyramid pooling module SPP is used to increase the receiving range of the backbone features, and the top-down FPN feature pyramid is used to improve the feature extraction capability.

3. The method for detecting and locating weld surface defects based on pseudo binocular vision according to claim 1, characterized in that: The calibration of the defect center point in step 3 is specifically as follows: The defect center point Q1 on the left image is found through the NCC stereo matching algorithm to find the corresponding point Q on the right image. 12 The defect center point Q2 on the right image is found through the NCC stereo matching algorithm to find the corresponding point Q on the left image. 21 , the stereo matching method based on NCC is expressed as follows: Where: S(s,t) is a two-dimensional isomorphic region of size M×N, T(s,t) is a template region of size M×N, is the average pixel value in S(s,t), is the average pixel value of T(s,t); Find Q1 and Q 21 The midpoint between 10 , find Q2 and Q 12 The midpoint between 20 , get points Q 10 The coordinates of point Q 20 The coordinates of are obtained by the following formula:

4. The method for detecting and locating weld surface defects based on pseudo binocular vision according to claim 1, characterized in that: The coordinate transformation algorithm in step 5 is specifically: Substitute the pixel coordinates (u1, v1) of the calibrated center point in the left image and the pixel coordinates (u2, v2) of the right image into the following formula to obtain the three-dimensional coordinates (x B ,y B ,z B ); 5. A detection and positioning system for weld surface defects based on pseudo binocular vision, characterized in that: include: Image acquisition module: take pictures of the weld surface on both sides of the weld to obtain two images; Target recognition and positioning module: The two images are transmitted to the trained YOLOv4 network, and the coordinates of the defect center points on the left and right images are output; Target calibration module: the midpoint between the defect center point on the left image and the corresponding point on the right image is used as the calibration point of the defect center point on the left image, and the midpoint between the defect center point on the right image and the corresponding point on the left image is used as the calibration point of the defect center point on the right image; Target solution module: Substitute the calibrated defect center point coordinates into the set coordinate transformation relationship to obtain the three-dimensional coordinates of the defect center point in the robot base coordinate system to achieve spatial positioning of the defect.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the steps of the method for detecting and locating weld surface defects based on pseudo binocular vision as claimed in claim 1 are implemented.

Images