Weld joint identifying and tracking method based on multi-modal information, automatic welding device and computer equipment

Abstract

The invention belongs to the technical field of weld joint identification and tracking, and particularly relates to a weld joint identification and tracking method based on multi-modal information, an automatic welding device and computer equipment. The method comprises the following steps: acquiring an RGB image and a depth image of a to-be-welded workpiece, and inputting the acquired RGB image and depth image into a trained weld joint identification model to obtain an identification result of a weld joint; the weld joint recognition model comprises a feature extraction module and a feature fusion module, the feature extraction module is used for extracting RGB features of different levels and depth features of different levels, and the feature fusion module is used for fusing the RGB features and the depth features of the same level to obtain multi-modal fusion features of the level; and the multi-modal fusion features of different levels are spliced and output. The welding seam can be accurately recognized, and smoke and light interference in the welding process is avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of weld seam identification and tracking, and specifically relates to a weld seam identification and tracking method based on multimodal information guidance, an automatic welding device and a computer equipment. Background Art

[0002] As a common metal connection method, welding is widely used in various fields of manufacturing. The quality of welding directly affects the quality and reliability of products. With the improvement of industrial automation level, welding robots are gradually replacing manual welding methods. Weld seam recognition plays a vital role in automatic welding. If the weld seam recognition is inaccurate, it is difficult to improve the welding quality. The weld seam recognition method in the prior art mainly relies on the visual system. The image or video of the welding process is captured by the camera, and then the image processing technology is used to analyze and process it to obtain the weld seam recognition result. However, a single visual image is often affected by changes in ambient light and smoke generated during welding, resulting in low weld seam recognition accuracy, which leads to low welding quality of the automatic welding device. Summary of the invention

[0003] The purpose of the present invention is to provide a weld recognition and tracking method, an automatic welding device and a computer device based on multimodal information, so as to solve the technical problem that a single visual image is easily affected by light changes and smoke, resulting in low welding accuracy, thereby affecting the welding quality.

[0004] In order to solve the above technical problems, the present invention provides a weld identification method based on multimodal information, comprising the following steps: Acquire an RGB image and a depth image of the workpiece to be welded, and input the acquired RGB image and depth image into a trained weld recognition model to obtain a weld recognition result; The weld recognition model includes a feature extraction module and a feature fusion module. The feature extraction module is used to extract RGB features of different levels and depth features of different levels. The feature fusion module is used to fuse the RGB features and depth features of the same level to obtain the multimodal fusion features of the level, and splice the multimodal fusion features of different levels for output.

[0005] Furthermore, the feature fusion module fuses the RGB features and depth features of the same level in a process including: converting the RGB features of the level into RGB query features, RGB key features and RGB value features, and converting the depth features of the level into depth query features, depth key features and depth value features; after element-by-element multiplication of the RGB query features and the depth key features of the level, obtaining the attention weights through a softmax operation, performing a product-sum operation on the attention weights and the depth value features of the level to obtain the first fused features of the level, connecting the first fused features of the level with the RGB features of the level to obtain the enhanced RGB features of the level; after element-by-element multiplication of the depth query features and the RGB key features of the level, obtaining the attention weights of the products through a softmax operation, performing a product-sum operation on the attention weights and the values ​​of the RGB features of the level to obtain the second fused features of the level, merging the second fused features of the level with the depth features to obtain the enhanced depth features of the level; fusing the enhanced RGB features with the enhanced depth features to obtain the multimodal fusion features.

[0006] Furthermore, the feature extraction module includes two convolutional neural networks of ResNet architectures, wherein the residual blocks connected in sequence in one convolutional neural network of ResNet architecture respectively output RGB features of different levels, and the residual blocks connected in sequence in the other convolutional neural network of ResNet architecture respectively output depth features of different levels.

[0007] Furthermore, different levels of multimodal fusion features include: deep multimodal fusion features, middle multimodal fusion features, and shallow multimodal fusion features; Correspondingly, the process of splicing multimodal fusion features at different levels includes: splicing deep multimodal fusion features and middle multimodal fusion features, and splicing the splicing results with shallow multimodal fusion features for output.

[0008] The present invention is an improved invention creation, and its beneficial effects are as follows: the present invention inputs two visual images, RGB image and depth image, into a weld recognition model to obtain multimodal fusion features, and obtains weld recognition results based on multimodal features, thereby avoiding the problem that the use of a single visual image is easily affected by light changes and smoke interference during the welding process, thereby affecting the weld recognition accuracy, and can improve the welding quality. Specifically, the weld recognition model includes a feature extraction module and a feature fusion module, the feature extraction module is used to extract RGB features of different levels and depth features of different levels, the feature fusion module is used to fuse RGB features and depth features of the same level to obtain multimodal fusion features of the level, and the weld recognition model obtains the weld recognition result according to the multimodal fusion features of different levels, thereby improving the weld recognition accuracy, thereby improving the welding quality.

[0009] To solve the above technical problems, the present invention further provides a computer device, comprising a processor, wherein the processor implements the method steps described in the weld identification method based on multimodal information of the present invention when executing a computer program.

[0010] The present invention is an improved invention, and its beneficial effects are the same as those of the weld identification method based on multimodal information of the present invention.

[0011] In order to solve the above technical problems, the present invention also provides a weld tracking method based on multimodal information, comprising the following steps: Acquire an RGB image and a depth image of the workpiece to be welded, and input the acquired RGB image and depth image into a trained weld recognition model to obtain a weld recognition result; The weld recognition model includes a feature extraction module and a feature fusion module. The feature extraction module is used to extract RGB features at different levels and depth features at different levels. The feature fusion module is used to fuse the RGB features and depth features at the same level to obtain multimodal fusion features at the same level, so that the weld recognition model splices the multimodal fusion features at different levels to obtain the weld recognition result. Weld seam tracking is performed based on the weld seam recognition results.

[0012] Furthermore, the process of tracking the weld according to the weld recognition result includes: if the weld recognition result is that there is a weld, controlling the welding gun to move to the starting position of the identified weld, and then inputting the image including the laser beam intersecting the weld into the trained weld point recognition model to identify the position of the weld, and tracking the weld according to the weld point position.

[0013] Furthermore, the solder joint recognition model is a yolov5-face model.

[0014] Furthermore, the starting position of the weld is the starting position of the weld in the robot arm coordinate system obtained according to the weld recognition result and the hand-eye conversion matrix of the eye on the hand.

[0015] Furthermore, the welding point position is the position of the welding point in the robot arm coordinate system.

[0016] Furthermore, the hand-eye conversion matrix is ​​obtained by a checkerboard calibration method.

[0017] The present invention is an improved invention, and its beneficial effects are the same as those of the weld identification method based on multimodal information of the present invention.

[0018] To solve the above technical problems, the present invention also provides an automatic welding device, comprising a processor and a robotic arm for controlling the movement of a welding gun, and also comprising an image detection module for acquiring an RGB image and a depth image of a workpiece to be welded. When executing a computer program, the processor implements the method steps described in the weld tracking method based on multimodal information of the present invention.

[0019] Furthermore, it also includes a laser weld seam tracking system, which includes a laser transmitter, and the laser transmitter is used to emit a laser beam that intersects the weld seam.

[0020] The present invention is an improved invention, and its beneficial effects are the same as those of the weld identification method based on multimodal information of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1a and Figure 1b They are respectively an RGB image and a depth image of an embodiment of the automatic welding device of the present invention; Figure 2 is a schematic diagram of a weld seam tracking system of an automatic welding device embodiment of the present invention; Figure 3 is a structural schematic diagram of a DFM feature fusion module of an automatic welding device embodiment of the present invention; Figure 4 is a structural schematic diagram of a weld seam recognition model of an automatic welding device embodiment of the present invention; Figure 5 is a weld prediction result diagram of an automatic welding device embodiment of the present invention; Figure 6 is a schematic diagram of welding points collected by a laser weld seam tracking system of an automatic welding device embodiment of the present invention; Figure 7 Schematic diagram of welding spot prediction of an automatic welding device embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention provides a weld recognition and tracking method based on multimodal information, an automatic welding device and a computer device. Two visual images, RGB image and depth image, are input into a weld recognition model to obtain multimodal fusion features. The weld recognition result obtained based on the multimodal feature is avoided, and the problem that the use of a single visual image is easily affected by light changes and smoke interference during the welding process, thereby affecting the weld recognition accuracy, can improve the welding quality. Specifically, the weld recognition model includes a feature extraction module and a feature fusion module. The feature extraction module is used to extract RGB features of different levels and depth features of different levels. The feature fusion module is used to fuse RGB features and depth features of the same level to obtain multimodal fusion features of the level. The weld recognition model obtains the weld recognition result according to the multimodal fusion features of different levels, improves the weld recognition accuracy, and thus can improve the welding quality.

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0024] Automatic welding device embodiment: The automatic welding device of the present invention comprises a robot arm, at the end of which a welding device is fixed and a camera (hereinafter also referred to as a 3D camera or a high-speed camera) is installed, the camera is used to obtain a depth image of the workpiece to be welded (such as Figure 1b ) and RGB images (as Figure 1a As shown in the figure), these multimodal data are processed through deep learning models to achieve accurate identification and segmentation of welds; a laser weld tracking system is also installed at the end of the robot arm. The laser weld tracking system uses the laser beam and the camera to accurately capture the image details of the weld area and determine the weld position. The position of the robot arm is adjusted according to the position of the weld and the weld recognition results, thereby improving the quality of automatic welding. The structure of the weld tracking system is shown in the figure. Figure 2 As shown, the weld seam tracking system includes a laser transmitter.

[0025] The automatic welding device of the present invention further comprises a processor, which implements the following steps when executing the computer program: 1. Identify the weld.

[0026] In order to improve welding quality, it is necessary to first improve the accuracy of weld recognition. Therefore, the present invention uses depth image and RGB image fusion for weld recognition, avoiding the problem that single visual information may be affected by factors such as ambient light changes and smoke interference, resulting in insufficient recognition accuracy. Specifically, the weld recognition steps of the present invention are as follows: 1.1. Obtain the depth image and RGB image of the workpiece to be welded captured by the camera.

[0027] 1.2. Input the depth image and RGB image of the workpiece to be welded into the weld recognition model to obtain the weld recognition result.

[0028] The weld recognition model includes a feature extraction module and a feature fusion module. The feature extraction module is used to extract RGB features of different levels and depth features of different levels. The feature fusion module is used to fuse the RGB features and depth features of the same level to obtain the multimodal fusion features of the level, so that the weld recognition model can splice the multimodal fusion features of different levels to obtain the weld recognition result.

[0029] The weld recognition model is used to extract features from the acquired depth image data and RGB color information, and the acquired depth image data and RGB color information are input into the weld recognition model for segmentation. The feature extraction module includes two convolutional neural networks of ResNet architecture, wherein each residual block connected sequentially in one convolutional neural network of ResNet architecture outputs RGB features of different levels respectively, and each residual block connected sequentially in another convolutional neural network of ResNet architecture outputs depth features of different levels respectively.

[0030] The specific structure is: using two ResNet architecture convolutional neural networks (CNN) as the backbone network, the specific structure is as follows Figure 4 As shown in the figure, a 3-channel RGB image and a 1-channel depth image are taken as input, the depth image is copied to the 3-channel image for processing, and two networks with the same structure (i.e., ResNet50) are used to extract appearance features and depth features respectively.

[0031] ResNet-50 is a widely used deep learning model with a depth of 50 layers. It solves the degradation problem in deeper network training by using a residual learning framework. The model includes multiple residual blocks, each of which has a weight layer. In this embodiment, skip connections are added between the input and output of the residual block so that the input can skip some layers, allowing the signal to propagate directly through the skip connection during network training, thereby solving the gradient vanishing problem in deep network training and maintaining network performance.

[0032] In this embodiment, the feature map of the middle convolutional layer of ResNet-50 is used as the shared feature map, and the outputs of the 10th, 22nd and 40th layers are specifically selected as shared map features. These feature maps contain rich color information, texture information and geometric information, which are crucial for fusing the two modalities. The rich color and texture information in the RGB image and the geometric information of the scene in the depth image are shared respectively. By sharing the feature map, the model can learn the relationship between the two types of information, thereby improving the understanding of the scene.

[0033] In the present invention, the output of each residual block in the first three residual blocks (ResBlock-1, ResBlock-2, ResBlock-3) of ResNet50 is input to the deep fusion module DFM corresponding to the residual block. The DFM module is used to enhance the model's perception of important features through the attention mechanism and promote the effective fusion of features between different modalities. The structure of the deep fusion module DFM is as follows: Figure 3 As shown in the figure, RGB-feature map is the RGB feature map, Depth-feature map is the depth feature map, and the definition of the DFM module is as follows: F= S R || S D S R =X_R ||(softmax(Q_R⊙K_D )⊕V_D) S D =X_D||(softmax(Q_D⊙K_R )⊕V_R) Here, || means combining the features to fuse RGB features with depth features to improve the accuracy and robustness of feature fusion.

[0034] In the formula, S R is the enhanced RGB feature, X_R (X R ) is the RGB feature, Q_R (Q R ) represents the RGB query feature, K_D (K D ) is the depth key feature, V_D (V D ) is the depth value feature; S D is the enhanced depth feature, in the X_D figure (X D ) is the depth feature, Q_D (Q D ) is the deep query feature, K_R (K R ) is the RGB key feature, V_R (V R ) is the RGB value feature.

[0035] The DFM module learns cross-modal information features from X_R and X_D. Prior to this, X_R is converted into Q_R, K_R and V_R, and X_D is converted into Q_D, K_D and V_D. Then, we generate complementary cross-modal features through the DFM module. Specifically, the DFM module calculates the attention weights between the query and the key through a softmax operation, and then uses these weights to weight the values ​​to achieve the focus point positioning of cross-modal features. The weighted deep feature K_D is Hadamard-multiplied (element-wise multiplication) with the original RGB query feature Q_R, and then the softmax function (not shown in the figure) is used to obtain the attention weights, which are multiplied by the value vector (V_D) of the deep feature and the results are summed ( Figure 3 The "Product andSummation" operation of the DFM module in the figure (product and sum operation) is used to obtain the first fused feature. Then, the first fused feature is merged with the RGB feature through a concatenation operation (Concatenation in the figure, also known as splicing or union) to produce an enhanced RGB feature. The weighted RGB feature K_R is Hadamard-producted (element-wise multiplication) with the original deep query feature Q_D, and the attention weights are obtained through the softmax function. These attention weights are then applied to the value (V_R) of the RGB feature, the attention weights are multiplied by the value vector, and the results are summed to finally obtain the output vector to obtain the second fused feature. Then, the second fused feature is merged with the deep feature through a concatenation operation (Concatenation) to produce an enhanced deep feature. Finally, the enhanced RGB feature is fused with the deep feature through a concatenation operation (Concatenation), and the features of the two modalities after deep fusion are output.

[0036] Correspondingly, in this embodiment, the multimodal fusion features at different levels include: deep multimodal fusion features (output of the first DFM module), middle multimodal fusion features (output of the second DFM module), and shallow multimodal fusion features (output of the third DFM module); The process of splicing the multimodal fusion features at different levels to obtain the weld recognition result includes: splicing the deep multimodal fusion features and the middle multimodal fusion features, and splicing the splicing result with the shallow multimodal fusion features to obtain the weld recognition result, ( Figure 4 The weld recognition results are as follows: Figure 5 shown.

[0037] 2. Weld tracking.

[0038] During the welding process, due to the vibration of the workpiece to be welded or other influences, the position and shape of the weld will change slightly. In order to solve this problem, the automatic welding device of this embodiment adds a weld tracking system to detect the weld point and control the movement of the robot arm so that the weld point moves along the welding route. The laser transmitter emits a laser beam that intersects the weld seam. There is no laser return at the weld seam, and the brightness is low. Figure 6 The white line in the figure is the image of the laser beam.

[0039] The specific weld seam tracking method is executed by the controller and includes the following steps: 2.1. Construct the hand-eye conversion matrix to obtain the welding route in the robot arm coordinate system.

[0040] The starting point, end point and welding route of the weld are obtained according to the weld recognition result. Since the final controlled object is a robotic arm, in order to accurately control the robotic arm, the weld recognition result needs to be converted from the camera coordinate system to the robotic arm coordinate system. In this embodiment, a hand-eye conversion matrix of the camera coordinate system and the robotic arm coordinate system is constructed, and then the hand-eye conversion matrix Rc1 b is used to convert the welding route from the camera coordinate system to the robotic arm coordinate system. And the starting point and end point of the welding are determined according to the welding route. The hand-eye conversion matrix is ​​obtained by hand-eye calibration. And the checkerboard calibration method is collected for calibration.

[0041] 2.2. Control the robot arm to move to the welding starting point and perform welding according to the welding route.

[0042] The control module controls the robot arm to move to the starting point of welding, and then according to the calculated welding path, the control module controls the robot arm to gradually weld the workpiece to be welded along the welding path. First, the control module controls the robot arm to move to the starting point of welding. After moving to the starting point, the robot arm can control the welding gun to move smoothly according to the predetermined welding route or the real-time calculated welding route to perform welding operations. In this embodiment, in order to improve the quality of welding, the control module also needs to accurately control the motion parameters of the robot arm, and the motion parameters of the robot arm include speed, acceleration, trajectory smoothness and other aspects. At the same time, the control module will also control the welding gun to adjust the welding parameters according to the welding process requirements, such as current, voltage and welding speed. Through this series of precise control measures, the system can ensure that the welding process is stable and reliable, and achieve the expected welding effect.

[0043] 2.3. Get the real-time position of the welding point.

[0044] The laser weld tracking system is used to obtain the weld position. By detecting the position of the key weld, it is determined whether the weld position is offset. During the welding process, through the synergy of the high-speed camera and the laser beam, the control module can quickly capture the slight changes in the weld position and convert these changes into image data. In this embodiment, the laser image intersecting the weld is input into the trained weld recognition model to identify the weld position. The weld recognition result is as follows: Figure 7 In this embodiment, the welding point recognition model adopts the yolov5-face model, and the five-point coordinate detection in the original model is modified to the one-point coordinate detection. The model inputs the image collected by the laser weld tracking system L1 and returns the pixel coordinates of the welding point in the laser weld tracking system L1.

[0045] 2.4. Adjust the welding gun position according to the welding point position and welding route to achieve the movement of the welding point along the welding route.

[0046] The hand-eye relationship matrix Rc2 b between the final weld system and the robot arm coordinates is constructed. During the real-time welding process, the system will perform coordinate conversion based on the weld point position data obtained in real time by the laser weld tracking system L1 and the hand-eye conversion matrix Rc2 b. To adjust the position of the robot arm, the position of the welding gun weld point is adjusted, so that the weld point tracks the weld operation.

[0047] Weld seam identification method embodiment: The weld identification method of the present invention is as described in “1. Weld identification” of the automatic welding device embodiment. The specific process, principle and beneficial effects of the method have been described in detail in the automatic welding device embodiment and will not be repeated in this embodiment.

[0048] Weld seam tracking method embodiment: The weld seam tracking method of the present invention is as described in “II. Weld Seam Tracking” of the embodiment of the automatic welding device. The specific process, principle and beneficial effects of the method have been described in detail in the embodiment of the automatic welding device and will not be repeated in this embodiment.

[0049] Computer equipment example: A computer device of the present invention includes a processor, and the processor implements the method steps of the weld identification method described in the embodiment of the automatic welding device when executing a computer program. The specific process, principle and beneficial effects of the method have been described in detail in the embodiment of the automatic welding device, and will not be repeated in this embodiment. Among them, the processor can be a processing device such as a microprocessor MCU, a programmable logic device FPGA, etc.

Claims

1. A weld identification method based on multimodal information, characterized in that: The following steps are involved: Obtaining an RGB image and a depth image of the workpiece to be welded, and inputting the obtained RGB image and depth image into a trained weld recognition model to obtain a weld recognition result; The weld recognition model includes a feature extraction module and a feature fusion module. The feature extraction module is used to extract RGB features of different levels and depth features of different levels. The feature fusion module is used to fuse the RGB features and depth features of the same level to obtain the multimodal fusion features of the level, and splice the multimodal fusion features of different levels for output.

2. The weld identification method based on multimodal information according to claim 1 is characterized in that: The process of the feature fusion module fusing the RGB features and the depth features of the same level includes: converting the RGB features of the level into RGB query features, RGB key features and RGB value features, and converting the depth features of the level into depth query features, depth key features and depth value features; after multiplying the RGB query features and the depth key features of the level element by element, obtaining the attention weights through a softmax operation, performing a product-sum operation on the attention weights and the depth value features of the level to obtain the first fused features of the level, connecting the first fused features of the level with the RGB features of the level to obtain the enhanced RGB features of the level; after multiplying the depth query features and the RGB key features of the level element by element, obtaining the attention weights of the products through a softmax operation, performing a product-sum operation on the attention weights and the values ​​of the RGB features of the level to obtain the second fused features of the level, merging the second fused features of the level with the depth features to obtain the enhanced depth features of the level; fusing the enhanced RGB features with the enhanced depth features to obtain the multimodal fusion features.

3. The weld identification method based on multimodal information according to claim 1 is characterized in that: The feature extraction module includes two convolutional neural networks of ResNet architecture, wherein the residual blocks connected in sequence in one convolutional neural network of ResNet architecture respectively output RGB features of different levels, and the residual blocks connected in sequence in the other convolutional neural network of ResNet architecture respectively output depth features of different levels.

4. The weld identification method based on multimodal information according to claim 1, characterized in that: Multimodal fusion features at different levels include: deep multimodal fusion features, middle multimodal fusion features and shallow multimodal fusion features; Correspondingly, the process of splicing multimodal fusion features at different levels includes: splicing deep multimodal fusion features and middle multimodal fusion features, and splicing the splicing results with shallow multimodal fusion features for output.

5. A computer device comprising a processor, characterized in that: The processor implements the steps of the weld identification method based on multimodal information as described in any one of claims 1 to 4 when executing the computer program.

6. A weld tracking method based on multimodal information, characterized in that: The following steps are involved: Obtaining an RGB image and a depth image of the workpiece to be welded, and inputting the obtained RGB image and depth image into a trained weld recognition model to obtain a weld recognition result; The weld recognition model includes a feature extraction module and a feature fusion module. The feature extraction module is used to extract RGB features at different levels and depth features at different levels. The feature fusion module is used to fuse the RGB features and depth features at the same level to obtain multimodal fusion features at the same level, so that the weld recognition model splices the multimodal fusion features at different levels to obtain the weld recognition result. Weld seam tracking is performed based on the weld seam recognition results.

7. The weld tracking method based on multimodal information according to claim 6, characterized in that: The process of tracking the weld according to the weld recognition result includes: if the weld recognition result is that there is a weld, the welding gun is controlled to move to the starting position of the identified weld, and then the image including the laser beam intersecting the weld is input into the trained weld point recognition model to identify the position of the weld, and the weld is tracked according to the position of the weld.

8. The weld tracking method based on multimodal information according to claim 7, characterized in that: The solder joint recognition model is a yolov5-face model.

9. The weld tracking method based on multimodal information according to claim 7, characterized in that: The starting position of the weld is the starting position of the weld in the robot arm coordinate system obtained according to the weld recognition result and the hand-eye conversion matrix of the eye on the hand.

10. The weld tracking method based on multimodal information according to claim 7, characterized in that: The welding point position is the position of the welding point in the robot arm coordinate system.

11. The weld tracking method based on multimodal information according to claim 9, characterized in that: The hand-eye conversion matrix is ​​obtained by a checkerboard calibration method.

12. An automatic welding device, comprising a processor and a mechanical arm for controlling the movement of a welding gun, characterized in that: It also includes an image detection module for acquiring an RGB image and a depth image of a workpiece to be welded. When executing a computer program, the processor implements a weld tracking method based on multimodal information as described in any one of claims 6 to 11.

13. The automatic welding device according to claim 12, characterized in that: The invention also comprises a laser weld seam tracking system, wherein the laser weld seam tracking system comprises a laser transmitter, and the laser transmitter is used for transmitting a laser beam intersecting the weld seam.

Images