Workpiece weld joint recognition method, system, device and medium based on multi-feature fusion

Abstract

The application discloses a workpiece weld joint recognition method, system, device and medium based on multi-feature fusion, wherein the method comprises the following steps: acquiring a plurality of two-dimensional images and a plurality of three-dimensional point clouds corresponding to a workpiece to be measured in a welding process, wherein the plurality of two-dimensional images and the plurality of three-dimensional point clouds have a time corresponding relationship; performing feature extraction on the plurality of two-dimensional images to obtain a plurality of image feature sequences; performing feature extraction on the plurality of three-dimensional point clouds to obtain a plurality of point cloud feature sequences; fusing the plurality of image feature sequences and the plurality of point cloud feature sequences according to the time corresponding relationship to obtain a plurality of initial fusion feature sequences; performing short-time adjacent frame fusion on the plurality of initial fusion feature sequences by using an attention mechanism to obtain a plurality of final fusion feature sequences; and detecting the plurality of final fusion feature sequences to obtain a plurality of weld joint recognition results. The application can improve the accuracy and robustness of weld joint recognition.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a method, system, device, and medium for identifying workpiece welds based on multi-feature fusion. Background Technology

[0002] Currently, welding technology has become an indispensable key process in the industrial field. With the development of production and the advancement of science and technology, its application scope covers aerospace, nuclear industry, shipbuilding, construction, and machinery manufacturing. Welding technology refers to the process of heating metallic or non-metallic materials to a molten state, then cooling and joining them together to achieve a strong connection. It is suitable for the manufacture and repair of various structures.

[0003] With the continuous evolution of welding technology, welding robots have become a key representative of welding automation. Equipped with a weld seam recognition system, they automatically adjust the welding torch position by monitoring the weld seam location in real time, ensuring the torch always welds along the seam and guaranteeing consistent and stable welding quality. However, existing weld seam recognition systems primarily detect weld seams based on the current frame image captured by the vision sensors carried by the welding robot, which can easily lead to low accuracy in weld seam recognition results. Summary of the Invention

[0004] This invention provides a method, system, device, and medium for identifying workpiece welds based on multi-feature fusion, in order to solve one or more technical problems existing in the prior art, and at least provide a beneficial option or create conditions.

[0005] Firstly, a method for identifying workpiece welds based on multi-feature fusion is provided, the method comprising:

[0006] Acquire multiple frames of two-dimensional images and multiple frames of three-dimensional point clouds corresponding to the workpiece under test during the welding process, wherein there is a time correspondence between the multiple frames of two-dimensional images and the multiple frames of three-dimensional point clouds;

[0007] Feature extraction is performed on the multiple frames of two-dimensional images to obtain multiple image feature sequences;

[0008] Feature extraction is performed on the multiple frames of 3D point cloud to obtain multiple point cloud feature sequences;

[0009] Based on the time correspondence, the multiple image feature sequences and the multiple point cloud feature sequences are fused to obtain multiple initial fused feature sequences;

[0010] An attention mechanism is used to perform short-time adjacent-frame fusion on the multiple initial fused feature sequences to obtain multiple final fused feature sequences;

[0011] The multiple final fused feature sequences are detected to obtain multiple weld seam identification results.

[0012] Furthermore, the step of extracting features from the multiple frames of two-dimensional images to obtain multiple image feature sequences includes:

[0013] For any frame of a two-dimensional image, a convolutional neural network is used to extract features from that frame of the two-dimensional image, resulting in several feature maps;

[0014] The average pooling operation is used to convert the several feature maps into several feature vectors of fixed length and form an image feature sequence.

[0015] Furthermore, the step of extracting features from the multiple frames of 3D point clouds to obtain multiple point cloud feature sequences includes:

[0016] For any frame of 3D point cloud, perform layered sampling on the 3D point cloud of that frame to obtain multiple point cloud sets;

[0017] A multilayer perceptron is used to extract features from the multiple point cloud sets to obtain multiple local features;

[0018] The multiple local features are aggregated using upsampling and max pooling operations to obtain a point cloud feature sequence.

[0019] Further, the step of fusing the multiple image feature sequences and the multiple point cloud feature sequences according to the time correspondence to obtain multiple initial fused feature sequences includes:

[0020] For image feature sequences and point cloud feature sequences with the same correlation time, a fully connected layer with a non-linear activation function is used to fuse the image feature sequences and the point cloud feature sequences to obtain an initial fused feature sequence.

[0021] Furthermore, the step of using an attention mechanism to perform short-time neighbor-to-neighbor frame fusion on the multiple initial fusion feature sequences to obtain multiple final fusion feature sequences includes:

[0022] The initial fusion feature sequence with the earliest association time is taken as its corresponding final fusion feature sequence;

[0023] For two initial fusion feature sequences with adjacent association times, the initial fusion feature sequence with the earlier association time is denoted as the first initial fusion feature sequence, and the initial fusion feature sequence with the later association time is denoted as the second initial fusion feature sequence. The similarity matrix between the two initial fusion feature sequences is then determined.

[0024] Multiply the similarity matrix with the first initial fusion feature sequence to obtain the reconstructed fusion feature sequence;

[0025] The reconstructed fusion feature sequence is weighted and summed with the second initial fusion feature sequence to obtain the final fusion feature sequence corresponding to the second initial fusion feature sequence.

[0026] Furthermore, the detection of the multiple final fused feature sequences to obtain multiple weld identification results includes:

[0027] A target detection network is used to process the multiple final fused feature sequences to obtain multiple initial weld identification results;

[0028] The initial weld identification results are corrected using a long short-term memory network to obtain the final weld identification results.

[0029] Furthermore, the target detection network includes a bounding box regression layer and a category prediction layer. The process of using the target detection network to process the multiple final fused feature sequences to obtain multiple initial weld seam identification results includes:

[0030] For any final fused feature sequence, the bounding box regression layer is used to process the final fused feature sequence to obtain several position parameters corresponding to several target bounding boxes;

[0031] The category prediction layer is used to process the final fused feature sequence to obtain several category information corresponding to several target bounding boxes. Then, all feature subsequences corresponding to all target bounding boxes belonging to the weld category are extracted from the final fused feature sequence.

[0032] Morphological operations are used to process all feature subsequences to obtain all shape parameters corresponding to all target bounding boxes, and then combined with all position parameters corresponding to all target bounding boxes to generate initial weld recognition results.

[0033] Secondly, a workpiece weld seam recognition system based on multi-feature fusion is provided, the system comprising:

[0034] The first module is used to acquire multiple frames of two-dimensional images and multiple frames of three-dimensional point clouds corresponding to the workpiece under test during the welding process, wherein there is a time correspondence between the multiple frames of two-dimensional images and the multiple frames of three-dimensional point clouds;

[0035] The second module is used to extract features from the multiple frames of two-dimensional images to obtain multiple image feature sequences.

[0036] The third module is used to extract features from the multi-frame 3D point cloud to obtain multiple point cloud feature sequences.

[0037] The fourth module is used to fuse the multiple image feature sequences and the multiple point cloud feature sequences according to the time correspondence to obtain multiple initial fused feature sequences;

[0038] The fifth module is used to perform short-term adjacent frame fusion on the multiple initial fusion feature sequences using an attention mechanism to obtain multiple final fusion feature sequences;

[0039] The sixth module is used to detect the multiple final fused feature sequences to obtain multiple weld identification results.

[0040] Thirdly, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the workpiece weld seam recognition method based on multi-feature fusion as described in the first aspect.

[0041] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the workpiece weld seam recognition method based on multi-feature fusion as described in the first aspect.

[0042] The present invention has at least the following beneficial effects: by using two-dimensional images and three-dimensional point clouds for feature fusion, the limitation of a single data source in traditional weld seam recognition technology can be overcome. Furthermore, by adopting a short-term feature fusion method based on attention mechanism and a long short-term memory network, feature information in different time dimensions can be fully utilized to better track the change process of the weld seam, and the impact of environmental interference in the welding process on the weld seam recognition results can be minimized, which helps to improve the accuracy and robustness of weld seam recognition. Attached Figure Description

[0043] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0044] Figure 1 This is a flowchart illustrating a workpiece weld seam recognition method based on multi-feature fusion in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of the composition of a workpiece weld seam recognition system based on multi-feature fusion in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the hardware structure of the computer device in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0048] It should be noted that although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," "third," "fourth," etc., used in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed and are inherent to these processes, methods, products, or apparatuses.

[0049] Existing weld seam recognition systems configured on welding robots suffer from the following drawbacks: First, images captured by the vision sensors carried by the welding robot are easily affected by environmental factors, such as smoke, sparks, and spatter that occur during welding, leading to a decrease in image quality, primarily manifested as blurred weld seam outlines. The welding robot's imperfectly stable motion during welding causes the vision sensors to shake, further increasing the overall blurriness and noise of the image. Second, the system's timeliness is limited, considering only the instantaneous current frame image and lacking analysis of global contextual information, making it difficult to capture the shape changes and overall evolution of the weld seam at different points in time. The system also has poor fault tolerance, with limited ability to handle noise and occasional interference in the image, easily leading to erroneous weld seam recognition. Therefore, this invention provides a workpiece weld seam recognition method, system, device, and medium based on multi-feature fusion, aiming to overcome the shortcomings of existing weld seam recognition systems as much as possible.

[0050] First, some of the terms used in this invention are explained as follows:

[0051] Convolutional Neural Networks (CNNs) are a class of feedforward neural networks that incorporate convolutional computations and have deep structures. They possess representation learning capabilities and can perform translation-invariant classification of input information according to their hierarchical structure. They are constructed by mimicking the visual perception mechanisms of biological organisms and can perform supervised and unsupervised learning. The shared parameters of the convolutional kernels within their hidden layers and the sparsity of the interlayer connections enable CNNs to learn gridded features, such as pixels and audio, with relatively low computational cost, stable performance, and no additional feature engineering requirements on the data.

[0052] A multilayer perceptron (MLP) is a feedforward artificial neural network model that maps multiple input datasets to a single output dataset.

[0053] Average pooling refers to taking the average value of all elements in each local region of the input feature map and using that local region as the output value.

[0054] Max pooling refers to taking each local region in the input feature map and using the maximum value within that local region as the output value for that local region, thereby preserving the most salient features.

[0055] Long Short-Term Memory (LSTM) is a deep learning model for processing time series data. It is an improvement on Recurrent Neural Network (RNN) and aims to solve the gradient vanishing and gradient exploding problems in RNN, while being able to better capture long-term dependencies.

[0056] The core idea of ​​LSTM is to introduce a gate structure to selectively control the flow of information. It consists of an input gate, a forget gate, and an output gate. Specifically: the input gate uses a sigmoid activation function to combine the current input with the previous memory state to obtain a value between 0 and 1, and then uses a tanh activation function to use this combination as a new memory candidate value; the forget gate uses a sigmoid activation function to evaluate the combination of the previous memory state and the current input to obtain a value between 0 and 1, and then multiplies this value by the previous memory state to determine the information to be retained; the output gate uses a sigmoid activation function to evaluate the combination of the current input and the memory state to obtain a value between 0 and 1, and then uses a tanh activation function to process the current memory state and multiply it by this value to determine the current output of the LSTM.

[0057] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a workpiece weld seam recognition method based on multi-feature fusion provided by an embodiment of the present invention. The method includes the following:

[0058] Step S110: Acquire multiple frames of two-dimensional images and multiple frames of three-dimensional point clouds corresponding to the workpiece under test during the welding process, wherein there is a time correspondence between the multiple frames of two-dimensional images and the multiple frames of three-dimensional point clouds;

[0059] Step S120: Extract features from the multiple frames of two-dimensional images to obtain multiple image feature sequences;

[0060] Step S130: Extract features from the multi-frame 3D point cloud to obtain multiple point cloud feature sequences;

[0061] Step S140: According to the time correspondence, the multiple image feature sequences and the multiple point cloud feature sequences are fused to obtain multiple initial fused feature sequences;

[0062] Step S150: Use an attention mechanism to perform short-term adjacent frame fusion on the multiple initial fusion feature sequences to obtain multiple final fusion feature sequences;

[0063] Step S160: Detect the multiple final fused feature sequences to obtain multiple weld identification results.

[0064] In some embodiments of the present invention, a welding robot is used to complete the welding task on the workpiece to be tested. The welding robot is equipped with a point cloud acquisition device and an image acquisition device. Preferably, the point cloud acquisition device and the image acquisition device are positioned on both sides of the welding robot to avoid mutual interference during data acquisition. The implementation of step S110 is as follows: the welding robot is controlled to weld the weld seam on the workpiece to be tested according to the user-input teaching trajectory. During the welding process, the point cloud acquisition device is controlled to acquire a frame of three-dimensional point cloud towards the welding surface of the workpiece to be tested using a timing method, and the image acquisition device is controlled to acquire a frame of two-dimensional image towards the welding surface of the workpiece to be tested. Through multiple timing control operations, multiple frames of three-dimensional point cloud and multiple frames of two-dimensional image with a time correspondence can be obtained. The multiple frames of three-dimensional point cloud are denoted as {P1, P2, ..., P...} according to the acquisition time from earliest to latest. N The multiple frames of two-dimensional images are denoted as {I1, I2, ..., I...} according to the order of their acquisition time from earliest to latest. N}, and a frame of 3D point cloud P i Acquisition time and one frame of two-dimensional image I i The data collection times are the same, i = 1, 2, ..., N.

[0065] In some embodiments of the present invention, any one of the two-dimensional images I contained in the multi-frame two-dimensional images is used. i For example, the implementation process of step S120 above includes, but is not limited to, the following:

[0066] Step S121, save the frame of two-dimensional image I i The input is fed into a convolutional neural network for feature extraction to obtain several feature maps; wherein, the convolutional neural network preferably adopts a pre-trained ResNet residual network, which mainly calculates the feature representation of the two-dimensional image in the network through forward propagation, in order to capture features from low level to high level;

[0067] Step S122: For each feature map contained in the plurality of feature maps, convert each feature map into a fixed-length feature vector using average pooling, and then combine the converted feature vectors into an image feature sequence and denoted as IS. i ;

[0068] After processing each frame of the two-dimensional image contained in the multi-frame two-dimensional image according to the above steps S121 to S122, multiple image feature sequences can be generated and denoted as {IS1, IS2, ..., IS...} N}

[0069] In some embodiments of the present invention, any one of the three-dimensional point cloud frames P contained in the multi-frame three-dimensional point cloud is used. i For example, step S130 above actually uses the PointNet++ algorithm to extract point cloud features. The corresponding implementation process includes, but is not limited to, the following:

[0070] Step S131: Perform an analysis of the 3D point cloud P in this frame. i Perform layered sampling to obtain the 3D point cloud P of this frame. i It is decomposed into multiple distinct point cloud sets;

[0071] Step S132: For each point cloud set contained in the plurality of point cloud sets, input each point cloud set into a multilayer perceptron for feature extraction to obtain local features;

[0072] Step S133: Aggregate the acquired local features using upsampling and max pooling to obtain a point cloud feature sequence containing key weld information, denoted as PS. i ;

[0073] After processing each frame of the 3D point cloud contained in the multi-frame 3D point cloud according to the above steps S131 to S133, multiple point cloud feature sequences can be generated and denoted as {PS1, PS2, ..., PS...} N}

[0074] In some embodiments of the present invention, any one of the image feature sequences IS included in the plurality of image feature sequences is used. i For example, IS is obtained from the plurality of point cloud feature sequences and compared with the image feature sequence. i Point cloud feature sequences with the same associated time PS i The implementation process of step S140 above includes, but is not limited to, the following:

[0075] Step S141: Set the basic parameters required for the fully connected layer, including the bias parameter b, the weight matrix W1 with respect to the image feature sequence and the weight matrix W2 with respect to the point cloud feature sequence, and limit the nonlinear activation function used by the fully connected layer to the ReLU (Rectified Linear Unit) function.

[0076] Step S142: Transfer the image feature sequence IS i and the point cloud feature sequence PS i The input is fed into the fully connected layer for fusion to obtain an initial fused feature sequence, which is denoted as... The mathematical expression used in this fusion process is: f refers to the ReLU function;

[0077] By processing the multiple point cloud feature sequences and the multiple image feature sequences according to the above step S142, multiple initial fused feature sequences can be generated and denoted as follows:

[0078] In some embodiments of the present invention, the initial fused feature sequence with the earliest associated time is used. It is directly defined as the final fusion feature sequence FS1 associated with it, which is the initial fusion feature sequence among the multiple initial fusion feature sequences that has a temporal adjacency relationship. and initial fusion feature sequence For example, the implementation process of step S150 above includes, but is not limited to, the following:

[0079] Step S151: Calculate the initial fusion feature sequence. With the initial fusion feature sequence The similarity matrix between them; this calculation process preferably uses a cosine similarity function, where each element in the similarity matrix represents the initial fused feature sequence. A certain feature data in the initial fused feature sequence The similarity between certain feature data in the data;

[0080] Step S152: Combine the similarity matrix and the initial fused feature sequence. Perform multiplication to obtain the reconstructed fused feature sequence, denoted as FS′. i-1 ;

[0081] Step S153: Reconstruct and fuse the feature sequence FS′ i-1 With the initial fusion feature sequence Perform a weighted summation to obtain the initial fusion feature sequence. The associated final fusion feature sequence FS i The mathematical expression used in this weighted summation process is: 'a' represents the pre-defined fusion weight;

[0082] By processing the multiple initial fusion feature sequences according to steps S151 to S153 above, multiple final fusion feature sequences can be generated and denoted as {FS1,FS2,...,FS...}. N}

[0083] In some embodiments of the present invention, the implementation process of step S160 includes, but is not limited to, the following:

[0084] Step S161: Process the multiple final fused feature sequences through a target detection network to obtain the corresponding multiple initial weld identification results;

[0085] The target detection network preferably adopts a pre-trained YOLO (You Only Look Once) series network, such as YOLOv3 network, YOLOv5s network, etc.; the target detection network includes a bounding box regression layer and a category prediction layer. The bounding box regression layer is used to detect the bounding box of the target object and learn to predict its position parameters. The category prediction layer is used to learn to predict the category parameters of the target object within each bounding box.

[0086] Step S162: Input the multiple initial weld identification results into a long short-term memory network for correction to obtain the final multiple weld identification results.

[0087] More specifically, taking any one of the multiple final fusion feature sequences FS... i For example, the implementation process of step S161 above includes, but is not limited to, the following:

[0088] Step S161.1: The final fused feature sequence FS i The data is input to the bounding box regression layer for processing to obtain several location parameters associated with several target bounding boxes;

[0089] Step S161.2: Based on the aforementioned positional parameters, the final fused feature sequence FS is... i The data is input to the category prediction layer for processing to obtain the category information associated with the target bounding boxes.

[0090] Step S161.3: Based on the aforementioned category information, determine all target bounding boxes belonging to the weld category among the aforementioned target bounding boxes, and then filter out all position parameters associated with all target bounding boxes from the aforementioned position parameters;

[0091] Step S161.4: Based on all the positional parameters, from the final fused feature sequence FS i The feature subsequences associated with all target bounding boxes are selected from the data.

[0092] Step S161.5: Perform morphological operations, such as dilation or erosion, on all the feature subsequences to obtain all shape parameters associated with all the target bounding boxes;

[0093] Step S161.6: For each target bounding box contained within all the target bounding boxes, bind associated position and shape parameters to each target bounding box as the detection result of each target bounding box, thereby forming the final fused feature sequence FS. i The associated initial weld identification results are recorded as

[0094] By processing the multiple final fused feature sequences according to steps S161.1 to S161.6 above, multiple initial weld identification results can be generated and denoted as follows:

[0095] More specifically, step S162 above is proposed to minimize the impact of environmental interference (such as equipment vibration, welding fumes, rust, burrs, oil stains, etc. on the surface of the steel) on the weld identification results in practical applications. The corresponding implementation method is as follows:

[0096] The initial weld seam identification results were originally arranged in chronological order of their association time. These initial weld seam identification results were then constructed into a sequence, which was then input into a pre-trained Long Short-Term Memory (LSTM) network to learn the feature relationships and feature change trends between adjacent time points. In other words, the initial weld seam identification result with the earliest association time was used... It is directly defined as the final weld identification result T1 associated with it, for the initial weld identification results among the multiple initial weld identification results that have an adjacent relationship in terms of association time. and initial weld identification results The internal structure of the Long Short-Term Memory network will be determined with the help of the initial weld identification results. The initial weld identification results are derived from adaptive smoothing. To obtain the initial weld identification result The associated final weld identification result T i Following this smoothing process, multiple weld identification results can be obtained and denoted as {T1, T2, ..., T...}. N}

[0097] In this embodiment of the invention, by using two-dimensional images and three-dimensional point clouds for feature fusion, the limitation of a single data source in traditional weld seam recognition technology can be overcome. Furthermore, by employing a short-term feature fusion method based on an attention mechanism and a long short-term memory network, feature information from different time dimensions can be fully utilized to better track the change process of the weld seam, minimizing the impact of environmental interference during the welding process on the weld seam recognition results, and helping to improve the accuracy and robustness of weld seam recognition.

[0098] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the composition of a workpiece weld seam recognition system based on multi-feature fusion provided in an embodiment of the present invention. The system includes:

[0099] The first module 210 is used to acquire multiple frames of three-dimensional point cloud and multiple frames of two-dimensional image associated with the workpiece under test during the welding process, wherein there is a time correspondence between the multiple frames of three-dimensional point cloud and the multiple frames of two-dimensional image;

[0100] The second module 220 is used to extract features from the multi-frame two-dimensional images to obtain corresponding multiple image feature sequences.

[0101] The third module 230 is used to extract features from the multi-frame 3D point cloud to obtain the corresponding multiple point cloud feature sequences.

[0102] The fourth module 240 is used to fuse the multiple point cloud feature sequences and the multiple image feature sequences according to the time correspondence to obtain multiple initial fused feature sequences;

[0103] The fifth module 250 is used to perform short-term adjacent frame fusion on the multiple initial fusion feature sequences through an attention mechanism to obtain the corresponding multiple final fusion feature sequences;

[0104] The sixth module 260 is used to detect the multiple final fused feature sequences to obtain the corresponding multiple weld identification results.

[0105] The content of the above method embodiments is applicable to this system embodiment. The functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are the same as those in the above method embodiments. Therefore, they will not be repeated here.

[0106] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements a workpiece weld seam recognition method based on multi-feature fusion as described in the above embodiments. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the storage device includes any medium on which a device (e.g., a computer, mobile phone, etc.) stores or transmits information in a readable form, and can be a read-only memory, a disk, or an optical disk, etc.

[0107] also, Figure 3 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. The computer device includes components such as a processor 320, a memory 330, an input unit 340, and a display unit 350. Those skilled in the art will understand that... Figure 3 The illustrated device structure is not intended to limit all devices and may include more or fewer components than shown, or combine certain components. The memory 330 can be used to store the computer program 310 and various functional modules. The processor 320 runs the computer program 310 stored in the memory 330, thereby performing various functional applications and data processing of the device. The memory can be internal memory or external memory, or include both internal and external memory. Internal memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. External memory may include hard disks, floppy disks, USB flash drives, magnetic tapes, etc. The memory 330 disclosed in the embodiments of this invention includes, but is not limited to, these types of memory. The memory 330 disclosed in the embodiments of this invention is only an example and not a limitation.

[0108] Input unit 340 is used to receive signal input and user-input keywords. Input unit 340 may include a touch panel and other input devices. The touch panel can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel) and drive the corresponding connection device according to a pre-set program; other input devices may include, but are not limited to, one or more of physical keyboards, function keys (such as play control buttons, power buttons, etc.), trackballs, mice, joysticks, etc. Display unit 350 can be used to display user-input information or information provided to the user, as well as various menus of the terminal device. Display unit 350 may be in the form of a liquid crystal display, organic light-emitting diode, etc. Processor 320 is the control center of the terminal device, connecting various parts of the entire device through various interfaces and lines, performing various functions and processing data by running or executing software programs and / or modules stored in memory 330, and calling data stored in memory 330.

[0109] As one embodiment, the computer device includes a processor 320, a memory 330, and a computer program 310, wherein the computer program 310 is stored in the memory 330 and configured to be executed by the processor 320, and the computer program 310 is configured to perform a workpiece weld seam recognition method based on multi-feature fusion in the above embodiment.

[0110] Although the description of this application has been quite detailed and particularly focused on several of the described embodiments, it is not intended to limit itself to any of these details or embodiments or any particular embodiment. Rather, it should be considered as effectively covering the intended scope of this application by referring to the appended claims and taking into account the prior art, which provides for a broad possible interpretation of these claims. Furthermore, the foregoing description of this application with respect to embodiments foreseeable by the inventors is intended to provide a useful description, and non-substantial modifications to this application that have not yet been foreseen may still represent equivalent modifications.

Claims

1. A method for identifying workpiece welds based on multi-feature fusion, characterized in that, The method includes: Acquire multiple frames of two-dimensional images and multiple frames of three-dimensional point clouds corresponding to the workpiece under test during the welding process, wherein there is a time correspondence between the multiple frames of two-dimensional images and the multiple frames of three-dimensional point clouds; Feature extraction is performed on the multiple frames of two-dimensional images to obtain multiple image feature sequences; Feature extraction is performed on the multiple frames of 3D point cloud to obtain multiple point cloud feature sequences; Based on the time correspondence, the multiple image feature sequences and the multiple point cloud feature sequences are fused to obtain multiple initial fused feature sequences; An attention mechanism is used to perform short-time adjacent-frame fusion on the multiple initial fused feature sequences to obtain multiple final fused feature sequences; The multiple final fused feature sequences are detected to obtain multiple weld identification results; The step of using an attention mechanism to perform short-time adjacent-frame fusion of the multiple initial fusion feature sequences to obtain multiple final fusion feature sequences includes: The initial fusion feature sequence with the earliest association time is taken as its corresponding final fusion feature sequence; For two initial fusion feature sequences with adjacent association times, the initial fusion feature sequence with the earlier association time is denoted as the first initial fusion feature sequence, and the initial fusion feature sequence with the later association time is denoted as the second initial fusion feature sequence. The similarity matrix between the two initial fusion feature sequences is then determined. Multiply the similarity matrix with the first initial fusion feature sequence to obtain the reconstructed fusion feature sequence; The reconstructed fusion feature sequence is weighted and summed with the second initial fusion feature sequence to obtain the final fusion feature sequence corresponding to the second initial fusion feature sequence.

2. The workpiece weld seam recognition method based on multi-feature fusion according to claim 1, characterized in that, The step of extracting features from the multiple frames of two-dimensional images to obtain multiple image feature sequences includes: For any frame of a two-dimensional image, a convolutional neural network is used to extract features from that frame of the two-dimensional image, resulting in several feature maps; The average pooling operation is used to convert the several feature maps into several feature vectors of fixed length and form an image feature sequence.

3. The workpiece weld seam recognition method based on multi-feature fusion according to claim 1, characterized in that, The step of extracting features from the multiple frames of 3D point clouds to obtain multiple point cloud feature sequences includes: For any frame of 3D point cloud, perform layered sampling on the 3D point cloud of that frame to obtain multiple point cloud sets; A multilayer perceptron is used to extract features from the multiple point cloud sets to obtain multiple local features; The multiple local features are aggregated using upsampling and max pooling operations to obtain a point cloud feature sequence.

4. The workpiece weld seam recognition method based on multi-feature fusion according to claim 1, characterized in that, The step of fusing the multiple image feature sequences and the multiple point cloud feature sequences according to the time correspondence to obtain multiple initial fused feature sequences includes: For image feature sequences and point cloud feature sequences with the same correlation time, a fully connected layer with a non-linear activation function is used to fuse the image feature sequences and the point cloud feature sequences to obtain an initial fused feature sequence.

5. The workpiece weld seam recognition method based on multi-feature fusion according to claim 1, characterized in that, The detection of the multiple final fused feature sequences to obtain multiple weld seam identification results includes: A target detection network is used to process the multiple final fused feature sequences to obtain multiple initial weld identification results; The initial weld identification results are corrected using a long short-term memory network to obtain the final weld identification results.

6. The workpiece weld seam recognition method based on multi-feature fusion according to claim 5, characterized in that, The target detection network includes a bounding box regression layer and a category prediction layer. The process of using the target detection network to process the multiple final fused feature sequences to obtain multiple initial weld seam identification results includes: For any final fused feature sequence, the bounding box regression layer is used to process the final fused feature sequence to obtain several position parameters corresponding to several target bounding boxes; The category prediction layer is used to process the final fused feature sequence to obtain several category information corresponding to several target bounding boxes. Then, all feature subsequences corresponding to all target bounding boxes belonging to the weld category are extracted from the final fused feature sequence. Morphological operations are used to process all feature subsequences to obtain all shape parameters corresponding to all target bounding boxes, and then combined with all position parameters corresponding to all target bounding boxes to generate initial weld recognition results.

7. A workpiece weld seam recognition system based on multi-feature fusion, characterized in that, The system includes: The first module is used to acquire multiple frames of two-dimensional images and multiple frames of three-dimensional point clouds corresponding to the workpiece under test during the welding process, wherein there is a time correspondence between the multiple frames of two-dimensional images and the multiple frames of three-dimensional point clouds; The second module is used to extract features from the multiple frames of two-dimensional images to obtain multiple image feature sequences. The third module is used to extract features from the multi-frame 3D point cloud to obtain multiple point cloud feature sequences. The fourth module is used to fuse the multiple image feature sequences and the multiple point cloud feature sequences according to the time correspondence to obtain multiple initial fused feature sequences; The fifth module is used to perform short-term adjacent frame fusion on the multiple initial fusion feature sequences using an attention mechanism to obtain multiple final fusion feature sequences; The sixth module is used to detect the multiple final fused feature sequences to obtain multiple weld identification results; The step of using an attention mechanism to perform short-time adjacent-frame fusion of the multiple initial fusion feature sequences to obtain multiple final fusion feature sequences includes: The initial fusion feature sequence with the earliest association time is taken as its corresponding final fusion feature sequence; For two initial fusion feature sequences with adjacent association times, the initial fusion feature sequence with the earlier association time is denoted as the first initial fusion feature sequence, and the initial fusion feature sequence with the later association time is denoted as the second initial fusion feature sequence. The similarity matrix between the two initial fusion feature sequences is then determined. Multiply the similarity matrix with the first initial fusion feature sequence to obtain the reconstructed fusion feature sequence; The reconstructed fusion feature sequence is weighted and summed with the second initial fusion feature sequence to obtain the final fusion feature sequence corresponding to the second initial fusion feature sequence.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The processor executes the computer program to implement the workpiece weld seam recognition method based on multi-feature fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the workpiece weld seam recognition method based on multi-feature fusion as described in any one of claims 1 to 6.

Images