A method and system for visually monitoring welding operation process and quality

Abstract

The present invention discloses a method for visual monitoring of welding operation process and quality. First, a welding defect image data set and a welding operation video data set are constructed. Then, the backbone network of the welding defect target detection network is constructed using the EDS‑SuqeezeNet network. The network is trained with the welding defect image data set to obtain a welding defect target detection model. At the same time, a welding operation key point action recognition network is constructed, and the welding operation recognition network is trained with the welding operation video data set to obtain a welding operation key point action recognition model. The trained model is used to identify and detect welding workpiece images and welding operation videos to obtain the defect category of the welding workpiece and the welding operation specification recognition results. The present invention can detect defects such as uniformity of welds, welding spatter and weld fractures, and timely feedback to workers for adjustment, which effectively improves the quality of welded workpieces and greatly improves production efficiency and consistency.

Description

Technical Field

[0001] The invention belongs to the field of image processing, computer vision and pattern recognition, and specifically relates to a method and system for visually monitoring welding operation procedures and quality. Background Art

[0002] Welding is not only a basic link in the manufacturing process of product parts, but also one of the important factors that determine product quality. Therefore, welding quality has become the main goal of enterprise on-site quality control and a key component of the production site quality assurance system, especially in the welding process of key parts such as aerospace, military industry, ships, and trains.

[0003] Machine vision is currently a relatively effective monitoring technology for welding. This type of system captures images after welding through a camera installed in the welding area, and uses image processing algorithms and software for analysis to detect the quality of the weld and identify potential defects. For example, the Chinese patent with application number 202410438671.X discloses a real-time control method and system for welding quality based on visual detection. The invention first collects the three-dimensional structure data of the weld, uses a visual micro-scanning method to read the outer contour shape of the three-dimensional structure of the weld, and then uses an object separation method to separate the convex weld and the small mouth, and then extracts the shape characteristics of the convex weld and the shape characteristics of the small mouth. Finally, according to the shape characteristics of the convex weld and the small mouth, the welding quality of the three-dimensional structure of the weld is compared. However, at present, most of these systems collect and analyze the weld after welding is completed, and cannot provide immediate feedback during the welding process, which means that if the worker encounters any problems during the welding process, such as welding speed is too fast or too slow, welding angle is incorrect, etc., the system cannot detect and remind the worker to make adjustments in time, which may cause welding quality problems. In addition, the system basically only monitors the quality of the workpiece, and does not monitor the welding operation specifications of the workers. In order to monitor the standardization of workers' welding postures and detect the quality of workpiece welds, etc., and improve production quality, it is necessary to capture the details of workers' welding operations and the welding process of workpieces in real time. Summary of the invention

[0004] Aiming at the problem of simultaneously realizing visual inspection of whether workers operate in accordance with specifications and the quality of welding process, the present invention provides a method and system for visually monitoring welding operation procedures and quality, which can detect the welding process while monitoring workers' operation, and is used for real-time monitoring, alarming, and record storage.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for visually monitoring welding operation process and quality, comprising the following steps:

[0006] S1. Construct a welding defect image dataset, including workpiece welding defect images and their corresponding defect category labels; construct a welding operation video dataset, including welding operation action videos and their corresponding operation category labels;

[0007] S2. Build a welding defect target detection network, and use the welding defect image data set to train the welding defect recognition network to obtain a welding defect target detection model; build a welding operation key point action recognition network, and use the welding operation video data set to train the welding operation recognition network to obtain a welding operation key point action recognition model;

[0008] S3. The real-time collected welding workpiece images and welding operation videos are input into the welding defect target detection model and the welding operation key point action recognition model respectively to obtain the defect category of the welding workpiece and the welding operation specification recognition results.

[0009] Furthermore, the aforementioned welding defect target detection network specifically includes: a backbone network, a neck network, and a decoding prediction network. Specifically, the backbone network of the welding defect target detection network is constructed using the EDS-SuqeezeNet network. The backbone network uses the Fire module as the basic connection module of each layer, and the input is a Conv convolution unit. First, the rough feature information of the input welding workpiece image is extracted, and then the first Maxpool maximum pooling layer is connected to perform dimensionality reduction and downsampling operations;

[0010] The neck network of the welding defect target detection network is constructed using the CNN-based cross-scale feature fusion module of RT-CCFM and the attention-based intra-scale feature interaction module of RT-AIFI;

[0011] Based on the constructed backbone network and neck network, a decoding prediction network is constructed. A fixed number of qurey objects are selected from the feature tokens output by the encoder through query selection, and the final target detection prediction confidence and prediction bounding box are obtained through encoder mapping. The training process is optimized using IOU wake-up.

[0012] The welding operation recognition network is specifically: Vision-Transformer is used as the backbone network of the welding operation recognition network, and deconvolution operation is used for prediction in the Decoder decoder. The backbone network embeds the input features into the token through several patch embedding layers, and then is processed by several visual transformers. Each transformer consists of a multi-head self-attention module and a feedforward network FFN.

[0013] Furthermore, the aforementioned backbone network is specifically: the first Maxpool maximum pooling layer sequentially connects three EDS-Fire modules, and then connects the global-avapool global maximum pooling layer, and the output after the global maximum pooling operation is sent to the Softmax classifier to obtain the final output prediction result; the three EDS-Fire modules include the first EDS-Fire module, the second EDS-Fire module, and the third EDS-Fire module;

[0014] The first EDS-Fire module includes a first Fire unit, a second Fire unit, a third Fire unit, and a first EDSBottleNeck unit;

[0015] Among them, the input end of the first Fire unit is connected to the output end of the Maxpool maximum pooling layer, the output end of the first Fire unit is respectively connected to the input end of the second Fire unit and the input end of the first EDSBottleNeck unit, the output end of the second Fire unit is connected to the input end of the third Fire unit, the output end of the first EDSBottleNeck unit is connected to the input end of the third Fire unit, and the output end of the third Fire unit serves as the output end of the first EDS-Fire module.

[0016] The second EDS-Fire module includes a fourth Fire unit, a fifth Fire unit, a sixth Fire unit, a seventh Fire unit, a second EDSBottleNeck unit, a third EDSBottleNeck unit, and a second Maxpool layer;

[0017] Among them, the input ends of the fourth Fire unit and the second EDSBottleNeck unit are respectively used as the input end of the second EDS-Fire module and connected to the output end of the third Fire unit. The output end of the fourth Fire unit and the output end of the second EDSBottleNeck unit are connected to the input end of the fifth Fire unit. The output end of the fifth Fire unit is connected to the sixth Fire unit and the input end of the third EDSBottleNeck unit. The output end of the sixth Fire unit and the output end of the third EDSBottleNeck unit are connected to the input end of the seventh Fire unit. The output end of the seventh Fire unit is connected to the input end of the second Maxpool layer, and the output end of the second Maxpool layer serves as the output end of the second EDS-Fire module.

[0018] The third EDS-Fire module includes an eighth Fire unit, a ninth Fire unit, and a fourth EDSBottleNeck unit;

[0019] The input end of the eighth Fire unit and the input end of the fourth EDSBottleNeck unit are used as the input end of the third EDS-Fire module, connected to the output end of the second Maxpool layer, the output end of the eighth Fire unit and the output end of the fourth EDSBottleNeck unit are connected to the input end of the ninth Fire unit, and the output end of the ninth Fire unit is used as the output end of the third EDS-Fire module, connected to the global-avapool global maximum pooling layer.

[0020] Furthermore, the aforementioned Fire unit includes a three-channel Squeeze-Conv module and an Expand module, the Expand module includes four 1×1Conv convolution units and four 3×3Conv convolution units; the input feature map passes through the Squeeze-Conv module in turn to obtain the upsampled features, and then is input to the 1×1Conv convolution unit and the 3×3Conv convolution unit in the Expand module respectively, and then input to the BN_ReLU activation function.

[0021] Furthermore, the aforementioned EDS-Fire module performs feature calculation and outputs the final information-enhanced output features, which are specifically as follows:

[0022] ,

[0023] in, F c represents the original input feature map, MLP represents the multi-layer perceptron operation, F s Indicates passing Squeeze The feature map obtained by the operation, F e Indicates passing Expand The feature map obtained by the operation, BN_ReLU represents Batch-Normalization and ReLU activation function, EDS Conv represents the EDS convolution operation, F j Represents the output features of the final information enhancement.

[0024] Furthermore, the aforementioned neck network is specifically: the input multi-dimensional features are expanded into vectors at the RT-AIFI module, and then the features are interacted with the FFN feedforward neural network module through the multi-head attention mechanism, and then reshape processing is performed, and finally cross-scale feature fusion is performed through the RT-CCFM module. The calculation method is as follows:

[0025] ,

[0026] Among them, Q, K, V are the keys of the attention mechanism, MH-Attention is a whole, indicating a multi-head attention mechanism; upsample indicates an upsampling operation, Flatten indicates a vector extension operation, Reshape indicates a vector reorganization operation, and RTCCFM indicates the cross-scale feature fusion operation. S 3 , S 4 , S 5 represents the output features of the backbone network, where S 3 , S 4 These are shallow features. S 5 For deep features; Represents a vector reorganization operation; F output The feature map representing the output;

[0027] The RT-CCFM module is an FPN structure, built on the Fusion module. The RT-CCFM module consists of two branches. The first branch includes a 1×1Conv, and the second branch includes a 1×1Conv and a Repblock reparameterizable module. The calculation method of the Fusion module is as follows: ,

[0028] Among them, Concat represents the vector concatenation operation. Represents a vector reorganization operation, X is the input feature map, Y is the output feature map, Rep represents a reparameterizable operation; X1 and X2 represent two input features respectively.

[0029] Furthermore, the aforementioned decoding prediction network uses IOU wake-up to optimize the training process, and the calculation method is as follows:

[0030] ,

[0031] in, Loss is the loss function for training, Loss Box is the loss function of the output prediction box, Loss Cls is the loss function for output classification, are the predicted value and the true value, c is the output prediction category, b Output prediction box; represents the predicted category value, Represents the predicted box.

[0032] FFN is calculated as follows:

[0033] ,

[0034] in, x is the eigenvector, W 1 , W 2 for The weight value of the fully connected network composed of b 1 , b 2 is the offset.

[0035] Furthermore, the aforementioned method for visual monitoring of welding operation process and quality uses a Transformer encoder module to ensure that the network has sufficient nonlinear processing capabilities, and optimizes and improves the token embedding process. The specific optimization process is as follows:

[0036] ,

[0037] .

[0038] Improved network embedding process:

[0039] ,

[0040] in, E orig and E opt Represent the embedding vectors of the original and optimized networks, respectively. X p represents the patch vector of the input image after segmentation, and represent the original and optimized embedding matrices respectively, and They are the original bias term and the optimized bias term respectively.

[0041] Furthermore, the aforementioned welding operation recognition network extracts the hand key point coordinates of the welding operation in the video stream, uses the hand key point coordinate information to convert it into a key point heat map through Gaussian mapping, splices each frame of the heat map in the video stream along the time domain direction to obtain a heat map sequence, and realizes the conversion from 2D heat map to 3D heat map stacking. Finally, the 3D heat map sequence is input into the 3DCNN to extract the semantic information contained therein, obtain the action classification result, and realize the welding operation action recognition.

[0042] The present invention also provides a system for visual monitoring of welding operation process and quality, including: a welding workpiece image acquisition module, a welding video stream acquisition module, a model training module, a voice prompt and a normative evaluation module;

[0043] The welding workpiece image acquisition module is used to collect welding workpieces on the workbench and build a welding defect image dataset, including workpiece welding defect images and their corresponding defects.

[0044] The welding video stream acquisition module is used to collect the welding operation video data from the first-person perspective on site and construct a welding operation video dataset, including welding operation action videos and their corresponding operation category labels;

[0045] The model training module is used to build a welding defect target detection network, train the welding defect recognition network with the welding defect image data set to obtain the welding defect target detection model; build a welding operation key point action recognition network, train the welding operation recognition network with the welding operation video data set to obtain the welding operation key point action recognition model;

[0046] The welding defect target detection network specifically includes: backbone network, neck network, decoding prediction network, specifically: the backbone network of the welding defect target detection network is constructed using the EDS-SuqeezeNet network. The backbone network uses the Fire module as the basic connection module of each layer. The input is a Conv convolution unit. First, the rough feature information of the input welding workpiece image is extracted, and then the first Maxpool maximum pooling layer is connected to perform dimensionality reduction and downsampling operations;

[0047] The neck network of the welding defect target detection network is constructed using the CNN-based cross-scale feature fusion module of RT-CCFM and the attention-based intra-scale feature interaction module of RT-AIFI;

[0048] Based on the constructed backbone network and neck network, a decoding prediction network is constructed. A fixed number of qurey objects are selected from the feature tokens output by the encoder through query selection, and the final target detection prediction confidence and prediction bounding box are obtained through encoder mapping. The training process is optimized using IOU wake-up.

[0049] The welding operation key point action recognition network is specifically: Vision-Transformer is used as the backbone network of the welding operation recognition network, and deconvolution operation is used for prediction in the Decoder decoder. Its backbone network embeds the input features into the token through several patch embedding layers, and then is processed by several visual transformers. Each transformer consists of a multi-head self-attention module and a feedforward network FFN.

[0050] The voice prompt and standardization evaluation module is used to provide voice prompts and broadcast prompts for the defect categories of welding workpieces and welding operation standardization recognition results output by the welding defect target detection model and the welding operation key point action recognition model, as well as the welding operation standardization.

[0051] Compared with the prior art, the beneficial technical effects of the above technical solutions adopted in the present invention are as follows: the WELD-DETR target detection model and SLIM-ViTPose proposed in the present invention both have lightweight backbone networks, which greatly reduce the amount of network parameters and can be deployed and run on edge computing devices with limited computing power.

[0052] The EDS-SuqeezeNet target detection backbone network constructed by the present invention achieves a large degree of compression without losing too much accuracy, and adds EDS convolution between layers, which enhances global context interaction and makes the backward transmission of features richer. The SLIM-ViTPose model proposed by the present invention reduces the number of parameters while ensuring that the recognition accuracy does not decrease, and uses 24 Transformer encoder modules to ensure that the network has sufficient nonlinear processing capabilities. The method of the present invention can also detect and monitor the standardization of the worker's welding posture and detect the welding quality of the workpiece, thereby improving the production quality of the factory; through the image processing algorithm, the system can analyze the worker's welding posture to ensure that it complies with the standard operating procedures and reduce human operation errors. At the same time, the system can detect defects such as weld uniformity, welding spatter and weld fracture, and can be fed back to the worker for adjustment and operation backtracking after production. The quality of welded workpieces is effectively improved to ensure that each workpiece meets strict quality standards.

[0053] Through image processing algorithms, the system can analyze the worker's welding posture to ensure that it complies with standard operating procedures and reduce human operating errors. At the same time, the system can detect defects such as weld uniformity, weld spatter and weld fracture, and provide timely feedback to workers for adjustment. In this way, not only the quality of welded workpieces is effectively improved, but also production efficiency and consistency are greatly improved, ensuring that each workpiece meets strict quality standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The present invention is a flow chart of the method of the present invention.

[0055] Figure 2 This is the internal structure diagram of the Fire module in the WELD-DETR welding defect target detection backbone network.

[0056] Figure 3 This is the backbone network structure diagram of WELD-DETR welding defect target detection.

[0057] Figure 4 Neck network structure diagram for WELD-DETR welding defect target detection.

[0058] Figure 5 This is the detection structure flow chart of the SLIM-ViTPose welding action recognition network.

[0059] Figure 6 It is a schematic diagram of the detection process of the system of the present invention. DETAILED DESCRIPTION

[0060] In order to better understand the technical content of the present invention, specific embodiments are given and described as follows in conjunction with the accompanying drawings.

[0061] Various aspects of the invention are described herein with reference to the accompanying drawings, in which many illustrative embodiments are shown. The embodiments of the invention are not limited to those described in the accompanying drawings. It should be understood that the invention is implemented by any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed in the invention are not limited to any implementation. In addition, some aspects disclosed in the invention may be used alone or in any appropriate combination with other aspects disclosed in the invention.

[0062] refer to Figure 1 The present invention provides a method for visually monitoring welding operation process and quality, comprising the following steps:

[0063] S1. Construct a welding defect image dataset, including workpiece welding defect images and their corresponding defect category labels; construct a welding operation video dataset, including welding operation action videos and their corresponding operation category labels;

[0064] S2. Build a welding defect target detection network, and use the welding defect image data set to train the welding defect recognition network to obtain a welding defect target detection model; build a welding operation key point action recognition network, and use the welding operation video data set to train the welding operation recognition network to obtain a welding operation key point action recognition model;

[0065] S3. The real-time collected welding workpiece images and welding operation videos are input into the welding defect target detection model and the welding operation key point action recognition model respectively to obtain the defect category of the welding workpiece and the welding operation specification recognition results.

[0066] The welding defect target detection network specifically includes: backbone network, neck network, decoding prediction network, specifically: the backbone network of the welding defect WELD-DETR target detection network is constructed using the EDS-SuqeezeNet network. The backbone network uses the Fire module as the basic connection module of each layer, and the input is a 7×7 Conv convolution unit. First, the rough feature information of the input welding workpiece image is extracted, and then the first Maxpool maximum pooling layer is connected to perform dimensionality reduction and downsampling operations;

[0067] The Neck network of the welding defect WELD-DETR target detection network is constructed using the CNN-based cross-scale feature fusion module of RT-CCFM and the attention-based intra-scale feature interaction module of RT-AIFI;

[0068] Based on the constructed backbone network and neck network, a decoding prediction network is constructed. A fixed number of qurey objects are selected from the feature tokens output by the encoder through query selection, and the final target detection prediction confidence and prediction bounding box are obtained through encoder mapping. At the same time, in order to improve the accuracy of the detector, the prediction box is made closest to Groud-Truth, and the IOU wake-up method is used to optimize the training process;

[0069] The welding operation recognition network is specifically: Vision-Transformer is used as the backbone network of the welding operation recognition SLIM-ViTPose network. The traditional cross-attention mechanism is not used in the Decoder decoder, and deconvolution operation is used for prediction. Its backbone embeds the input features into the token through multiple patch embedding layers, and then is processed by several visual transformers. Each transformer consists of a multi-head self-attention module and a feedforward network FFN.

[0070] Figure 3 The backbone network of the welding defect recognition network is shown. Specifically, the first Maxpool layer sequentially connects three EDS-Fire modules, and then connects the global-avapool global maximum pooling layer. After the global maximum pooling operation, the output is sent to the Softmax classifier to obtain the final output prediction result. The EDSBottleNeck module includes an EDS-Conv and BottleNeck residual bottleneck blocks.

[0071] The three EDS-Fire modules include a first EDS-Fire module, a second EDS-Fire module, and a third EDS-Fire module; the first EDS-Fire module includes a first Fire unit, a second Fire unit, a third Fire unit, and a first EDSBottleNeck unit;

[0072] Among them, the input end of the first Fire unit is connected to the output end of the Maxpool maximum pooling layer, the output end of the first Fire unit is respectively connected to the input end of the second Fire unit and the input end of the first EDSBottleNeck unit, the output end of the second Fire unit is connected to the input end of the third Fire unit, the output end of the first EDSBottleNeck unit is connected to the input end of the third Fire unit, and the output end of the third Fire unit serves as the output end of the first EDS-Fire module;

[0073] The second EDS-Fire module includes a fourth Fire unit, a fifth Fire unit, a sixth Fire unit, a seventh Fire unit, a second EDSBottleNeck unit, a third EDSBottleNeck unit, and a second Maxpool layer;

[0074] Among them, the input ends of the fourth Fire unit and the second EDSBottleNeck unit are respectively used as the input end of the second EDS-Fire module and connected to the output end of the third Fire unit. The output end of the fourth Fire unit and the output end of the second EDSBottleNeck unit are connected to the input end of the fifth Fire unit. The output end of the fifth Fire unit is connected to the sixth Fire unit and the input end of the third EDSBottleNeck unit. The output end of the sixth Fire unit and the output end of the third EDSBottleNeck unit are connected to the input end of the seventh Fire unit. The output end of the seventh Fire unit is connected to the input end of the second Maxpool layer, and the output end of the second Maxpool layer serves as the output end of the second EDS-Fire module.

[0075] The third EDS-Fire module includes an eighth Fire unit, a ninth Fire unit, and a fourth EDSBottleNeck unit;

[0076] The input end of the eighth Fire unit and the input end of the fourth EDSBottleNeck unit are used as the input end of the third EDS-Fire module, connected to the output end of the second Maxpool layer, the output end of the eighth Fire unit and the output end of the fourth EDSBottleNeck unit are connected to the input end of the ninth Fire unit, and the output end of the ninth Fire unit is used as the output end of the third EDS-Fire module, connected to the global-avapool global maximum pooling layer.

[0077] Each EDS-Fire module performs feature calculation and outputs the final information-enhanced output features, as shown in the following formula:

[0078] ,

[0079] in, F c represents the original input feature map, MLP represents the multi-layer perceptron operation, F s Indicates passing Squeeze The feature map obtained by the operation, F e Indicates passing Expand The feature map obtained by the operation, BN_ReLU represents Batch-Normalization and ReLU activation function, EDS Conv represents the EDS convolution operation, F j Represents the output features of the final information enhancement.

[0080] like Figure 2 As shown in the figure, the specific structure of the Fire unit is shown. The Fire unit includes a three-channel Squeeze-Conv module and an Expand module. The Expand module includes four 1×1Conv convolution units and four 3×3Conv convolution units. The input feature map passes through the Squeeze-Conv module in turn to obtain the upsampled features, and then is input to the 1×1Conv convolution unit and the 3×3Conv convolution unit in the Expand module respectively, and then input to the BN_ReLU activation function.

[0081] like Figure 4 As shown in the figure, the neck network is specifically: the input multi-dimensional features are expanded into vectors at the RT-AIFI module, and then the features are interacted with the FFN feedforward neural network module through the multi-head attention mechanism, and then reshaped, and finally the cross-scale feature fusion is performed through the RT-CCFM module. The calculation method is as follows: ,

[0082] Among them, Q, K, V are the keys of the attention mechanism, MH-Attention is a whole, indicating a multi-head attention mechanism; upsample indicates an upsampling operation, Flatten indicates a vector extension operation, Reshape indicates a vector reorganization operation, and RTCCFM indicates the cross-scale feature fusion operation. S 3 , S 4 , S 5 represents the output features of the backbone network, where S 3, S 4 These are shallow features. S 5 It is a deep feature with more semantic information; Represents a vector reorganization operation; F output Represents the output feature map;.

[0083] The RT-CCFM module is an FPN structure, built on the Fusion module. The RT-CCFM module consists of two branches. The first branch includes a 1×1Conv, and the second branch includes a 1×1Conv and a Repblock reparameterizable module. The calculation method of the Fusion module is as follows:

[0084] ,

[0085] Among them, Concat represents the vector concatenation operation. Represents a vector reorganization operation, X is the input feature map, Y is the output feature map, X1 and X2 represent two input features respectively; Rep represents a reparameterizable operation.

[0086] The decoding prediction network uses IOU wake-up to optimize the training process. The calculation method is as follows:

[0087] ,

[0088] in, Loss is the loss function for training, Loss Box is the loss function of the output prediction box, Loss Cls is the loss function for output classification, are the predicted value and the true value, c is the output prediction category, b is the output prediction box, represents the predicted category value, Represents the predicted box.

[0089] FFN is calculated as follows:

[0090] ,

[0091] in, x is the eigenvector, W 1 , W 2 for The weight value of the fully connected network composed of b 1 , b 2 is the offset.

[0092] refer to Figure 5 The welding operation recognition network extracts the hand key point coordinates of the welding operation in the video stream, and uses the hand key point coordinate information to convert it into a key point heat map through Gaussian mapping. Each frame of the heat map in the video stream is spliced ​​along the time domain to obtain a heat map sequence, realizing the conversion from 2D heat map to 3D heat map stacking. Finally, the 3D heat map sequence is input into the 3DCNN to extract the semantic information contained therein, and the action classification result is obtained to realize the welding operation action recognition.

[0093] The welding operation recognition SLIM-ViTPose network reduces the embedding dimension of the standard Vision Transformer model from 768 to 384, and halves the number of channels of the model accordingly, making the number of parameters and FLOPs of the model about one-quarter of the original standard model (ViT-B). It uses 24 Transformer encoder modules to ensure that the network retains sufficient nonlinear processing capabilities and optimizes and improves the token embedding process. The specific optimization process is as follows:

[0094] ,

[0095] .

[0096] Improved network embedding process:

[0097] ,

[0098] in, E orig and E opt Represent the embedding vectors of the original and optimized networks, respectively. X p represents the patch vector of the input image after segmentation, and represent the original and optimized embedding matrices respectively, and They are the original bias term and the optimized bias term respectively.

[0099] like Figure 6 As shown, the present invention provides a system for visually monitoring welding operation procedures and quality, including: a welding workpiece image acquisition module, a welding video stream acquisition module, a model training module, a voice prompt and a normative evaluation module;

[0100] The welding workpiece image acquisition module is used to collect welding workpieces on the workbench and build a welding defect image dataset, including workpiece welding defect images and their corresponding defects.

[0101] The welding video stream acquisition module is used to collect the welding operation video data from the first-person perspective on site and construct a welding operation video data set, including welding operation action videos and their corresponding operation category labels; the welding operation video data from the first-person perspective on site can be collected using a collection device worn on the operator's head;

[0102] The model training module is used to build a welding defect target detection network, train the welding defect recognition network with the welding defect image data set to obtain the welding defect target detection model; build a welding operation key point action recognition network, train the welding operation recognition network with the welding operation video data set to obtain the welding operation key point action recognition model;

[0103] The welding defect target detection network specifically includes: backbone network, neck network, decoding prediction network, specifically: the backbone network of the welding defect target detection network is constructed using the EDS-SuqeezeNet network. The backbone network uses the Fire module as the basic connection module of each layer. The input is a Conv convolution unit. First, the rough feature information of the input welding workpiece image is extracted, and then the first Maxpool maximum pooling layer is connected to perform dimensionality reduction and downsampling operations;

[0104] The neck network of the welding defect target detection network is constructed using the CNN-based cross-scale feature fusion module of RT-CCFM and the attention-based intra-scale feature interaction module of RT-AIFI;

[0105] Based on the constructed backbone network and neck network, a decoding prediction network is constructed. A fixed number of qurey objects are selected from the feature tokens output by the encoder through query selection, and the final target detection prediction confidence and prediction bounding box are obtained through encoder mapping. The training process is optimized using IOU wake-up.

[0106] The welding operation key point action recognition network is specifically: Vision-Transformer is used as the backbone network of the welding operation recognition network, and deconvolution operation is used for prediction in the Decoder decoder. Its backbone network embeds the input features into the token through several patch embedding layers, and then is processed by several visual transformers. Each transformer consists of a multi-head self-attention module and a feedforward network FFN.

[0107] The voice prompt and standardization evaluation module is used to provide voice prompts and broadcast prompts for the defect categories of welding workpieces and welding operation standardization recognition results output by the welding defect target detection model and the welding operation key point action recognition model, as well as the welding operation standardization.

[0108] In this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk, etc.

[0109] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0110] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application.

[0111] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0112] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0115] Although the present invention has been described above with preferred embodiments, it is not intended to limit the present invention. A person skilled in the art of the present invention may make various modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the definition of the claims.

Claims

1. A method for visually monitoring welding operation process and quality, characterized in that: The following steps are involved: S1. Construct a welding defect image dataset, including workpiece welding defect images and their corresponding defect category labels; construct a welding operation video dataset, including welding operation action videos and their corresponding operation category labels; S2. Build a welding defect target detection network, and use the welding defect image data set to train the welding defect recognition network to obtain a welding defect target detection model; build a welding operation key point action recognition network, and use the welding operation video data set to train the welding operation key point action recognition network to obtain a welding operation key point action recognition model; The welding defect target detection network specifically includes: backbone network, neck network, decoding prediction network, specifically: the backbone network of the welding defect target detection network is constructed using the EDS-SuqeezeNet network. The backbone network uses the Fire module as the basic connection module of each layer. The input is a Conv convolution unit. First, the rough feature information of the input welding workpiece image is extracted, and then the first Maxpool maximum pooling layer is connected to perform dimensionality reduction and downsampling operations; The neck network of the welding defect target detection network is constructed using the CNN-based cross-scale feature fusion module of RT-CCFM and the attention-based intra-scale feature interaction module of RT-AIFI; Based on the constructed backbone network and neck network, a decoding prediction network is constructed. A fixed number of qurey objects are selected from the feature tokens output by the encoder through query selection, and the final target detection prediction confidence and prediction bounding box are obtained through encoder mapping. The training process is optimized using IOU wake-up. The welding operation key point action recognition network is specifically: using Vision-Transformer as the backbone network of the welding operation recognition network, using deconvolution operation for prediction in the Decoder decoder, and its backbone network embeds input features into tokens through several patch embedding layers, and then is processed by several visual transformers, each of which consists of a multi-head self-attention module and a feed-forward network FFN; S3. The real-time collected welding workpiece images and welding operation videos are input into the welding defect target detection model and the welding operation key point action recognition model respectively to obtain the defect category of the welding workpiece and the welding operation specification recognition results.

2. A method for visually monitoring welding operation and quality according to claim 1, characterized in that: The backbone network is specifically as follows: the first Maxpool layer sequentially connects three EDS-Fire modules, and then connects the global-avapool global maximum pooling layer. After the global maximum pooling operation, the output is sent to the Softmax classifier to obtain the final output prediction result; the three EDS-Fire modules include the first EDS-Fire module, the second EDS-Fire module, and the third EDS-Fire module; The first EDS-Fire module includes a first Fire unit, a second Fire unit, a third Fire unit, and a first EDSBottleNeck unit; Among them, the input end of the first Fire unit is connected to the output end of the Maxpool maximum pooling layer, the output end of the first Fire unit is respectively connected to the input end of the second Fire unit and the input end of the first EDSBottleNeck unit, the output end of the second Fire unit is connected to the input end of the third Fire unit, the output end of the first EDSBottleNeck unit is connected to the input end of the third Fire unit, and the output end of the third Fire unit serves as the output end of the first EDS-Fire module; The second EDS-Fire module includes a fourth Fire unit, a fifth Fire unit, a sixth Fire unit, a seventh Fire unit, a second EDSBottleNeck unit, a third EDSBottleNeck unit, and a second Maxpool layer; Among them, the input ends of the fourth Fire unit and the second EDSBottleNeck unit are respectively used as the input end of the second EDS-Fire module and connected to the output end of the third Fire unit. The output end of the fourth Fire unit and the output end of the second EDSBottleNeck unit are connected to the input end of the fifth Fire unit. The output end of the fifth Fire unit is connected to the sixth Fire unit and the input end of the third EDSBottleNeck unit. The output end of the sixth Fire unit and the output end of the third EDSBottleNeck unit are connected to the input end of the seventh Fire unit. The output end of the seventh Fire unit is connected to the input end of the second Maxpool layer, and the output end of the second Maxpool layer serves as the output end of the second EDS-Fire module. The third EDS-Fire module includes an eighth Fire unit, a ninth Fire unit, and a fourth EDSBottleNeck unit; The input end of the eighth Fire unit and the input end of the fourth EDSBottleNeck unit are used as the input end of the third EDS-Fire module, connected to the output end of the second Maxpool layer, the output end of the eighth Fire unit and the output end of the fourth EDSBottleNeck unit are connected to the input end of the ninth Fire unit, and the output end of the ninth Fire unit is used as the output end of the third EDS-Fire module, connected to the global-avapool global maximum pooling layer.

3. A method for visually monitoring welding operation and quality according to claim 2, characterized in that: The Fire unit includes a three-channel Squeeze-Conv module and an Expand module. The Expand module includes four 1×1Conv convolution units and four 3×3Conv convolution units. The input feature map passes through the Squeeze-Conv module in turn to obtain the upsampled features, and then is input to the 1×1Conv convolution unit and 3×3Conv convolution unit in the Expand module respectively, and then input to the BN_ReLU activation function.

4. A method for visually monitoring welding operation and quality according to claim 3, characterized in that: The EDS-Fire module performs feature calculation and outputs the final information-enhanced output features, as shown in the following formula: ， in F c represents the original input feature map, MLP represents the multi-layer perceptron operation, F s Indicates passing Squeeze The feature map obtained by the operation, F e Indicates passing Expand The feature map obtained by the operation, BN_ReLU represents Batch-Normalization and ReLU activation function, EDS Conv represents the EDS convolution operation, F j Represents the output features of the final information enhancement.

5. A method for visually monitoring welding operation and quality according to claim 1, characterized in that: The neck network is as follows: the input multi-dimensional features are expanded into vectors at the RT-AIFI module, and then the features are interacted with the FFN feedforward neural network module through the multi-head attention mechanism, and then reshaped, and finally the cross-scale feature fusion is performed through the RT-CCFM module. The calculation method is as follows: ， Among them, Q, K, V are the keys of the attention mechanism, MH-Attention is a whole, indicating a multi-head attention mechanism; upsample indicates an upsampling operation, Flatten indicates a vector extension operation, Reshape indicates a vector reorganization operation, and RTCCFM indicates the cross-scale feature fusion operation. S 3 , S 4 , S 5 represents the output features of the backbone network, where S 3 , S 4 These are shallow features. S 5 For deep features; Represents vector reorganization operation; F output The feature map representing the output; The RT-CCFM module is an FPN structure and is built on the Fusion module. The RT-CCFM module consists of two branches. The first branch includes a 1×1Conv, and the second branch includes a 1×1Conv and a Repblock reparameterizable module. The calculation method of the Fusion module is as follows: , Among them, Concat represents the vector concatenation operation. Represents a vector reorganization operation, X is the input feature map, Y is the output feature map, X1 and X2 represent two input features respectively; Rep represents a reparameterizable operation.

6. A method for visually monitoring welding operation and quality according to claim 1, characterized in that: The decoding prediction network uses IOU wake-up to optimize the training process. The calculation method is as follows: ， in, Loss is the loss function for training, Loss Box is the loss function of the output prediction box, Loss Cls is the loss function for output classification, are the predicted value and the true value, c is the output prediction category, b is the output prediction box, represents the predicted category value, The prediction box represents the prediction; FFN is calculated as follows: ， in, x is the eigenvector, W 1 , W 2 for The weight value of the fully connected network composed of b 1 , b 2 is the offset.

7. A method for visually monitoring welding operation and quality according to claim 1, characterized in that: The Transformer encoder module is used to ensure that the network has sufficient nonlinear processing capabilities and to optimize and improve the token embedding process. The specific optimization process is as follows: ， ， Improved network embedding process: ， in, E orig and E opt Represent the embedding vectors of the original and optimized networks, respectively. X p represents the patch vector of the input image after segmentation, and represent the original and optimized embedding matrices respectively, and They are the original bias term and the optimized bias term respectively.

8. A method for visually monitoring welding operation and quality according to claim 7, characterized in that: The welding operation recognition network extracts the hand key point coordinates of the welding operation in the video stream, and uses the hand key point coordinate information to convert it into a key point heat map through Gaussian mapping. Each frame of the heat map in the video stream is spliced ​​along the time domain to obtain a heat map sequence, thereby realizing the conversion from 2D heat map to 3D heat map stacking. Finally, the 3D heat map sequence is input into the 3DCNN to extract the semantic information contained therein, obtain the action classification result, and realize the welding operation action recognition.

9. A system for visually monitoring welding operation and quality, characterized in that: include: Welding workpiece image acquisition module, welding video stream acquisition module, model training module, voice prompt and normative evaluation module; The welding workpiece image acquisition module is used to collect welding workpieces on the workbench and build a welding defect image dataset, including workpiece welding defect images and their corresponding defects. The welding video stream acquisition module is used to collect the welding operation video data from the first-person perspective on site and construct a welding operation video dataset, including welding operation action videos and their corresponding operation category labels; The model training module is used to build a welding defect target detection network, and use the welding defect image data set to train the welding defect recognition network to obtain the welding defect target detection model; build a welding operation key point action recognition network, and use the welding operation video data set to train the welding operation recognition network to obtain the welding operation key point action recognition model; the welding defect target detection network specifically includes: a backbone network, a neck network, and a decoding prediction network, specifically: use the EDS-SuqeezeNet network to build the backbone network of the welding defect target detection network, the backbone network uses the Fire module as the basic connection module of each layer, and the input is a Conv convolution unit to first extract the rough feature information of the input welding workpiece image, and then connect the first Maxpool maximum pooling layer to perform dimensionality reduction and downsampling operations; The neck network of the welding defect target detection network is constructed using the CNN-based cross-scale feature fusion module of RT-CCFM and the attention-based intra-scale feature interaction module of RT-AIFI; Based on the constructed backbone network and neck network, a decoding prediction network is constructed. A fixed number of qurey objects are selected from the feature tokens output by the encoder through query selection, and the final target detection prediction confidence and prediction bounding box are obtained through encoder mapping. The training process is optimized using IOU wake-up. The welding operation key point action recognition network is specifically: using Vision-Transformer as the backbone network of the welding operation recognition network, using deconvolution operation for prediction in the Decoder decoder, and its backbone network embeds input features into tokens through several patch embedding layers, and then passes through several visual transformers for processing, each transformer consisting of a multi-head self-attention module and a feed-forward network FFN; The voice prompt and standardization evaluation module is used to provide voice prompts and broadcast prompts for the defect categories of welding workpieces and welding operation standardization recognition results output by the welding defect target detection model and the welding operation key point action recognition model, as well as the welding operation standardization.

Images