Special vehicle welding state intelligent identification method and identification system based on multi-modal fusion and deep learning

Abstract

The invention provides a special vehicle welding state intelligent identification method and identification system based on multi-modal fusion and deep learning. According to the method, three improved R-ShuffleNet v2 network models are used for processing a welding current two-dimensional image (converted through a Grubrum angle field method), an electric arc image, a molten pool image and a heat radiation image respectively. Each model outputs probability density distribution of different weld defects, the distribution serves as three evidence sources of an evidence theory, evidence fusion is conducted through the improved evidence theory, and finally a classification result is obtained. Compared with the prior art, the accuracy and reliability of weld defect classification can be remarkably improved, and meanwhile efficient calculation performance is kept.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of welding technology for special vehicles, and particularly relates to a welding state intelligent identification method for a welding process. BACKGROUND

[0002] In the welding process, the current size has a direct impact on the weld quality: too small current is easy to cause defects such as incomplete fusion, incomplete penetration, porosity, etc., and the root of the weld is not melted and the penetration is shallow due to insufficient heating, the molten pool is weakly stirred, and the gas is difficult to escape; too large current is easy to cause problems such as weld collapse and undercut, and the arc blowing force is too large to cause the molten metal to overflow or the edge of the welding part to be melted too much. When these defects occur, they will have a feedback effect on the current: incomplete fusion and incomplete penetration reduce the welding cross-sectional area, increase the resistance, and reduce the current; porosity reduces the electrical conductivity, resulting in uneven current distribution and reduced transmission efficiency; weld collapse changes the welding cross-sectional area and current path, causing changes in current density distribution.

[0003] At the same time, these defects also present different characteristics in the molten pool image, arc image and thermal radiation image: when incomplete fusion occurs, the molten pool image boundary is not clear, the arc image is jittered, and the thermal radiation image temperature distribution is uneven; when incomplete penetration occurs, the molten pool image geometry is small, the arc image has an un-melted area at the root of the weld, and the thermal radiation image has a low temperature and dark un-melted area; when porosity occurs, the molten pool image has small black spots, the arc image has abnormal light spots or flickering, and the thermal radiation image has a slightly lower temperature at the porosity area, appearing as a dark spot; when weld collapse occurs, the molten pool image has a low gray round area at the tail, the arc image has an abnormally elongated or swinging arc, and the thermal radiation image has an abnormal temperature at the weld collapse area. These image characteristics provide an important basis for monitoring and preventing weld defects.

[0004] To solve the above problems, the present application proposes a special vehicle welding state intelligent identification method based on multi-modal fusion and deep learning. This method uses three improved R-ShuffleNet v2 network models to process the welding current two-dimensional image (converted by the Gram angle field method) and the arc image, the molten pool image and the thermal radiation image, respectively. Each model outputs the probability density distribution of different weld defects, which are used as three evidence sources of the evidence theory, and the evidence is fused by the improved evidence theory to obtain the final classification result. Through this method based on the combination of deep learning and evidence theory, the accuracy and reliability are ensured, and the defect state in the welding process can be effectively identified and evaluated, which can be migrated to new welding tasks, thereby realizing accurate detection of the welding quality and reliability of titanium alloy thick plate, and providing strong technical support for the guarantee of welding quality. SUMMARY

[0005] The application provides a special vehicle welding state intelligent identification method and system based on multi-modal fusion and deep learning to solve the technical problems mentioned in the background art. The application provides a special vehicle welding state intelligent identification method based on multi-modal fusion and deep learning, comprising the following steps: S1. In the welding test process, weld defects are generated by systematically changing the welding current, welding speed and wire feeding speed, and welding current, molten pool image, arc image and thermal radiation image signals in the welding process are collected to construct a weld defect dataset; S2. The welding current one-dimensional time series data is converted into a welding current two-dimensional image using the Gram angle field method. The welding current two-dimensional image, molten pool image, arc image and thermal radiation image signal are subjected to operations such as cropping, padding or adjusting resolution to make the sizes of the images the same, and then subjected to filtering and data enhancement processing. The welding current two-dimensional image, molten pool image, arc image and thermal radiation image signal are divided into a training set, a validation set and a test set; S3. Three R-Shufflenetv2 network models are constructed, and the R-Shufflenetv2 network model is an improved weld defect classification model. The R-Shufflenetv2 network model comprises a focus module (Foucs), stage2, stage3, stage4, a convolution layer, a standardization, an activation function, a DPSA module, an average pooling layer, a full connection layer and a Softmax output layer; S4. The welding current two-dimensional image and the molten pool image are input into the first R-Shufflenetv2 model for training and verification. The welding current two-dimensional image and the arc image are input into the first R-Shufflenetv2 model for training and verification. The welding current two-dimensional image and the thermal radiation image are input into the first R-Shufflenetv2 model for training and verification; S5. The three R-Shufflenetv2 models trained are optimized using a knowledge distillation technology; S6. The three R-Shufflenetv2 weld defect classification models are used as three evidences of the evidence theory, the weld state probability distribution output by the three R-Shufflenetv2 weld defect classification models is used as a corresponding basic belief degree allocation function, a conflict evidence fusion algorithm in the improved D-S evidence theory is used to fuse the three evidences to determine the final weld state classification result.

[0006] Preferably, S1 specifically comprises the following steps: S11. An industrial camera is selected as a visual sensor, and a high-precision welding current sensor is used. S12, using a high-precision welding current sensor to collect welding current signals, obtaining continuous current time sequence signals, and using a high-speed camera to collect molten pool images, arc images and thermal radiation images respectively; Preferably, the S2 converts the one-dimensional time sequence data of the welding current into a two-dimensional image of the welding current by using the Gram angular field method, and specifically includes the following steps: S21, standardizing the welding current, and normalizing the current time sequence signal to the interval [-1, 1]:

[0007] S22, polar coordinate mapping, converting the standardized value into a polar coordinate angle:

[0008] In the formula, t i represents the time index, N is the sequence length.

[0009] S23, GASF image conversion, capturing the in-phase cooperative variation characteristics of the sequence through the angle and cosine value, and the formula is as follows:

[0010] Preferably, the R-Shufflenetv2 network model in the S3 takes the Shufflenetv2 network model as a baseline model; the Shufflenetv2 network model is an original classification model, and the Shufflenetv2 network model includes a convolution layer 1, a maximum pooling layer, a stage2, a stage3, a stage4, a convolution layer 2, a standardization, an activation function, an average pooling layer, a full connection layer and an output layer connected in sequence.

[0011] Wherein, the R-Shufflenetv2 network model is relative to the Shufflenetv2 network model, the initial convolution layer of the Shufflenetv2 network model is exchanged into a focus module (Foucs), the Foucs block can play a role in rapidly increasing the number of network channels without losing image information, and an ECA attention mechanism module is added after stage2; in order to realize multi-scale feature fusion while ensuring that the inference speed does not slow down, in the R-Shufflenetv2 network model, the feature extraction blocks in stage3 and stage4 of the Shufflenetv2 network model are changed into a re-parameterized reshaping convolution (RCR) module, and an ECA attention mechanism module is added after stage3 and stage4; in order to retain key feature information while reducing the parameter amount, a DPSA module is added after the convolution layer 2; a phased fusion strategy is adopted, and the improved R-Shufflenetv2 network model is used for independent feature extraction of two types of input images, and then fusion feature extraction is performed.

[0012] The DPSA module first reduces the dimension of the output features of stage4 through a 3x3 maximum pooling layer (Maxpool) with a step of 2, and then passes through a channel mean (CMean) module. The module takes the average of the feature values of every two adjacent channels, specifically, the CMean module calculates the average value of the input features of the i-th and i+1-th channels, and takes it as the input feature of the i / 2-th channel. This design effectively compresses the input features from three dimensions of width, height and channel while retaining key feature information, significantly reducing the parameter amount of the subsequent module.

[0013] Further, the improved R-Shufflenetv2 network model adopts a phased fusion step, which includes: S31, inputting two images into respective focus modules (Foucs); S32, inputting the results of S31 into stage2 to preliminarily extract features of the two images to stage2; S33, splicing the feature maps of the two images in the channel dimension; S34, inputting the fused feature map into the subsequent stage3 and stage4; S35, inputting the fused feature map extracted by stage4 through the convolution layer 2, the DPSA module, the standardization, the activation function, the average pooling layer, the fully connected layer and the Softmax output layer for classification.

[0014] Preferably, the step S4 specifically includes the following steps: Step S41: set training parameters, use Adam algorithm for optimization training, set parameters including training required hyperparameter setting, batch training data amount, initial learning rate, weight decay coefficient, training momentum; Step S42: put the weld defect training data set and the verification data set divided in step S13 into three improved R-Shufflenetv2 weld defect classification models for training, the input of the first improved R-Shufflenetv2 weld defect classification model is the welding current two-dimensional graph and the arc image, the input of the second improved R-Shufflenetv2 weld defect classification model is the welding current two-dimensional graph and the molten pool image, and the input of the third improved R-Shufflenetv2 weld defect classification model is the welding current two-dimensional graph and the thermal radiation image; Step S43: train the model according to the set training parameters, adjust the learning rate and the number of iterations of the model training by observing the change trend of the loss function, until the change trend of the loss function tends to be stable, and obtain the trained weld defect classification model.

[0015] Preferably, the step S5 specifically comprises the following steps: S51, taking the Resnet50 model as a teacher model, and taking the three R-Shufflenetv2 network models trained as student models; S52, training the three teacher models by using the weld defect data set to obtain the trained teacher models; S53, selecting two scene images from the weld defect data set and inputting them into the three trained teacher models and the three trained student models respectively to obtain the outputs of the three teacher models and the student models, and then calculating the loss of the corresponding teacher model and student model by using the outputs of the three teacher models and student models, thereby obtaining the loss function of the three teacher models L 蒸馏 , and the loss function of the three student models L 原学生模型 ; Wherein, the calculation formula of the loss function of the teacher model is as follows:

[0016] In the formula, represents the KL divergence, represents the smoothed value of the i-th teacher neural network, represents the smoothed value of the i-th student neural network; S54. Knowledge distillation is used to distill the trained student model, and the distillation balance factor is calculated. The distillation balance factor is the proportion of influence of knowledge distillation during the training process of the student model. Then, the loss function of the student model after distillation is constructed. L 学生模型 The specific calculation formula is as follows:

[0017] The formula for knowledge distillation is as follows:

[0018] In the formula, z i Indicates the first i The value of Logits, where Logits represents the output of the last layer of the teacher model. T Indicates the distillation temperature. q i Exp represents the probability value, and exp represents the natural exponential function. S55, repeat S53 to S54 until the loss function after student model distillation converges, and update the weight parameters of the trained student model to obtain the optimized R-Shufflenetv2 model. Preferably, step S6 specifically includes the following steps: S61. Define the identification framework Θ, which contains six complete and mutually exclusive hypothetical commands: normal, incomplete fusion, incomplete penetration, porosity, weld collapse, and undercut. Its power set is 2. Θ ={A1,A2,A3,A4,A5,A6}; S62. The three optimized R-Shufflenetv2 weld defect classification models from S5 are used as three pieces of evidence in the evidence theory. The three pieces of evidence are defined as follows: E 1, E 2, E 3, then E 1, E 2, E The basic trust assignment functions for 3 are as follows: m 1, m 2, m 3. The probability distributions of weld states (normal, incomplete fusion, incomplete penetration, porosity, weld collapse, and undercut) output by the three R-Shufflenetv2 weld defect classification models are used as the basic confidence assignment function. m 1, m 2, m The basic probability assignment value of 3; S63, using improved D-S evidence theory conflict evidence fusion algorithm, first to pre-process the evidence, and then apply D-S evidence synthesis rule for multi-source information fusion; Further, the improved D-S evidence theory conflict evidence fusion algorithm in S63 is an improvement on Murphy method, a new similarity function is used to represent the similarity of evidence, and the support degree of evidence is obtained through normalization processing, D-S evidence synthesis rule is applied for multi-source information fusion, the specific steps are as follows: (1) calculate the average value of each evidence , the formula is as follows:

[0019] (2) solve the evidence E i (A k ) and the similarity function of the average value of evidence

[0020] (3) solve the support degree coefficient , the formula is:

[0021] In the formula, The sum of and is 1, which can be used as the weight of m i ; (4) take the support degree coefficient of evidence As the correction weight of the corresponding focus element, correct it as follows:

[0022] (5) normalize , get the final synthesis result , the formula is:

[0023] (6) replace the original With , using Dempster combination rule, first fuse In turn, with this result again and Fusion, with this result and Fusion three times, Dempster combination rule is as follows:

[0024] ​The application provides a special vehicle welding state intelligent identification system based on multi-modal fusion and deep learning, adopts the special vehicle welding state intelligent identification method based on multi-modal fusion and deep learning to realize feature vehicle welding state identification, and the system comprises a data acquisition module, a data preprocessing module, a feature extraction and prediction module and an improved evidence theory decision layer fusion module. The data acquisition module acquires welding current signals through a current sensor, and acquires thermal radiation images, molten pool images and arc images through an infrared thermal imager and a high-speed camera device respectively, and captures welding part temperature distribution, molten pool dynamics and arc morphology information in the welding process; The data preprocessing module converts one-dimensional time sequence data of the welding current into a two-dimensional image by using a Gram angle field method, filters and performs data enhancement processing on the molten pool image, the arc image and the thermal radiation image signal, and divides a training set, a verification set and a test set; The feature extraction and prediction module constructs three R-Shufflenetv2 network models, sequentially performs training, verification and testing, and then optimizes training by using a knowledge distillation technology; The improved evidence theory decision layer fusion module defines an identification framework Θ, the identification framework Θ comprises six complete mutually incompatible hypothesis commands, namely normal, non-fusion, non-penetration, porosity, weld collapse and undercut, the power set 2Θ is {A1, A2, A3, A4, A5, A6}, the three R-Shufflenetv2 weld defect classification models optimized by the knowledge distillation are used as three pieces of evidence of the evidence theory, the three pieces of evidence are defined as E1, E2 and E3, the basic belief degree distribution functions of E1, E2 and E3 are m1, m2 and m3 respectively, and the probability distribution of the welding state (normal, non-fusion, non-penetration, porosity, weld collapse and undercut) output by the three R-Shufflenetv2 weld defect classification models is used as the basic probability distribution value of the basic belief degree distribution functions m1, m2 and m3, then the improved conflict evidence fusion algorithm in the D-S evidence theory is used to pre-process the evidence, and then the D-S evidence synthesis rule is applied to multi-source information fusion; Compared with the prior art, the application has the beneficial effects that: The welding current one-dimensional signal is converted into a two-dimensional image through the Gram angle field method, time sequence data can be converted into an image, and the time and frequency information of the original signal can be retained; secondly, the original ShuffleNet v2 network structure is optimized by combining the welding current two-dimensional image and different types of images (arc image, molten pool image, and thermal radiation image), an improved R-Shufflenetv2 welding defect classification model is obtained, the image features of the welding current two-dimensional image and different types of images (arc image, molten pool image, and thermal radiation image) are further extracted, and the calculation efficiency and classification performance are improved; finally, the evidence theory is improved, and the improved evidence theory is used to integrate the outputs of multiple models, and the reliability and accuracy of classification are improved.

[0025] The present application proposes a welding defect classification method based on multi-modal data fusion and evidence fusion. The method includes three improved R-Shufflenetv2 network models, which process welding current two-dimensional images (converted by the Gram angle field method) and arc images, molten pool images, and thermal radiation images, respectively. Each model outputs the probability density distribution of different welding defects, which serves as the three evidence sources of the evidence theory, and the evidence fusion is performed through the improved evidence theory to obtain the final classification result. The present application can significantly improve the accuracy and reliability of welding defect classification while maintaining high computational performance. The system and method are expected to be further expanded to more welding scenarios and defect types, and developed into a real-time detection system to provide stronger support for welding quality control. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a flowchart of a special vehicle welding state intelligent identification method based on multi-modal fusion and deep learning of the present application.

[0027] Figure 2 is a structure diagram of the improved R-Shufflenetv2 network model of the present application.

[0028] Figure 3 is a RepConv in the Reparameterized Reshaped Convolution (RCR) module of the improved R-Shufflenetv2 network model of the present application.

[0029] Figure 4 is a RepConv in the Reparameterized Reshaped Convolution (RCR) module of the improved R-Shufflenetv2 network model of the present application.

[0030] Figure 4 in the middle: (a) training a multi-branch structure; (b) convolution and batch layer fusion; (c) single convolution structure for inference.

[0031] Figure 5The DPSA module of the improved R-Shufflenetv2 network model is the application.

[0032] Figure 6 The application is a special vehicle welding state intelligent identification system based on multi-modal fusion and deep learning. DETAILED DESCRIPTION

[0033] The application will be further described below with reference to the accompanying drawings Figures 1-6 The application will be further described below with reference to the accompanying drawings

[0034] The application will be further described below with reference to the accompanying drawings Figure 1 A special vehicle welding state intelligent identification method based on multi-modal fusion and deep learning, comprising the following steps: S1. In the welding test process, by systematically changing the welding current, welding speed and wire feeding speed to generate weld defects, collecting the welding current, molten pool image, arc image and thermal radiation image signals in the welding process, and constructing a weld defect data set; S2. The welding current one-dimensional time series data is converted into a welding current two-dimensional image using the Gram angle field method. The welding current two-dimensional image, molten pool image, arc image and thermal radiation image signal are first subjected to operations such as cropping, padding or adjusting resolution to make the sizes of the images the same, and then subjected to filtering and data enhancement processing. The welding current two-dimensional image, molten pool image, arc image and thermal radiation image signal are divided into a training set, a validation set and a test set; S3. Three R-Shufflenetv2 network models are constructed, and the R-Shufflenetv2 network model is an improved weld defect classification model. The R-Shufflenetv2 network model comprises a focus module (Foucs), stage2, stage3, stage4, a convolution layer, a standardization, an activation function, a DPSA module, an average pooling layer, a full connection layer and a Softmax output layer; S4. The welding current two-dimensional image and the molten pool image are input into the first R-Shufflenetv2 model for training and verification. The welding current two-dimensional image and the arc image are input into the first R-Shufflenetv2 model for training and verification. The welding current two-dimensional image and the thermal radiation image are input into the first R-Shufflenetv2 model for training and verification; S5. The three R-Shufflenetv2 models after training are optimized using the knowledge distillation technology; S6, the three R-Shufflenetv2 weld defect classification models are taken as three evidences of the evidence theory, the weld state probability distribution output by the three R-Shufflenetv2 weld defect classification models is taken as a corresponding basic belief degree assignment function, and a conflict evidence fusion algorithm in the improved D-S evidence theory is used to fuse the three evidences, so as to determine a final weld state classification result.

[0035] Preferably, the S1 specifically comprises the following steps: S11, an industrial camera is selected as a visual sensor, and a high-precision welding current sensor is used; S12, a high-precision welding current sensor is used to collect a welding current signal, a continuous current time sequence signal is obtained, and a high-speed camera is used to collect a molten pool image, an arc image and a thermal radiation image respectively; Preferably, the S2 specifically comprises the following steps: S21, the welding current is standardized, and the current time sequence signal is normalized to the interval [-1, 1]:

[0036] S22, polar coordinate mapping is performed, and the standardized value is converted into a polar coordinate angle:

[0037] In the formula, t i represents a time index, N is a sequence length.

[0038] S23, GASF image conversion is performed, and the in-phase phase cooperative change characteristics of the sequence are captured through the angle and the cosine value, and the formula is as follows:

[0039] Preferably, the R-Shufflenetv2 network model in the S3 takes a Shufflenetv2 network model as a baseline model; the Shufflenetv2 network model is an original classification model, and the Shufflenetv2 network model comprises a convolution layer 1, a maximum pooling layer, a stage2, a stage3, a stage4, a convolution layer 2, a standardization, an activation function, an average pooling layer, a full connection layer and an output layer connected in sequence.

[0040] Wherein, the R-Shufflenetv2 network model is relative to the Shufflenetv2 network model, the initial convolution layer of the Shufflenetv2 network model is exchanged into a focus module (Foucs), the Foucs block can play a role in rapidly increasing the number of network channels without losing image information, and an ECA attention mechanism module is added after stage2; in order to realize multi-scale feature fusion while ensuring that the inference speed does not slow down, in the R-Shufflenetv2 network model, the feature extraction blocks in stage3 and stage4 of the Shufflenetv2 network model are changed into a re-parameterized reshaping convolution (RCR) module, and an ECA attention mechanism module is added after stage3 and stage4; in order to retain key feature information while reducing the parameter amount, a DPSA module is added after the convolution layer 2; a phased fusion strategy is adopted, and the improved R-Shufflenetv2 network model is used for independent feature extraction of two types of input images, and then fusion feature extraction is performed.

[0041] The DPSA module first reduces the dimension of the output features of stage4 through a 3x3 maximum pooling layer (Maxpool) with a step of 2, and then passes through a channel mean (CMean) module. The module takes the average of the feature values of every two adjacent channels, specifically, the CMean module calculates the average value of the input features of the i-th and i+1-th channels, and takes it as the input feature of the i / 2-th channel. This design effectively compresses the input features from three dimensions of width, height and channel while retaining key feature information, significantly reducing the parameter amount of the subsequent module.

[0042] Further, the improved R-Shufflenetv2 network model adopts a phased fusion step, which includes: S31, inputting two images into respective focus modules (Foucs); S32, inputting the results of S31 into stage2 to preliminarily extract features of the two images to stage2; S33, splicing the feature maps of the two images in the channel dimension; S34, inputting the fused feature map into the subsequent stage3 and stage4; S35, passing the fused feature map extracted by stage4 through the convolution layer 2, the DPSA module, the standardization, the activation function, the average pooling layer, the fully connected layer and the Softmax output layer for classification.

[0043] Preferably, the step S4 specifically includes the following steps: Step S41: set training parameters, use Adam algorithm for optimization training, set parameters including training required hyperparameter setting, batch training data amount, initial learning rate, weight decay coefficient, training momentum; Step S42: put the weld defect training data set and the verification data set divided in step S13 into three improved R-Shufflenetv2 weld defect classification models for training, the input of the first improved R-Shufflenetv2 weld defect classification model is the welding current two-dimensional graph and the arc image, the input of the second improved R-Shufflenetv2 weld defect classification model is the welding current two-dimensional graph and the molten pool image, and the input of the third improved R-Shufflenetv2 weld defect classification model is the welding current two-dimensional graph and the thermal radiation image; Step S43: train the model according to the set training parameters, adjust the learning rate and the number of iterations of the model training by observing the change trend of the loss function, until the change trend of the loss function tends to be stable, and obtain the trained weld defect classification model.

[0044] Preferably, the step S5 specifically comprises the following steps: S51, taking the Resnet50 model as a teacher model, and taking the three R-Shufflenetv2 network models trained as student models; S52, training the three teacher models by using the weld defect data set to obtain the trained teacher models; S53, selecting two scene images from the weld defect data set and inputting them into the three trained teacher models and the three trained student models respectively to obtain the outputs of the three teacher models and the student models, and then calculating the loss of the corresponding teacher model and student model by using the outputs of the three teacher models and student models, thereby obtaining the loss function of the three teacher models L 蒸馏 , and the loss function of the three student models L 原学生模型 ; Wherein, the calculation formula of the loss function of the teacher model is as follows:

[0045] In the formula, KL divergence represents the KL divergence, represents the i-th smoothed value of the teacher neural network, represents the i-th smoothed value of the student neural network. ​S54, distilling the trained student model by using knowledge distillation, and calculating a distillation balance factor, the distillation balance factor being a proportion of influence of knowledge distillation in the training process of the student model, and then constructing a loss function of the student model after distillation L 学生模型 The calculation formula is as follows:

[0046] The formula of the knowledge distillation is as follows:

[0047] In the formula, z i represents the value of the first Logits, and Logits represents the output of the last layer of the network of the teacher model, i represents the distillation temperature, T q i represents a probability value, and exp represents a natural exponential function; S55, repeating S53 to S54 until the loss function of the student model after distillation converges, and updating the weight parameters of the trained student model, so as to obtain an optimized R-Shufflenetv2 model; Preferably, the step S6 specifically comprises the following steps: S61, defining an identification framework Θ, the identification framework Θ containing 6 complete mutually incompatible hypothesis commands, i.e., normal, not fused, not penetrated, porosity, weld collapse, undercut, and the power set 2 Θ ={A1,A2,A3,A4,A5,A6}; S62, taking the three R-Shufflenetv2 weld defect classification models optimized in S5 as three evidences of the evidence theory, and the three evidences are defined as E 1, E 2, E 3, then E 1, E 2, E The basic belief degree distribution functions of the three R-Shufflenetv2 weld defect classification models are respectively m 1, m 2, m 3, and the probability distribution of the weld state (normal, not fused, not penetrated, porosity, weld collapse, and undercut) output by the three R-Shufflenetv2 weld defect classification models is taken as the basic belief degree distribution function m 1, m 2, m The basic probability distribution value of 3;​ S63, using improved D-S evidence theory conflict evidence fusion algorithm, first to pre-process the evidence, and then apply D-S evidence synthesis rule for multi-source information fusion; Further, the improved D-S evidence theory conflict evidence fusion algorithm in S63 is an improvement on Murphy method, a new similarity function is used to represent the similarity of evidence, and the support degree of evidence is obtained through normalization processing, D-S evidence synthesis rule is applied for multi-source information fusion, the specific steps are as follows: (7) calculate the average value of each evidence , the formula is as follows:

[0048] (8) solve the evidence E i (A k ) and the similarity function of the average value of evidence

[0049] (9) solve the support degree coefficient , the formula is:

[0050] In the formula, The sum of and is 1, which can be used as the weight of m i ; (10) take the support degree coefficient of evidence As the correction weight of the corresponding focus element, correct it as follows:

[0051] (11) normalize , get the final synthesis result , the formula is:

[0052] (12) replace the original With , using Dempster combination rule, first fuse In turn, with this result again and Fusion, with this result and Fusion three times, Dempster combination rule is as follows:

[0053] ​Another aspect of the present application provides a special vehicle welding state intelligent identification system based on multi-modal fusion and deep learning, which adopts the above-mentioned special vehicle welding state intelligent identification method based on multi-modal fusion and deep learning to realize feature vehicle welding state identification, and the system comprises a data acquisition module, a data preprocessing module, a feature extraction and prediction module, and an improved evidence theory decision layer fusion module, wherein: The data acquisition module acquires welding current signals through a current sensor, and acquires thermal radiation images, molten pool images and arc images by means of an infrared thermal imager and a high-speed camera device respectively, so as to capture the welding part temperature distribution, molten pool dynamics and arc morphology information in the welding process. The data preprocessing module converts the one-dimensional time sequence data of the welding current into a two-dimensional image by using the Gram angle field method, filters and data enhances the molten pool image, the arc image and the thermal radiation image signal, and divides the training set, the verification set and the test set. The feature extraction and prediction module constructs three R-Shufflenetv2 network models, and sequentially performs training, verification and testing, and then adopts knowledge distillation technology to optimize the training. The improved evidence theory decision layer fusion module defines an identification framework Θ, which contains six complete mutually incompatible hypotheses, i.e. normal, incomplete fusion, incomplete penetration, porosity, weld collapse and undercutting, and its power set 2Θ={A1, A2, A3, A4, A5, A6}}, and takes the three R-Shufflenetv2 weld defect classification models optimized by knowledge distillation as three evidences of the evidence theory, i.e. E1, E2, E3, and the basic belief degree distribution functions of E1, E2, E3 are m1, m2, m3 respectively, and the probability distribution of the welding state (normal, incomplete fusion, incomplete penetration, porosity, weld collapse and undercutting) output by the three R-Shufflenetv2 weld defect classification models is taken as the basic probability distribution value of the basic belief degree distribution functions m1, m2, m3, and then the improved conflict evidence fusion algorithm in the D-S evidence theory is adopted to pre-process the evidence, and then the D-S evidence synthesis rule is applied to multi-source information fusion.

Claims

1. A special vehicle welding state intelligent identification method based on multi-modal fusion and deep learning, characterized in that, Comprise the following steps: S1, in the welding test process, by systematically changing the welding current, welding speed and wire feeding speed to produce weld defects, collect the welding current, molten pool image, arc image, thermal radiation image signal in the welding process, build the weld defect data set; S2, convert the welding current one-dimensional time series data into welding current two-dimensional image using the Gram angle field method, first cut, fill or adjust the resolution of the welding current two-dimensional image, molten pool image, arc image, thermal radiation image signal, so that the size of the image is the same, then filter, data enhancement processing, divide the welding current two-dimensional image, molten pool image, arc image, thermal radiation image signal into training set, validation set and test set; S3, build three R-Shufflenetv2 network models, the R-Shufflenetv2 network model is an improved weld defect classification model; The R-Shufflenetv2 network model comprises a focus module (Foucs), stage2, stage3, stage4, convolution layer, standardization, activation function, DPSA module, average pooling layer, full connection layer and Softmax output layer; S4, input the welding current two-dimensional image and the molten pool image into the first R-Shufflenetv2 model for training and verification, input the welding current two-dimensional image and the arc image into the first R-Shufflenetv2 model for training and verification, input the welding current two-dimensional image and the thermal radiation image into the first R-Shufflenetv2 model for training and verification; S5, use knowledge distillation technology to optimize the three R-Shufflenetv2 models trained; S6, the above three R-Shufflenetv2 weld defect classification models are used as three evidences of evidence theory, the welding state probability distribution output by the three R-Shufflenetv2 weld defect classification models is used as the corresponding basic belief degree allocation function, the improved conflict evidence fusion algorithm in the D-S evidence theory is used to fuse the three evidences, and the final weld state classification result is determined.

2. The method according to claim 1, characterized in that: The R-Shufflenetv2 network model is improved from the Shufflenetv2 network model, the initial convolution layer of the Shufflenetv2 network model is replaced by a focus module (Foucs), the Foucs block can quickly increase the number of network channels without losing image information, and an ECA attention mechanism module is added after stage 2; in order to realize multi-scale feature fusion while ensuring that the inference speed does not slow down, in the R-Shufflenetv2 network model, the feature extraction blocks in stage 3 and stage 4 of the Shufflenetv2 network model are changed into re-parameterized reshaped convolution (RCR) modules, and ECA attention mechanism modules are added after stage 3 and stage 4; in order to retain key feature information while reducing the number of parameters, a DPSA module is added after the convolution layer 2; a phased fusion strategy is adopted, and the improved R-Shufflenetv2 network model is used to perform independent feature extraction on two types of input images first, and then perform fusion feature extraction.

3. The method according to claim 2, characterized in that: The improved R-Shufflenetv2 network model adopts a phased fusion step, which comprises the following steps: Step 1, inputting two images into respective focus modules (Foucs); Step 2, inputting the results of S31 into stage 2 to preliminarily extract features of the two images to stage 2; Step 3, splicing the feature maps of the two images in the channel dimension; Step 4, inputting the fused feature maps into subsequent stage 3 and stage 4; Step 5, classifying the fused feature maps extracted by stage 4 through a convolution layer 2, a DPSA module, a standardization, an activation function, an average pooling layer, a fully connected layer and a Softmax output layer.

4. The method according to claim 1, characterized in that: Preferably, the improved conflict evidence fusion algorithm in the D-S evidence theory is an improvement on the Murphy method, a new similarity function is used to represent the similarity of evidence, the support degree of evidence is obtained through normalization processing, multi-source information fusion is performed by applying the D-S evidence synthesis rule, and the specific steps are as follows: (1) Calculate the average of each piece of evidence The formula is as follows: (2) Solving for the evidence E i (A k ) and the average of the evidence Similarity function: (3) Solving the support coefficient Sus i (A k ), the formula is: wherein Sus i (A k ) is 1, can be used as a weight corresponding to m i . (4) The support coefficient Sus of the evidence is calculated as follows: i (A k ) The modified weight value of the corresponding focus element is modified as follows: (5) to normalization, the final synthesis result is obtained The formula is: (6) the original m i (A k ) is replaced by Using the Dempster combination rule, first, the fusion in turn, with this result again with fusion, and with this result and fusion three times, the Dempster combination rule is as follows: m (Φ) = 0 5. A special vehicle welding state intelligent identification system based on multi-modal fusion and deep learning, characterized in that, The system comprises a data acquisition module, a data preprocessing module, a feature extraction and prediction module, and a decision layer fusion module based on an improved evidence theory; The data acquisition module acquires welding current signals through a current sensor, and acquires thermal radiation images, molten pool images and arc images through infrared thermal imagers and high-speed cameras, respectively, to capture the temperature distribution of the welding part, the molten pool dynamics and the arc shape information during the welding process; The data preprocessing module converts one-dimensional time sequence data of the welding current into a two-dimensional image by using the Gram angle field method, filters and enhances the data of the molten pool image, the arc image and the thermal radiation image signal, and divides the training set, the validation set and the test set; The feature extraction and prediction module constructs three R-Shufflenetv2 network models, and sequentially performs training, verification and testing, and then optimizes the training by using the knowledge distillation technology; The improved evidence-based decision layer fusion module defines a recognition framework Θ, which contains six complete mutually incompatible hypotheses, i.e., normal, unfused, unfused, porosity, weld collapse, and undercut, and its power set is 2Θ={A1,A2,A3,A4,A5,A6}}. Three R-Shufflenetv2 weld defect classification models optimized by knowledge distillation are used as three pieces of evidence of the evidence theory, and the three pieces of evidence are defined as E1, E2, and E3. The basic belief assignment functions of E1, E2, and E3 are m1, m2, and m3, respectively. The probability distribution of the weld state (normal, unfused, unfused, porosity, weld collapse, and undercut) output by the three R-Shufflenetv2 weld defect classification models is used as the basic probability assignment value of the basic belief assignment functions m1, m2, and m3. Then, the improved conflict evidence fusion algorithm in the D-S evidence theory is used to preprocess the evidence, and the D-S evidence synthesis rule is applied for multi-source information fusion.

Images