A real-time prediction method for aviation tank wall panel welding quality based on spatiotemporal information fusion

Abstract

The present invention relates to the field of intelligent manufacturing automated detection technology, and solves the technical problems that traditional methods rely on a single information source, are prone to misjudgment, missed detection, and lack of real-time performance. In particular, it relates to a real-time prediction method for the welding quality of aviation tank wall panels based on spatiotemporal information fusion, including: collecting real-time temperature data, strain data, and weld texture image data during the welding process of aviation tank wall panels, and comprehensively generating spatiotemporal variation data during the welding process of aviation tank wall panels; constructing a welding quality prediction model; using joint loss to train and update the weight of the welding quality prediction model; inputting the spatiotemporal variation data into the welding quality prediction model to obtain the welding quality rating of the aviation tank wall panel weld. The present invention can realize real-time and accurate prediction of the welding quality of the weld of the aviation tank wall panel, improve the accuracy and efficiency of detection, and at the same time ensure the demand for high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing automated detection technology, and in particular to a real-time prediction method for aviation tank wall panel welding quality based on spatiotemporal information fusion. Background Art

[0002] In the field of industrial monitoring, traditional welding quality inspection methods face many challenges and limitations. Traditional manual inspection methods rely on the experience and subjective judgment of inspectors, and are easily affected by factors such as fatigue and emotions, resulting in instability and inconsistency in inspection results. They are also inefficient and difficult to meet the requirements of modern aerospace manufacturing for high efficiency and high precision. At the same time, they cannot achieve real-time monitoring and control of the welding process.

[0003] Traditional automated detection methods usually only rely on a single information source, such as ultrasonic detection, X-ray detection, etc., and lack comprehensive analysis and fusion of multi-source spatiotemporal information in the welding process. It is difficult to fully and accurately reflect the actual situation of welding quality, and they have poor adaptability to complex structures, are prone to misjudgment and missed detection, and lack real-time performance. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a real-time prediction method for the welding quality of aviation tank wall panels based on spatiotemporal information fusion. By using machine learning and deep learning algorithms to analyze and model a large amount of data in the welding process, the complex relationship between welding quality and various factors can be discovered, the intelligent prediction of welding quality can be achieved, the accuracy and efficiency of detection can be improved, and real-time diagnosis of welding quality can be achieved.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a real-time prediction method for aviation tank wall panel welding quality based on spatiotemporal information fusion, the method comprising the following steps:

[0006] S1. Collect real-time temperature data, strain data and weld texture image data during the welding process of the aviation tank wall panel, and comprehensively generate the spatiotemporal variation data during the welding process of the aviation tank wall panel, including two-dimensional weld texture image sequence data , temperature series data And strain series data ;

[0007] S2. Constructing a welding quality prediction model for real-time prediction of welding quality ratings of aviation tank wall panels, including a spatiotemporal feature embedding module, a spatiotemporal feature alignment module, a spatiotemporal feature fusion module, and a weld quality prediction module;

[0008] S3. Using joint loss The welding quality prediction model is trained and weights are updated to obtain the final welding quality prediction model;

[0009] S4. Input the spatiotemporal variation data into the welding quality prediction model to obtain the welding quality rating of the aviation tank wall panel weld.

[0010] Furthermore, in step S1, the specific process includes the following steps:

[0011] S11. Use industrial cameras to monitor and collect the changes in weld texture images during the welding process of aviation tank wall panels in real time to obtain two-dimensional weld texture image sequence data. , , indicating the Frame weld texture image data;

[0012] S12. During the welding process of the aviation tank wall panel, the infrared laser temperature sensor and the displacement strain sensor are used to jointly monitor the contact area between the friction stir welding head and the aviation tank wall panel material in real time, and the real-time temperature data and strain data are obtained, which correspond to the two-dimensional weld texture image sequence data. Sampling to obtain temperature series data and strain series data , , indicating the Temperature and strain data at each moment;

[0013] S13, 2D weld texture image sequence data of integrated aviation tank wall panel welding , real-time temperature data and strain data during the welding process, and obtain the spatiotemporal variation data of the aviation tank wall panel welding process containing rich spatiotemporal information.

[0014] Furthermore, the combined loss Includes multi-classification cross entropy loss for evaluating and optimizing the prediction accuracy of welding quality prediction models , and the overall alignment loss used to evaluate and optimize the overall alignment effect between embedded features , joint loss The expression is:

[0015] ;

[0016] in, is a hyperparameter that balances the two losses;

[0017] Multi-classification cross entropy loss The expression is:

[0018] ;

[0019] in, is the one-hot encoding vector of the classification rating label of the actual welding quality, with the corresponding rating category taking the value of 1 and the non-corresponding rating category taking the value of 0; is the probability vector measured by the die welding quality prediction model;

[0020] Overall alignment loss The expression is:

[0021] ;

[0022] in, It represents the maximum square mean difference between the two. Embedding features for local aligned temporal temperature; embedding features for local aligned temporal strain; Embed a feature for the weld texture.

[0023] Furthermore, in step S4, the specific process includes the following steps:

[0024] S41, input the spatiotemporal variation data into the spatiotemporal feature embedding module, extract the real-time feature information in the spatiotemporal variation data through the corresponding feature encoder, and map the real-time feature information to the high-dimensional space through the multi-layer perceptron to generate the weld texture embedding feature , time series temperature embedding feature and temporal strain embedding features ;

[0025] S42, embed the time series temperature in the welding process into the feature through the spatiotemporal feature alignment module and temporal strain embedding features Same as weld texture embedding feature Perform local alignment to generate embedded features including time series temperature and temporal strain embedding features The spatiotemporal information of the embedded features is aligned;

[0026] S43, using the spatiotemporal feature fusion module to align the spatiotemporal information embedding features and weld texture embedding features Fully interact with spatiotemporal feature information to generate spatiotemporal information fusion features ;

[0027] S44. Fusion of spatiotemporal information features Input into the weld quality prediction module to generate the final weld quality prediction result.

[0028] Furthermore, in step S41, the specific process includes the following steps:

[0029] S411. Input real-time 2D weld texture image sequence data during the welding process of aviation tank wall panels , the image encoder is used to extract features to obtain the weld texture feature map , and the weld texture feature map is transformed into Mapping to high-dimensional space, outputting weld texture embedding features containing rich weld texture spatial information ;

[0030] S412. Input temperature series data of aviation tank wall panel welding and strain series data And the temperature series data is analyzed by multi-layer perceptron and strain series data Perform preliminary feature extraction to obtain real-time rough temperature feature sequence and rough strain feature sequence;

[0031] S413, inputting the temperature coarse feature sequence and the strain coarse feature sequence into the coding structure formed by stacking six time sequence coding modules for feature coding, and obtaining the real-time temperature feature sequence of the welding process. And strain characteristics ;

[0032] S414, the temperature characteristic And strain characteristics Input into the multilayer perceptron network with two hidden layers, and transform the temperature feature and strain characteristics Mapping the time series temperature embedding features into high-dimensional feature sequence And the time-series strain embedding feature .

[0033] Furthermore, in step S411, the specific process includes the following steps:

[0034] S4111, input 2D weld texture image sequence data , a shallow image feature extraction is performed through a common convolution layer to obtain the shallow feature map of the weld texture ;

[0035] S4112, DSCP module is used to input the shallow feature map of weld texture Adaptively extract deep image features to obtain deep feature maps of weld texture images ;

[0036] S4113, through the residual connection, the shallow feature map of the weld texture is obtained and deep feature map of weld texture image The features are spliced ​​in the channel dimension, and then a 1×1 convolution layer is used to reduce the dimension of the spliced ​​features in the channel dimension to further integrate the shallow and deep image features, thereby obtaining the weld texture feature map. ;

[0037] S4114, weld texture feature map Input into the multi-layer perceptron network with two hidden layers, and transform the weld texture feature map Weld texture embedding features mapped into high-dimensional feature sequences .

[0038] Furthermore, the DSCP module extracts the deep feature map of the weld texture image It is divided into two stages. The first stage contains two branch structures. In the first branch of the first stage, the shallow feature map of the weld texture is input , feature extraction is performed through an architecture consisting of a DCBL module, a residual module composed of three residual components connected in sequence, and a normal convolutional layer sequence; in the second branch, the shallow feature map of the weld texture is input Extract features directly through a common convolutional layer, and then concatenate the feature maps output by the two branches;

[0039] In the second stage, the obtained feature map is batch normalized, and then passes through an activation function and a DCBL module to finally output a deep feature map of the weld texture image containing rich global and local information. .

[0040] Furthermore, the process of feature extraction of the residual component is:

[0041] Input feature map After a feature extraction branch consisting of two DCBL modules connected in sequence and a direct edge branch, the feature maps output by the two branches are spliced ​​and then output; the calculation formula is as follows:

[0042] ;

[0043] in, represents the residual component, Represents a splicing operation;

[0044] The DCBL module consists of a deformable convolution layer, a batch normalization operation and an activation function; the input weld texture shallow feature map After entering the DCBL module, the sampling position is adaptively adjusted through the deformable convolution layer to capture the texture features of welds of different shapes and scales; then the batch normalization operation is used to reduce the internal covariate shift; finally, the activation function is used to enhance the fitting performance. The calculation formula is as follows:

[0045] ;

[0046] in, is an activation function with smooth, non-monotonic and self-regularizing properties; is the batch normalization function, It is a deformable convolution operation.

[0047] Furthermore, in step S43, the specific process includes the following steps:

[0048] S431, for local alignment of timing temperature embedding features and temporal strain embedding features and weld texture embedding features , interlaced combination is performed at each time step to construct a unified spatiotemporal information feature sequence ;

[0049] S432, unify the spatiotemporal information feature sequence The input is processed into the spatiotemporal feature fusion network constructed by stacking three Mamba modules to generate spatiotemporal information fusion features. .

[0050] Furthermore, the spatiotemporal feature fusion network generates spatiotemporal information fusion features The process is:

[0051] Input unified spatiotemporal information feature sequence , first undergoes layer normalization, and then enters the first branch and the second branch;

[0052] In the first branch, the feature sequence is first mapped to a higher dimensional space through a multi-layer perceptron to generate a high-dimensional feature sequence, and then the local features of the high-dimensional feature sequence are extracted using a common convolution module. After the SiLU activation function, the local features are processed by a selective state space model and the long-term dependencies in the feature sequence are captured;

[0053] In the second branch, the layer-normalized feature sequence is directly processed by a multi-layer perceptron and SiLU activation function;

[0054] Finally, the output results of the first branch and the second branch are multiplied by the corresponding position elements to achieve feature fusion, and then mapped back to the original dimension through a multi-layer perceptron and combined with the original unified spatiotemporal information feature sequence. Add them together to complete further fusion. The expression is as follows:

[0055] ;

[0056] in, represents a multi-layer perceptron; represents the selective state space model; Represents a normal convolution operation; Representation layer normalization processing; represents the SiLU activation function; , , They are unified spatiotemporal information feature sequences The fusion results after one, two and three layers of Mamba modules.

[0057] By means of the above technical solution, the present invention provides a real-time prediction method for aviation tank wall panel welding quality based on spatiotemporal information fusion, which has at least the following beneficial effects:

[0058] 1. The present invention can realize real-time and accurate prediction of the welding quality of the weld seam of the aviation tank wall panel, improve the accuracy and efficiency of detection, and ensure the requirement of high accuracy.

[0059] 2. The present invention uses machine learning and deep learning algorithms to analyze and model a large amount of data in the welding process, which can discover the complex relationship between welding quality and various factors, realize intelligent prediction of welding quality, improve the accuracy and efficiency of detection, and realize real-time diagnosis of welding quality.

[0060] 3. The present invention improves the input feature extraction and fusion in the traditional quality prediction model, uses a specific encoder and a spatiotemporal feature alignment and fusion method to extract and fuse the features of multiple input data, so that the neural network model can comprehensively utilize the rich spatiotemporal information contained in the multiple input data to achieve real-time and accurate prediction of the welding quality of the aviation tank wall panel. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0062] Figure 1 It is a flow chart of the real-time prediction method of the welding quality of the aviation tank wall panel in the present invention;

[0063] Figure 2 is a network structure diagram of the welding quality prediction model in the present invention;

[0064] Figure 3 is a network structure diagram of the image encoder in the present invention;

[0065] Figure 4 It is a network structure diagram of the DSCP module in the present invention;

[0066] Figure 5 It is a network structure diagram of the temperature encoder and the strain encoder in the present invention;

[0067] Figure 6 It is a network structure diagram of the pyramid attention module in the present invention;

[0068] Figure 7 It is a network structure diagram of the spatiotemporal feature fusion module in the present invention;

[0069] Figure 8 This is a network structure diagram of the weld quality prediction module in the present invention. DETAILED DESCRIPTION

[0070] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0071] With the rise of deep learning and artificial intelligence technology, deep learning-based methods no longer require artificial features, but use technologies such as convolutional neural networks to obtain advanced features, making it easier to capture complex patterns and relationships from input data. The rise of multi-source information fusion technology enables more comprehensive and accurate acquisition of relevant information on welding quality by fusing multiple information sources in the welding process (such as sensor data, image information, process parameters, etc.), improving the accuracy and reliability of detection. In particular, spatiotemporal information fusion technology can effectively capture the dynamic changes and spatiotemporal correlations in the welding process, providing richer information support for real-time prediction of welding quality.

[0072] For the difficulties in real-time prediction of welding quality of aviation tank wall panels, please refer to Figure 1-Figure 8 This embodiment proposes a real-time prediction method for the welding quality of aviation tank wall panels based on spatiotemporal information fusion, which can realize intelligent real-time prediction of the welding quality of aviation tank wall panels, improve the accuracy and efficiency of detection, and ensure the requirement of high accuracy. Figure 1 As shown, the method comprises the following steps:

[0073] S1. Collect real-time temperature data, strain data and weld texture image data during the welding process of the aviation tank wall panel, and comprehensively generate the spatiotemporal variation data during the welding process of the aviation tank wall panel, including two-dimensional weld texture image sequence data , temperature series data And strain series data The specific process includes the following steps:

[0074] S11. Use industrial cameras to monitor and collect the changes in weld texture images during the welding process of aviation tank wall panels in real time to obtain two-dimensional weld texture image sequence data. , , indicating the The weld texture image data of the frame can be set to 0.2s in a very short time in this embodiment.

[0075] S12. During the welding process of the aviation tank wall panel, the infrared laser temperature sensor and displacement strain sensor are used to monitor the contact area between the friction stir welding head and the aviation tank wall panel material in real time, and obtain real-time temperature data and strain data in a very short time, and correspond to the two-dimensional weld texture image sequence data. Sampling to obtain temperature series data and strain series data , , indicating the Temperature and strain data at each moment;

[0076] S13, 2D weld texture image sequence data of integrated aviation tank wall panel welding , real-time temperature data and strain data during the welding process, and obtain the spatiotemporal variation data of the aviation tank wall panel welding process containing rich spatiotemporal information.

[0077] S2. Construct a welding quality prediction model for real-time prediction of the welding quality rating of the aviation tank wall weld, such as Figure 2 As shown, the welding quality prediction model includes a spatiotemporal feature embedding module, a spatiotemporal feature alignment module, a spatiotemporal feature fusion module and a weld quality prediction module.

[0078] S3. Using joint loss The welding quality prediction model is trained and weights are updated to obtain the final welding quality prediction model. In this embodiment, multi-classification cross entropy loss is used. To supervise and optimize the aviation tank wall panel welding quality prediction training process, and adopt the overall alignment loss To optimize the overall alignment effect between the local aligned temporal temperature embedding features and the temporal strain embedding features and the weld texture embedding features.

[0079] Specifically, joint losses Includes multi-classification cross entropy loss for evaluating and optimizing the prediction accuracy of welding quality prediction models , and the overall alignment loss used to evaluate and optimize the overall alignment effect between embedded features , joint loss The expression is:

[0080] ;

[0081] in, is a hyperparameter that balances the two losses.

[0082] In this example, the multi-class cross entropy loss The expression is:

[0083] ;

[0084] in, is the one-hot encoding vector of the classification rating label of the actual welding quality, where the corresponding rating category takes the value of 1 and the non-corresponding rating category takes the value of 0; is the probability vector measured by the die welding quality prediction model.

[0085] Overall alignment loss It is used to evaluate and optimize the overall alignment effect between embedded features. The formula is expressed as:

[0086] ;

[0087] in, It represents the maximum square mean difference between the two. Embedding features for local aligned temporal temperature; embedding features for local aligned temporal strain; Embed a feature for the weld texture.

[0088] S4, input the spatiotemporal variation data into the welding quality prediction model to obtain the welding quality rating of the aviation tank wall panel weld. The specific process includes the following steps:

[0089] S41, input the spatiotemporal variation data into the spatiotemporal feature embedding module, extract the real-time feature information in the spatiotemporal variation data through the corresponding feature encoder, and map the real-time feature information to the high-dimensional space through the multi-layer perceptron to generate the weld texture embedding feature , time series temperature embedding feature and temporal strain embedding features The specific process includes the following steps:

[0090] S411. Input real-time 2D weld texture image sequence data during the welding process of aviation tank wall panels , the image encoder is used to extract features to obtain the weld texture feature map , and the weld texture feature map is transformed into Mapping to high-dimensional space, outputting weld texture embedding features containing rich weld texture spatial information The specific process includes the following steps:

[0091] S4111, input 2D weld texture image sequence data , a shallow image feature extraction is performed through a common convolution layer to obtain the shallow feature map of the weld texture ;

[0092] S4112, DSCP module is used to input the shallow feature map of weld texture Adaptively extract deep image features to obtain deep feature maps of weld texture images .

[0093] like Figure 4 The following is the network structure diagram of the DSCP module (deformable convolution residual module). The DSCP module is divided into two stages. The first stage contains two branch structures. In the first branch of the first stage, the shallow feature map of the weld texture is input. , feature extraction is performed through an architecture consisting of a DCBL module, a residual module composed of three residual components connected in sequence, and a normal convolutional layer sequence; in the second branch, the shallow feature map of the weld texture is input Extract features directly through a common convolutional layer, and then concatenate the feature maps output by the two branches;

[0094] In the second stage, the obtained feature map is batch normalized, and then passes through an activation function and a DCBL module to finally output a deep feature map of the weld texture image containing rich global and local information. .

[0095] The architecture of the DCBL module (deformable convolution module) is as follows Figure 4 As shown, it consists of a deformable convolution layer, a batch normalization operation and an activation function; the input weld texture shallow feature map After entering the DCBL module, the sampling position is adaptively adjusted through the deformable convolution layer to capture the texture features of welds of different shapes and scales; the batch normalization operation is then used to reduce the internal covariate offset, thereby avoiding the gradient explosion and vanishing problems, so as to improve the stability and training speed of the neural network model; finally, the activation function (using the Mish activation function) is used to enhance the fitting performance of the neural network model. The calculation formula is as follows:

[0096] ;

[0097] in, is an activation function with smooth, non-monotonic and self-regularizing properties; is the batch normalization function, For deformable convolution operation, an offset is introduced into the ordinary convolution operation to enhance the adaptability of the convolution operation.

[0098] Specifically, a normal convolution operation is performed on the input feature map, and the input feature map is processed through an offset generation network to obtain an offset feature map with 2N channels, which contains the offset information in the x and y directions of all positions of the original feature map. The offset information is then added to the result of the normal convolution operation to obtain the final deformable convolution result. The calculation formula is as follows:

[0099] ;

[0100] in, Represents the input feature map location; Represents an offset of the convolution kernel (for example, a 3×3 convolution kernel has 9 offsets); is the convolution kernel at the offset The weight of Represents the input feature map In Location The pixel value size at ; The offsets are learned by the offset generation network (a normal convolution layer containing 2N convolution kernels). These offsets can flexibly adjust the convolution kernels, allowing the neural network to adaptively process different structures and shapes of the input feature map. is the pixel value of the input feature map at the adjusted position, calculated by bilinear interpolation to cope with non-integer sampling positions;

[0101] In addition, the activation function The calculation formula is as follows:

[0102] ;

[0103] In the formula, represents the input feature vector of the activation function; Represents the natural logarithm function of x.

[0104] The residual component is: Input feature map After a feature extraction branch consisting of two DCBL modules connected in sequence and a direct edge branch, the feature maps output by the two branches are spliced ​​and then output; the residual structure can increase the gradient value of back propagation between the layers of the neural network, avoiding the gradient vanishing problem caused by the deepening of the neural network model, so that more fine-grained high-dimensional features can be extracted without worrying about network performance degradation. The calculation formula is as follows:

[0105] ;

[0106] in, represents the residual component, Represents a concatenation operation.

[0107] S4113, through the residual connection, the shallow feature map of the weld texture is obtained and deep feature map of weld texture image The features are spliced ​​in the channel dimension, and then a 1×1 convolution layer is used to reduce the dimension of the spliced ​​features in the channel dimension to further integrate the shallow and deep image features, thereby obtaining the weld texture feature map. ;

[0108] S4114, weld texture feature map The input is a multi-layer perceptron network with two hidden layers. The powerful fitting ability of the multi-layer perceptron is used to learn the rich spatial information in the weld texture feature map. Weld texture embedding features mapped into high-dimensional feature sequences .

[0109] S412. Input temperature series data of aviation tank wall panel welding and strain series data And the temperature series data is analyzed by multi-layer perceptron and strain series data Perform preliminary feature extraction to obtain real-time rough temperature feature sequence and rough strain feature sequence. and strain series data Input into the temperature encoder and strain encoder respectively, such as Figure 5 As shown in the figure, the two have the same structural design and are used to process temperature and strain data respectively.

[0110] S413, inputting the temperature coarse feature sequence and the strain coarse feature sequence into the coding structure formed by stacking six time sequence coding modules for feature coding, and obtaining the real-time temperature feature sequence of the welding process. And strain characteristics In the temporal coding module, the input temperature coarse feature sequence and strain coarse feature sequence first pass through the pyramid attention module, and use the attention mechanism to weight the temperature coarse feature sequence and strain coarse feature sequence to obtain the attention weighted features, and then undergo layer normalization operation. The calculation results are directly added to the original input temperature coarse feature sequence and strain coarse feature sequence through residual connection to achieve feature fusion; the fused features are then input into the feedforward neural network (multilayer perceptron) for further processing, and the same layer normalization and residual connection addition operations are performed to enhance the model fitting ability.

[0111] The feature sequence processed by one layer of timing coding module is input into the next timing coding module for further coding. After a total of six layers of timing coding modules, the final output of the temperature (or strain) feature encoder is obtained, that is, the real-time temperature feature of the welding process. And strain characteristics .

[0112] In this embodiment, the network structure diagram of the pyramid attention module is as follows: Figure 6 As shown in the figure, the input temperature coarse feature sequence and strain coarse feature sequence data are sliced ​​and divided into four short sequence data of the same length, and then self-attention calculation is performed on each short sequence data to achieve local weighting within the sequence data. After one attention weighting, the obtained feature sequences are spliced ​​in pairs, and self-attention is calculated again to achieve a larger range of local weighting effects. Finally, the two are spliced ​​and self-attention is calculated again to achieve global weighting of the sequence and obtain attention weighted features. Using this pyramid structure attention mechanism, the weighted fusion of local features and global features of sequence data is achieved.

[0113] S414, the temperature characteristic And strain characteristics The input is fed into a multilayer perceptron network with two hidden layers. The powerful fitting ability of the multilayer perceptron is used to further learn the rich time series information in the temperature and strain data during the welding process. and strain characteristics Mapping the time series temperature embedding features into high-dimensional feature sequence And the time-series strain embedding feature .

[0114] Specifically, by inputting the temperature series data of the aviation tank wall panel welding and strain series data , respectively use the temperature encoder and strain encoder to extract features and obtain the real-time temperature characteristics during welding And strain characteristics And the temperature features are respectively and strain characteristics All are mapped to a high-dimensional space, and the output contains rich information about the time series temperature and strain during the welding process. And the time-series strain embedding feature .

[0115] S42, embed the time series temperature in the welding process into the feature through the spatiotemporal feature alignment module and temporal strain embedding features Same as weld texture embedding feature Perform local alignment to generate embedded features including time series temperature and temporal strain embedding features The spatiotemporal information of the embedded features is aligned.

[0116] Specifically, in order to capture the weld texture embedding features Embedding features with time series temperature and temporal strain embedding features The fine-grained (word-level) correspondence between the feature sequences is considered as a discrete probability distribution, and the time series temperature is embedded into the feature by using the optimal transport method. and temporal strain embedding features The corresponding probability distribution is converted into weld texture embedding features Corresponding probability distribution, thus realizing the weld texture embedding feature Embedding features with time series temperature and temporal strain embedding features The fine-grained alignment between them obtains the spatiotemporal information alignment embedding feature, that is, the local aligned temporal temperature embedding feature and temporal strain embedding features .

[0117] To ensure local alignment of the timing temperature embedding features and temporal strain embedding features , and the weld texture embedding feature In order to ensure the consistency of the overall distribution level between the three embedded features, all orders of the statistics in the high-dimensional reproducing kernel Hilbert space (RKHS) are compared to measure and minimize the statistical differences between different embedded features and ensure the overall distribution consistency between the embedded features.

[0118] S43, using the spatiotemporal feature fusion module to align the spatiotemporal information embedding features and weld texture embedding features Fully interact with spatiotemporal feature information to generate spatiotemporal information fusion features The specific process includes the following steps:

[0119] S431, for local alignment of timing temperature embedding features and temporal strain embedding features and weld texture embedding features , interlaced combination is performed at each time step to construct a unified spatiotemporal information feature sequence ;

[0120] S432, unify the spatiotemporal information feature sequence The input is processed into the spatiotemporal feature fusion network constructed by stacking three Mamba modules to generate spatiotemporal information fusion features. .

[0121] The network structure diagram of the Mamba module is as follows Figure 7 As shown, the unified spatiotemporal information feature sequence is input, firstly processed by layer normalization, and then enters the first branch and the second branch;

[0122] In the first branch, the feature sequence is first mapped to a higher-dimensional space through a multi-layer perceptron to generate a high-dimensional feature sequence, thereby capturing more detailed and complex features. Then, the local features of the high-dimensional feature sequence are extracted using a common convolutional module. After the SiLU activation function, the local features are processed through a selective state space model and the long-term dependencies in the feature sequence are captured.

[0123] In the second branch, the layer-normalized feature sequence is directly processed by a multi-layer perceptron and SiLU activation function;

[0124] Finally, the output results of the first branch and the second branch are multiplied by the corresponding position elements to achieve feature fusion, and then mapped back to the original dimension through a multi-layer perceptron and combined with the original unified spatiotemporal information feature sequence. Add them together to complete further fusion. The expression is as follows:

[0125] ;

[0126] in, represents a multi-layer perceptron; represents the selective state space model; Represents a normal convolution operation; Representation layer normalization processing; represents the SiLU activation function; , , They are unified spatiotemporal information feature sequences The fusion results after one, two and three layers of Mamba modules;

[0127] also, Represents the SiLU activation function, which is calculated as follows:

[0128] ;

[0129] In the formula, represents the input feature vector of the activation function; express The natural logarithm function of .

[0130] S44. Fusion of spatiotemporal information features The result is input into the weld quality prediction module to generate the final weld quality prediction result. Input into the prediction head network, such as Figure 8 As shown, the input spatiotemporal information fusion features First, the layer is normalized and then mapped by a multi-layer perceptron. The mapping result is activated by the Softmax function to output the prediction scores of the three levels (category 1, category 2, and category 3) of the weld quality rating of the aviation tank wall panel. The prediction category with the highest score is selected as the rating of the weld quality of the predicted aviation tank wall panel. The expression of the Softmax activation function is:

[0131] ;

[0132] In the formula, represents the i-th element of the input feature vector of the activation function, express The natural logarithm function of Represents the dimension of the input feature vector.

[0133] In summary, the present invention improves the input feature extraction and fusion in the traditional quality prediction model, uses a specific encoder and a spatiotemporal feature alignment and fusion method to extract and fuse the features of multiple input data, so that the neural network model can comprehensively utilize the rich spatiotemporal information contained in the multiple input data to achieve real-time and accurate prediction of the welding quality of the aviation tank wall panel.

[0134] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0135] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A real-time prediction method for aviation tank wall panel welding quality based on spatiotemporal information fusion, characterized in that: The method comprises the following steps: S1. Collect real-time temperature data, strain data and weld texture image data during the welding process of the aviation tank wall panel, and comprehensively generate the spatiotemporal variation data during the welding process of the aviation tank wall panel, including two-dimensional weld texture image sequence data , temperature series data And strain series data ; S2. Constructing a welding quality prediction model for real-time prediction of welding quality ratings of aviation tank wall panels, including a spatiotemporal feature embedding module, a spatiotemporal feature alignment module, a spatiotemporal feature fusion module, and a weld quality prediction module; S3. Using joint loss The welding quality prediction model is trained and weights are updated to obtain the final welding quality prediction model; S4. Input the spatiotemporal variation data into the welding quality prediction model to obtain the welding quality rating of the aviation tank wall panel weld. The specific process includes the following steps: S41, input the spatiotemporal variation data into the spatiotemporal feature embedding module, extract the real-time feature information in the spatiotemporal variation data through the corresponding feature encoder, and map the real-time feature information to the high-dimensional space through the multi-layer perceptron to generate the weld texture embedding feature , time series temperature embedding feature and temporal strain embedding features ; S42, embed the time series temperature in the welding process into the feature through the spatiotemporal feature alignment module and temporal strain embedding features Same as weld texture embedding feature Perform local alignment to generate embedded features including time series temperature and temporal strain embedding features The spatiotemporal information of the embedded features is aligned; S43, using the spatiotemporal feature fusion module to align the spatiotemporal information embedding features and weld texture embedding features Fully interact with spatiotemporal feature information to generate spatiotemporal information fusion features ; S44. Fusion of spatiotemporal information features Input into the weld quality prediction module to generate the final weld quality prediction result.

2. The real-time prediction method for welding quality of aviation tank wall panels according to claim 1 is characterized in that: In step S1, the specific process includes the following steps: S11. Use industrial cameras to monitor and collect the changes in weld texture images during the welding process of aviation tank wall panels in real time to obtain two-dimensional weld texture image sequence data. , , indicating the Frame weld texture image data; S12. During the welding process of the aviation tank wall panel, the infrared laser temperature sensor and the displacement strain sensor are used to jointly monitor the contact area between the friction stir welding head and the aviation tank wall panel material in real time, and the real-time temperature data and strain data are obtained, which correspond to the two-dimensional weld texture image sequence data. Sampling to obtain temperature series data and strain series data , , indicating the Temperature and strain data at each moment; S13, 2D weld texture image sequence data of integrated aviation tank wall panel welding , real-time temperature data and strain data during the welding process, and obtain the spatiotemporal variation data of the aviation tank wall panel welding process containing rich spatiotemporal information.

3. The real-time prediction method for welding quality of aviation tank wall panels according to claim 1 is characterized in that: The joint loss Includes multi-classification cross entropy loss for evaluating and optimizing the prediction accuracy of welding quality prediction models , and the overall alignment loss used to evaluate and optimize the overall alignment effect between embedded features , joint loss The expression is: ； in, is a hyperparameter that balances the two losses; Multi-classification cross entropy loss The expression is: ； in, is the one-hot encoding vector of the classification rating label of the actual welding quality, with the corresponding rating category taking the value of 1 and the non-corresponding rating category taking the value of 0; is the probability vector measured by the die welding quality prediction model; Overall alignment loss The expression is: ； in, It represents the maximum square mean difference between the two. Embedding features for local aligned temporal temperature; embedding features for local aligned temporal strain; Embed a feature for the weld texture.

4. The real-time prediction method for aviation tank wall panel welding quality according to claim 1 is characterized in that: In step S41, the specific process includes the following steps: S411. Input real-time 2D weld texture image sequence data during the welding process of aviation tank wall panels , the image encoder is used to extract features to obtain the weld texture feature map , and the weld texture feature map is transformed into Mapping to high-dimensional space, outputting weld texture embedding features containing rich weld texture spatial information ; S412. Input temperature series data of aviation tank wall panel welding and strain series data And the temperature series data is analyzed by multi-layer perceptron and strain series data Perform preliminary feature extraction to obtain real-time rough temperature feature sequence and rough strain feature sequence; S413, inputting the temperature coarse feature sequence and the strain coarse feature sequence into the coding structure formed by stacking six time sequence coding modules for feature coding, and obtaining the real-time temperature feature sequence of the welding process. And strain characteristics ; S414, the temperature characteristic And strain characteristics Input into the multilayer perceptron network with two hidden layers, and transform the temperature feature and strain characteristics Mapping the time series temperature embedding features into high-dimensional feature sequence And the time-series strain embedding feature .

5. The real-time prediction method for welding quality of aviation tank wall panels according to claim 4 is characterized in that: In step S411, the specific process includes the following steps: S4111, input 2D weld texture image sequence data , a shallow image feature extraction is performed through a common convolution layer to obtain the shallow feature map of the weld texture ; S4112, DSCP module is used to input the shallow feature map of weld texture Adaptively extract deep image features to obtain deep feature maps of weld texture images ; S4113, through the residual connection, the shallow feature map of the weld texture is obtained and deep feature map of weld texture image The features are spliced ​​in the channel dimension, and then a 1×1 convolution layer is used to reduce the dimension of the spliced ​​features in the channel dimension to further integrate the shallow and deep image features, thereby obtaining the weld texture feature map. ; S4114, weld texture feature map Input into the multi-layer perceptron network with two hidden layers, and transform the weld texture feature map Weld texture embedding features mapped into high-dimensional feature sequences .

6. The real-time prediction method for welding quality of aviation tank wall panels according to claim 5 is characterized in that: The DSCP module extracts the deep feature map of the weld texture image It is divided into two stages. The first stage contains two branch structures. In the first branch of the first stage, the shallow feature map of the weld texture is input , feature extraction is performed through an architecture consisting of a DCBL module, a residual module composed of three residual components connected in sequence, and a normal convolutional layer sequence; in the second branch, the shallow feature map of the weld texture is input Extract features directly through a common convolutional layer, and then concatenate the feature maps output by the two branches; In the second stage, the obtained feature map is batch normalized, and then passes through an activation function and a DCBL module to finally output a deep feature map of the weld texture image containing rich global and local information. .

7. The real-time prediction method for welding quality of aviation tank wall panels according to claim 6 is characterized in that: The process of feature extraction of the residual component is as follows: Input feature map After a feature extraction branch consisting of two DCBL modules connected in sequence and a direct edge branch, the feature maps output by the two branches are spliced ​​and then output; the calculation formula is as follows: ； in, represents the residual component, Represents a splicing operation; The DCBL module consists of a deformable convolution layer, a batch normalization operation and an activation function; the input weld texture shallow feature map After entering the DCBL module, the sampling position is adaptively adjusted through the deformable convolution layer to capture the texture features of welds of different shapes and scales; then the batch normalization operation is used to reduce the internal covariate shift; finally, the activation function is used to enhance the fitting performance. The calculation formula is as follows: ； in, is an activation function with smooth, non-monotonic and self-regularizing properties; is the batch normalization function, It is a deformable convolution operation.

8. The real-time prediction method for aviation tank wall panel welding quality according to claim 1 is characterized in that: In step S43, the specific process includes the following steps: S431, for local alignment of timing temperature embedding features and temporal strain embedding features and weld texture embedding features , interlaced combination is performed at each time step to construct a unified spatiotemporal information feature sequence ; S432, unify the spatiotemporal information feature sequence The input is processed into the spatiotemporal feature fusion network constructed by stacking three Mamba modules to generate spatiotemporal information fusion features. .

9. The real-time prediction method for welding quality of aviation tank wall panels according to claim 8 is characterized in that: The spatiotemporal feature fusion network generates spatiotemporal information fusion features The process is: Input unified spatiotemporal information feature sequence , first undergoes layer normalization, and then enters the first branch and the second branch; In the first branch, the feature sequence is first mapped to a higher dimensional space through a multi-layer perceptron to generate a high-dimensional feature sequence, and then the local features of the high-dimensional feature sequence are extracted using a common convolution module. After the SiLU activation function, the local features are processed by a selective state space model and the long-term dependencies in the feature sequence are captured; In the second branch, the layer-normalized feature sequence is directly processed by a multi-layer perceptron and SiLU activation function; Finally, the output results of the first branch and the second branch are multiplied by the corresponding position elements to achieve feature fusion, and then mapped back to the original dimension through a multi-layer perceptron and combined with the original unified spatiotemporal information feature sequence. Add them together to complete further fusion. The expression is as follows: ； in, represents a multi-layer perceptron; represents the selective state space model; Represents a normal convolution operation; Representation layer normalization processing; represents the SiLU activation function; , , They are unified spatiotemporal information feature sequences The fusion results after one, two and three layers of Mamba modules.

Images