Multi-sensor weld penetration state recognition method for improving attention mechanism

Abstract

The invention provides a multi-sensor weld penetration state recognition method for improving an attention mechanism, relates to the technical field of weld recognition, and solves the technical problem of inaccurate weld quality judgment caused by insufficient recognition precision of a single signal on a penetration state in the prior art. According to the technical scheme, a sound signal in the welding process and a molten pool image in the welding process are collected; effective sound signals are extracted, and then the sound signals are visualized by using a Gramer angle field method; performing de-exposure processing on the overexposure area through a decoupling polymerization exposure correction model; a channel attention and space attention mechanism is introduced into sound and image feature extraction, deformable convolution self-adaptive sampling is utilized, a double-cross attention fusion strategy is combined to fuse multi-source information, fusion penetration state classification is completed, and therefore the quality of a welding seam is judged. According to the method, the penetration state can be accurately classified through the sound signal extracted in the welding process and the molten pool image, and a certain thought is provided for weld quality detection.

Description

Technical Field

[0001] The present invention relates to the technical field of weld identification, and specifically to a multi-sensor weld penetration state identification method with an improved attention mechanism. Background Art

[0002] The detection of the welding state is a crucial stage to ensure welding quality and safety. Through detection, welding defects can be discovered in time to avoid equipment failures or accidents caused by defects, providing guarantee for efficient industrial production and safety. The penetration state is usually achieved when the heat input is moderate and the weld is reasonable; incomplete penetration is mostly caused by insufficient heat or too fast welding speed resulting in the joint not being fully penetrated; burn-through is due to excessive heat input and too slow welding speed causing the molten pool to penetrate the bottom of the weld. For welding state identification methods, there are usually image and sound signal detections. Image signal detection uses a camera to capture the welding arc light and the morphology of the molten pool, and extracts welding state information through image processing technology. It can provide complete visual information on spatial distribution characteristics and weld geometry. Through image processing technology, parameters such as the size, shape, flow state of the molten pool and the forming quality of the weld can be extracted in real time. Sound signal detection records characteristics such as arc sound and metal impact sound during the welding process through sensors, and judges the welding state through spectrum analysis or pattern recognition methods. The sound signal acquisition device is simple and has a low cost, and is not affected by strong arc light, smoke or spatter during the welding process. At the same time, the sound signal directly reflects the energy release and physical change characteristics during the welding process, facilitating real-time detection and rapid response.

[0003] However, identifying the welding state by single sound signal is easily interfered by environmental noise, affecting the accuracy of the signal. At the same time, its information dimension is relatively single, and it can only reflect the dynamic characteristics during the welding process, unable to provide key geometric information such as weld morphology. While single image recognition can intuitively present the weld morphology, it is highly sensitive to environmental factors such as light and smoke, easily causing image quality degradation, which may lead to inaccurate or incomplete recognition of abnormalities. Therefore, this application proposes a multi-sensor weld penetration state identification method with an improved attention mechanism. This method first collects the welding process sound signal through a microphone sensor and uses a CCD sensor to collect the molten pool image during the welding process; uses the VMD technology to extract the effective sound signal, and then visualizes the sound signal by the Gramian Angular Field (GASF) method; performs de-exposure processing on the overexposed area through the decoupled aggregation exposure correction model; introduces a channel attention mechanism and a spatial attention mechanism in the extraction of sound and image features, uses deformable convolution for adaptive sampling and combines the dual-cross attention fusion strategy to fuse multi-source information, and completes the penetration state classification to judge the weld quality. Summary of the Invention

[0004] Therefore, the present invention solves the problem that the traditional single signal has insufficient recognition accuracy for the penetration state, resulting in inaccurate determination of the weld quality, and proposes a multi-sensor weld penetration state recognition method with an improved attention mechanism, which can break through the technical bottlenecks faced by the traditional single detection method in complex working conditions, such as high environmental sensitivity, narrow feature coverage dimension, and incomplete abnormal recognition, and significantly improve the classification accuracy of the penetration state in the welding process and the reliability of weld quality determination.

[0005] A multi-sensor weld penetration state recognition method with an improved attention mechanism provided by the present invention mainly includes four steps:

[0006] S1. Collect the welding process sound signal through a microphone sensor, and use a CCD sensor to collect the molten pool image during the welding process;

[0007] S2. Use the VMD technology to extract the effective sound signal, and then use the Gramian angular field (GASF) method to visualize the sound signal;

[0008] S3. Perform de-exposure processing on the overexposed area of the molten pool image through a decoupled aggregation exposure correction model;

[0009] S4. By introducing a channel attention mechanism and a spatial attention mechanism in the extraction of sound and image features, using deformable convolution for adaptive sampling and combining a dual-cross attention fusion strategy to fuse multi-source information, complete the penetration state classification and thus determine the weld quality.

[0010] Preferably, in step S1, it specifically includes:

[0011] S101. Weld HSLA-115 high-strength low-alloy steel with different thicknesses but the same area by using a continuous wave CO2 laser welding device. A penetration experiment was carried out on a 200mm×100mm×6mm steel plate, a non-penetration experiment was carried out on a 200mm×100mm×10mm steel plate, and a burn-through experiment was carried out on a 200mm×100mm×14mm steel plate. And a stepping motor control console was used to control the welding platform, so that the welded part moves along the welding direction during the welding process to ensure the stability of the welding state.

[0012] S102. During the welding process, identify the arc sound signal through a microphone sensor. The identified signal is processed by a four-channel input sound and vibration analyzer and then displayed and stored in the Labshop software. At the same time, use a CCD sensor to collect the molten pool image, and add a narrow-band filter and a protective mirror in front of the CCD lens to prevent splashing during the welding process. During the collection process, set the collection time of the CCD by writing a LabVIEW program, and transfer the collected molten pool image to an image acquisition card, and store it in a PC after processing.

[0013] Preferably, in step S2, it specifically includes:

[0014] S201. The audio signal is represented as a discrete time series through sampling. By analyzing the time-domain waveform of the audio signal, the amplitude envelope A(t) is calculated:

[0015]

[0016] where y(t) is the extracted time series, is the derivative of the signal, representing the rate of change of the signal.

[0017] By calculating the amplitude envelope, the energy change trend of the audio signal is revealed. Since the low-frequency part usually contains the main energy of the signal and carries the main information of the sound, therefore, the high-frequency components in the signal are removed, and the effective signal characteristics of the low-frequency are retained to extract the effective sound waveform.

[0018] S202. The time-domain signal is decomposed by VMD technology to obtain multiple intrinsic mode functions IMF, and each intrinsic mode function represents the characteristics of the signal in different frequency band ranges:

[0019]

[0020] where, u k (t) represents the k-th mode, ω k represents the corresponding central frequency, * represents convolution, represents the partial derivative operation, δ(t) represents the unit impulse function.

[0021] The constraint condition is:

[0022]

[0023] where, f(t) represents the signal to be decomposed, u k (t) represents the k-th mode

[0024] The penalty factor a and the Lagrange multiplier λ are introduced to transform the constraint into an unconstrained variational problem, and the above objective function is solved. The augmented Lagrange expression is:

[0025]

[0026] where a is the penalty factor, represents the partial derivative operation, δ(t) represents the unit impulse function, u k (t) represents the k-th mode, ω k represents the corresponding central frequency, f(t) represents the signal to be decomposed, and λ(t) is the Lagrange multiplier.

[0027] The multiplication operator alternating algorithm is used to iteratively solve this problem, and u is continuously updated through iteration. k , ω k and λ. During the update process, the modal components and center frequencies and the Lagrangian operator expressions are as follows:

[0028]

[0029] where f(ω) is the Fourier transform of f(t), λ(ω) is the Fourier transform of λ(t), u k (ω) is the Fourier transform of u k (t), and τ is the noise tolerance.

[0030] Multiple intrinsic mode functions IMF are obtained through VMD technology. Effective pulses are extracted from the decomposed modes, non-related modal noises are removed, and the optimized signal most relevant to the welding sound state is extracted.

[0031] S203. The extracted key effective acoustic signal is a one-dimensional signal. To accurately fuse with the two-dimensional molten pool image features, the GASF method is used to convert the extracted one-dimensional sound signal into a two-dimensional image. The extracted sound signal is used as the original signal and is normalized to the interval [-1, 1] to eliminate the amplitude difference:

[0032]

[0033] where X = {x1, x2,..., x N} is the original signal, x i ′ is the value after normalization and x i ′ ∈ [-1, 1].

[0034] Then the normalized value x i ′ is converted into the polar coordinate angle φ i :

[0035] φ i = arccos(x i ′) (9)

[0036] where φ i ∈ [0, π].

[0037] An N×N matrix is constructed, and each element (i, j) represents the sum of the cosine values of the angles φ i and φ j , and finally the GASF diagram is obtained.

[0038]

[0039] where GASF(i, j) represents the element value of the matrix at (i, j), that is, the gray value of the corresponding two-dimensional image.

[0040] Preferably, in step S3, it specifically includes:

[0041] S301. There is an exposure error in the molten pool image collected by the sensor and the acquisition card. Therefore, an end-to-end exposure correction model is used to process the overexposed area. The model is a generator that includes an image encoder, continuous residual blocks, and an image decoder, and decouples contrast enhancement and detail restoration during each convolution process. In the decoupling operation, the local smoothness assumption is adopted. We select the central pixel of R in the local patch as the reference pixel, denoted as p c , and the pixel intensity at other positions can be expressed as the sum of the central pixel intensity and the deviation n i :

[0042] x(p i ) = x(p c ) + n i (11)

[0043] where x(p i ) represents the pixel intensity at any position in the local patch, x(p c ) represents the central pixel intensity (low-frequency component) of the local patch, and n i is the deviation (high-frequency component) of pixel p i relative to p c .

[0044] According to Equation (11), the convolution process can be expressed as:

[0045]

[0046] where y(p c ) represents the output of the convolution operation at the central pixel p c , w(p i ) is the weight of the convolution kernel at pixel p i , and x(p i ) represents the pixel intensity at p i .

[0047] It can be observed from Equation (12) that in the traditional convolution feature response, the low-frequency response and the high-frequency response are mixed. We separate the high-frequency convolution response from the low-frequency convolution response through central surrounding difference operation:

[0048]

[0049] where y h (p c ) represents the high-frequency output of the convolution operation at the central pixel p c , and y l (p c) represents the low-frequency output of the convolution operation at the central pixel p c of, w(p i ) is the weight of the convolution kernel at pixel p i , and n i is the bias of pixel p i relative to p c .

[0050] By performing differential operations around the center, the high-frequency and low-frequency responses can be suppressed separately. The convolution kernel with high-frequency response in the convolution operation is called the detail-aware kernel (DA), and the convolution kernel with low-frequency response is called the contrast-aware kernel (CA). DA and CA are connected in parallel to replace the traditional convolution kernel. However, the differential and addition operations in DA and CA may lead to an imbalance in the amplitudes of the high-frequency and low-frequency responses. To address this, we introduce a dynamic adjustment coefficient on each branch to adjust the amplitude of the feature response for aggregation:

[0051]

[0052] where s is the sigmoid activation function used to constrain the distribution of the adjustment coefficient from 0 to 1. Through continuous training, the adjustment coefficient is continuously updated to balance the response amplitudes of CA and DA.

[0053] S302. After using DA and CA to replace the traditional convolution kernel, the encoder in the generator is used to extract image features, which are sent to consecutive residual blocks to edit the feature representation. Finally, the modified features pass through the image decoder to produce the final output. At the same time, a discriminator is used to distinguish real samples and fake samples and represent the adversarial loss. It contains several consecutive convolutional layers to downsample the image to produce the final output. By combining the perceptual loss and the pixel loss, the image quality is finally improved to achieve the de-exposure effect.

[0054]

[0055] L pixel = ||y′ - y||1 (17)

[0056] L = L GAN (G, D) + λL pixel (G) + βL per (G) (18)

[0057] where L per represents the perceptual loss, L pixel represents the pixel loss, c i represents the weight coefficient of the i-th feature layer, y′ represents the generated image, y represents the target image, L i (y) represents the feature map of the target image at the i-th feature layer, L i(y′) represents the feature map of the generated image at the i-th feature layer, ||||1 represents the L1 norm, D represents the discriminator, G represents the generator, λ and β are coefficients indicating the impact of each loss on the final loss, and λ, β ∈ [0, 1].

[0058] Preferably, in step S4, it specifically includes:

[0059] S401. After obtaining the GASF image, the improved ResNet50 network is used to extract features from the molten pool image and the GASF image. In the traditional ResNet50 network, the last fully connected layer is deleted, and the image features are finally output. Since the shape of the molten pool in the molten pool image is irregular, deformable convolutions are used to replace some of the traditional convolutional kernels. Deformable convolutions introduce learnable offsets for each sampling point, enabling the convolutional kernel to adaptively adjust the sampling area according to the local content of the input feature map.

[0060]

[0061] Where y(p0) represents the value of the output feature map of the traditional convolutional kernel at position p0, y′(p0) represents the value of the output feature map of the deformable convolutional kernel at position p0, R is the receptive field of the convolutional kernel, p n represents the sampling position of the convolutional kernel within the receptive field R, p0 represents a certain position on the output feature map, w(p n ) represents the weight of the convolutional kernel at p n , and Δp n is the learnable offset.

[0062] S402. At the same time, in order to meet the accuracy of adaptive feature extraction, the CBAM module is added. During the GASF image feature process, emphasis is placed on the channel attention mechanism. By performing global average pooling on the input feature map, the information of each channel is compressed into a scalar, and the weights of each channel are calculated through a fully connected layer, dynamically enhancing the key channels related to the periodic features of the time series and suppressing redundant or noisy channels.

[0063] M c (F) = σ(MLP(AvgPool(F)) + MLP(MaxPool(F))) (21)

[0064] Where F represents the input feature map, AvgPool(F) represents the average pooling operation of the input feature map F, MaxPool(F) represents the max pooling operation of the input feature map F, MLP(·) represents the multi-layer perceptron, which further processes the pooled features, σ(·) is the Sigmoid function, mapping the output of the MLP to between [0, 1], and M c (F) represents the channel attention map, indicating the importance weights of each channel.

[0065] During the process of extracting the characteristics of the molten pool image, the spatial attention mechanism is emphasized. Global max pooling and average pooling are performed on the input feature map along the channel dimension, and the two feature maps are concatenated to generate a spatial weight map. The spatial weight map is multiplied by the original feature map position by position to highlight the characteristics of the key areas related to the dynamics of the molten pool image.

[0066] M s (F) = σ(Conv([AvgPool(F); MaxPool(F)])) (22)

[0067] Where F represents the input feature map, AvgPool(F) represents the average pooling operation of the input feature map F, MaxPool(F) represents the max pooling operation of the input feature map F, [AvgPool(F); MaxPool(F)] represents the concatenation of the average pooling and max pooling results, Conv(·) represents a three-dimensional convolutional layer, which performs a convolutional operation on the concatenated features, and σ(·) is the Sigmoid function, mapping the output of the MLP to between [0,1], and M s (F) represents the spatial attention map, indicating the importance weight of each spatial position.

[0068] S403. A multi-modal double-cross attention fusion strategy is adopted. The dual-branch features extracted are concatenated through the multi-head attention mechanism to form cross-modal features, and a bidirectional balance module is used to simultaneously learn the cross-attention mapping from the two images. In the first branch, the query comes from the molten pool image features, while the key and value come from the audio features in the GASF map. The input image I and audio A are processed by the learnable weight matrices W i Q ,W i K ,W i V to obtain the query Q, key K, and value V.

[0069] Q = IW i Q (23)

[0070] K = AW i K (24)

[0071] V = AW i V (25)

[0072]

[0073] Where W i Q ,W iK , W i V is a learnable weight matrix, Q I is an image query vector, K A is an audio key vector, V A is an audio value vector, Attention(Q I , K A , V A ) represents the final attention output.

[0074] S404. The cross-attention calculates the similarity between the image features and the audio features, and then further calculates the multi-head cross-attention to capture various features from multiple dimensions. Set the number of heads to h:

[0075]

[0076] Mulitihead = Concat(head1,..., head h )W h (28)

[0077] where head i represents the output of the i-th attention head, represents the image query vector of the i-th head, represents the audio key vector of the i-th head, V i A represents the audio value vector of the i-th head, W h represents the weight matrix.

[0078] S405. Then, the gradient vanishing is alleviated through the residual connection and the normalization layer to stabilize the training, and linearization is performed through the feed-forward network and the rectified linear unit, and finally the first branch feature is output.

[0079] O = LayerNorm(I + Multihead(I)) (29)

[0080] FFN(O) = ReLu(OW1 + b1)W2 + b2 (30)

[0081] O I = LayerNorm(O + FFN(O)) (31)

[0082] where I represents the input image features, Multihead(I) represents the output image features of the multi-head attention, LayerNorm represents layer normalization, + represents the residual connection, O represents the output features of the multi-head attention module, FFN(.) represents the mapping of the feed-forward neural network, W1 and W2 represent the learnable linear transformation weight matrices, b1 and b2 represent the corresponding bias terms, ReLu represents the activation function, which increases the non-linear feature expression ability, O I represents the final output image features of the first branch.

[0083] Similarly, in the second branch, the query comes from the audio features, while the key and value come from the image features, and the output features O of the second branch are calculated in the same way. A Finally, the output features of the two branches are concatenated, and then concatenated with the global image features and global audio features, and sent to the classifier for classification.

[0084] The present invention has the following advantages compared with the prior art:

[0085] 1. A method for identifying the penetration state of multi-sensor welds with an improved attention mechanism provided by the present invention collects the sound signals during the welding process through a microphone sensor, and uses a CCD sensor to collect the molten pool images during the welding process; uses the VMD technology to extract the effective sound signals, and then visualizes the sound signals by using the Gramian Angular Summation Field method; performs de-exposure processing on the overexposed area through the decoupled aggregation exposure correction model; by introducing channel attention and spatial attention mechanisms in the extraction of sound and image features, adaptively samples using deformable convolution and combines the dual-cross attention fusion strategy to fuse multi-source information, and completes the penetration state classification to judge the weld quality. It can effectively and accurately classify the penetration state through the sound signals and molten pool images extracted during the welding process, providing some ideas for weld quality detection.

[0086] 2. A method for identifying the penetration state of multi-sensor welds with an improved attention mechanism provided by the present invention can break through the technical bottlenecks faced by traditional single detection methods in complex working conditions, such as high environmental sensitivity, narrow feature coverage dimension, and incomplete abnormal recognition, and significantly improve the classification accuracy of the penetration state during the welding process and the reliability of weld quality judgment. Brief Description of the Drawings

[0087] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0088] Figure 1 is the specific flowchart of the present invention;

[0089] Figure 2 is the hardware experimental diagram in the present invention;

[0090] Figure 3 is the original signal diagram of VMD decomposition;

[0091] Figure 4 is the effective signal diagram extracted by VMD;

[0092] Figure 5 is the conversion of the optimal modal diagram into the GASF diagram;

[0093] Figure 6 is the original molten pool image collected;

[0094] Figure 7 is the molten pool image processed by de - exposure;

[0095] Figure 8 is the improved ResNet50 network structure diagram;

[0096] Figure 9 is the double - cross attention structure diagram;

[0097] Figure 10 is the penetration feature distribution diagram;

[0098] Figure 11 is the penetration state recognition and classification diagram. Specific Embodiments

[0099] Next, the technical solution of the present invention will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0100] Embodiment 1

[0101] Refer to Figure 1 , a method for recognizing the penetration state of multi - sensor welds with an improved attention mechanism. It mainly includes the following four steps:

[0102] Step S1: Collect the welding process sound signal through a microphone sensor and collect the molten pool image during the welding process using a CCD sensor;

[0103] Step S2: Use the VMD technology to extract the effective sound signal, and then use the Gram - Angle Field (GASF) method to visualize the sound signal;

[0104] Step S3: The overexposed area of the molten pool image is de-exposed by the decoupled aggregation exposure correction model;

[0105] Step S4: By introducing channel attention mechanism and spatial attention mechanism in sound and image feature extraction, using deformable convolution for adaptive sampling and combining dual cross-attention fusion strategy to fuse multi-source information, the penetration state classification is completed to judge the weld quality.

[0106] Reference Figure 2 , the implementation steps of the S1 technical solution provided by the present invention are as follows:

[0107] Step S101: Use a continuous wave CO2 laser welding device to weld HSLA-115 high-strength low-carbon alloy steel with the same area but different thicknesses. The alloy steel is first placed on the welding platform and fixed by clamping plates. The welding platform is smoothly moved by controlling the stepper motor as shown in the figure, so that the welded part moves along the welding direction during the welding process to ensure the stability of the welding state. And use the laser device as shown for welding. Weld penetration experiments are carried out on steel plates of 200mm×100mm×6mm respectively, non-weld penetration experiments are carried out on steel plates of 200mm×100mm×10mm, and weld-through experiments are carried out on steel plates of 200mm×100mm×14mm.

[0108] Step S102: During the welding process, the arc sound signal is acquired by a 4189 microphone sensor. The acquired signal is processed by a 3050A-040 LAN-XI sound and vibration analyzer with four-channel input and then displayed and stored in the Labshop software. At the same time, a CCD sensor is used to collect the molten pool image, and a narrow-band filter and a protective mirror are added in front of the CCD lens to prevent splashing during the welding process. During the collection process, the collection time of the CCD is set by writing a LabVIEW program, and the collected molten pool image is transmitted to a PXI-8252 image acquisition card and stored in a PC after processing.

[0109] In this example, through the continuous wave CO2 laser welding system as shown, combined with the acquisition of arc sound signals and molten pool images, the acquisition of multi-source data under different welding penetration states is effectively realized, laying a solid data foundation for subsequent welding quality feature extraction and recognition. At the same time, HSLA-115 high-strength low-carbon alloy steel plates with different thicknesses are adjusted, and controllable experiments in three states of weld penetration, non-weld penetration and weld-through are successfully realized, providing diverse samples for algorithm training. Also through the multi-sensor synchronous acquisition as shown in the figure, acoustic data and visual data are obtained, combined with the LabVIEW and LabShop systems to ensure the time alignment of the data, providing support for subsequent feature fusion analysis.

[0110] Reference Figure 3 ,Figure 4 , Figure 5 The implementation steps of the S2 technical solution provided by the present invention are as follows:

[0111] Step S201: Use digital processing to convert continuous audio signals into discrete time series. By analyzing the time-domain waveform of the audio signal, calculate the amplitude envelope A(t):

[0112]

[0113] where y(t) is the extracted time series, is the derivative of the signal, representing the rate of change of the signal.

[0114] The original audio sampling frequency is 131 kHz, the total number of sampling points is 900k, and the time is 13 s. The overall amplitude of the original audio sound signal waveform has obvious randomness and a poor signal-to-noise ratio, making it difficult to extract an effective waveform for further processing. Therefore, by calculating the amplitude envelope, the energy change trend of the audio signal is revealed. Since the low-frequency part usually contains the main energy of the signal and carries the main information of the sound, the high-frequency components in the signal are removed, and the effective signal characteristics of the low-frequency are retained to extract an effective sound waveform. The extracted effective waveform is as shown in the original signal part of Figure 3 , with a total of 1000 data points corresponding to 15000 μs.

[0115] Step S202: Decompose the time-domain signal through the VMD technology to obtain multiple intrinsic mode functions IMF, and each intrinsic mode function represents the characteristics of the signal in different frequency bands:

[0116]

[0117] where, u k (t) represents the k-th mode, ω k represents the corresponding central frequency, * represents convolution, represents the partial derivative operation, and δ(t) represents the unit impulse function.

[0118] The constraint condition is:

[0119]

[0120] where, f(t) represents the signal to be decomposed, u k (t) represents the k-th mode

[0121] Introduce the penalty factor a and the Lagrange multiplier λ to transform the constraint into an unconstrained variational problem, solve the above objective function, and the augmented Lagrange expression is:

[0122]

[0123] where \(a\) is a penalty factor, \(\partial\) represents the partial derivative operation, \(\delta(t)\) represents the unit impulse function, and \(u\) k (t) represents the \(k\)th mode, and \(\omega\) k represents the corresponding center frequency, \(f(t)\) represents the signal to be decomposed, and \(\lambda(t)\) is the Lagrange multiplier.

[0124] The multiplicative operator alternating algorithm is used to iteratively solve this problem. By continuously iterating and updating \(u\) k , \(\omega\) k and \(\lambda\), during the update process, the modal components, center frequency, and Lagrange operator expressions are:

[0125]

[0126] where \(f(\omega)\) is the Fourier transform of \(f(t)\), \(\lambda(\omega)\) is the Fourier transform of \(\lambda(t)\), and \(u\) k (\omega)\) is the Fourier transform of \(u\) k (t), and \(\tau\) is the noise tolerance.

[0127] Finally, it is decomposed into the modal diagram shown in Figure 3 . In this diagram, the original signal is decomposed into the modes of IMF1 - IMF9. The abscissa of each mode is selected as 15000 μs, and the ordinate is the amplitude.

[0128] Remove the irrelevant modal noise, extract the optimized signal most relevant to the welding sound state. The finally extracted optimal mode is as shown in Figure 4 . This mode is the optimization of mode 1 in the above Figure 3 . The amplitude of this mode is more regular than other modes and is close to the amplitude of the original low - frequency signal, without obvious high - frequency or mixed components, and has a good signal - to - noise ratio.

[0129] Step S203: The key effective sound signal extracted is a one - dimensional signal as shown in Figure 4 . In order to accurately fuse the features with the two - dimensional molten pool image, the GASF method is used to convert the extracted one - dimensional sound signal into a two - dimensional image as shown in Figure 5 . Taking the extracted sound signal as the original signal, it is normalized to the interval \([-1, 1]\) to eliminate the amplitude difference:

[0130]

[0131] where \(X=\{x_1, x_2, \cdots, x\) N \} is the original signal, and \(x\) i ' is the value after normalization and \(x\) i ' \(\in [-1, 1]\).

[0132] Next, the normalized value x i ′ is converted into the polar coordinate angle φ i :

[0133] φ i = arccos(x i ′) (9)

[0134] where φ i ∈[0, π].

[0135] Construct an N×N matrix, where each element (i, j) represents the sum of the cosine values of the angles φ i and φ j , and finally obtain the GASF diagram.

[0136]

[0137] where GASF(i, j) represents the element value of the matrix at (i, j), that is, the gray value of the corresponding two-dimensional image.

[0138] In this embodiment, the problem of excessive noise and large interference in the audio signal and low quality of the voice signal can be solved. For the problem of excessive clutter and interference and poor signal-to-noise ratio in the original voice signal, the method of VMD decomposition is used, the decomposition mode number K = 9, a = 2500, and finally it is decomposed into Figure 3 the shown modal diagram. The abscissa of each mode is 15000 μs selected, and the ordinate is the amplitude. Then select the optimal mode, and the selected optimal modal diagram is as Figure 4 shown. This mode is optimized from mode 1 in the above Figure 3 . The amplitude of this mode is more regular than that of other modes, the signal-to-noise ratio is good, and the amplitude is between -1V and 1V, which is not much different from the original signal, that is, it is used as the effective signal of the welding sound. Finally, the one-dimensional voice signal such as Figure 4 is converted into the GASF image such as Figure 5 shown by the GASF method. In Figure 4 , the amplitude size of the one-dimensional signal is represented by the color depth of the GASF image. In Figure 4 , the peak appears in 6000 μs - 7500 μs. Similarly Figure 5 , the darkest part is located at the same time position; and Figure 4 the main periodic characteristics in are in 6000 μs - 9000 μs, and there are certain periodic characteristics with low amplitudes in the first 3000 μs and 10500 μs - 13500 μs, which are also shown in Figure 5 .

[0139] Refer to Figure 6 , Figure 7 The implementation steps of the S3 technical solution provided by the present invention are as follows:

[0140] Step S301. There is an exposure error in the molten pool image collected by the CCD sensor and the PXI-8252 image acquisition card. For example, Figure 6 , due to high exposure, the contrast between the bright area and the molten pool edge area is not obvious enough, resulting in the features of the molten pool not being prominent enough. Therefore, an end-to-end exposure correction model is used to process the overexposed area. The model contains a generator, which includes an image encoder, continuous residual blocks, and an image decoder, and decouples contrast enhancement and detail restoration during each convolution process. In the decoupling operation, the local smoothness assumption is adopted. We select the central pixel of R in the local patch as the reference pixel, denoted as p c , and the pixel intensity at other positions can be expressed as the sum of the central pixel intensity and the deviation n i :

[0141] x(p i ) = x(p c ) + n i (11)

[0142] where x(p i ) represents the pixel intensity at any position in the local patch, x(p c ) represents the central pixel intensity (low-frequency component) in the local patch, and n i is the deviation (high-frequency component) of pixel p i relative to p c .

[0143] According to equation (11), the convolution process can be expressed as:

[0144]

[0145] where y(p c ) represents the output of the convolution operation at the central pixel p c , w(p i ) is the weight of the convolution kernel at pixel p i , and x(p i ) represents the pixel intensity at p i .

[0146] It can be observed from equation (12) that in the traditional convolution feature response, the low-frequency response and the high-frequency response are mixed. We separate the high-frequency convolution response from the low-frequency convolution response through the central surrounding difference operation:

[0147]

[0148] where y h (p c ) represents the high-frequency output of the convolution operation at the central pixel p c , and y l (p crepresents the low-frequency output of the convolution operation at the central pixel p c of, w(p i ) is the weight of the convolution kernel at pixel p i , and n i is the bias of pixel p i relative to p c .

[0149] By means of the differential operation around the center, the high-frequency and low-frequency responses can be suppressed respectively. The convolution kernel with high-frequency response in the convolution operation is called the detail perception kernel (DA), and the convolution kernel with low-frequency response becomes the contrast perception kernel (CA). DA and CA are connected in parallel to replace the traditional convolution kernel. However, the differential and addition operations in DA and CA may lead to an imbalance in the amplitudes of the high-frequency and low-frequency responses. To address this, we introduce a dynamic adjustment coefficient on each branch to adjust the amplitude of the feature response for aggregation:

[0150]

[0151] where s is the sigmoid activation function used to constrain the distribution of the adjustment coefficient from 0 to 1. Through continuous training, the adjustment coefficient is continuously updated to balance the response amplitudes of CA and DA.

[0152] Step S302: After using DA and CA to replace the traditional convolution kernel, use the encoder in the generator to extract the image features, and send them to consecutive residual blocks to edit the feature representation. Finally, the changed features generate the final output through the image decoder. At the same time, a discriminator is adopted to distinguish real samples and fake samples and perform adversarial loss representation. It contains several consecutive convolutional layers to downsample the image to produce the final output. And by combining the perceptual loss and the pixel loss, the image quality is finally improved to achieve the de-exposure effect.

[0153]

[0154] L pixel = ||y′ - y||1 (17)

[0155] L = L GAN (G, D)+λL pixel (G)+βL per (G) (18)

[0156] where L per represents the perceptual loss, L pixel represents the pixel loss, c i represents the weight coefficient of the i-th feature layer, y′ represents the generated image, y represents the target image, L i (y) represents the feature map of the target image at the i-th feature layer, L i(y′) represents the feature map of the generated image at the i-th feature layer, ||||1 represents the L1 norm, D represents the discriminator, G represents the generator, λ and β are coefficients indicating the impact of each loss on the final loss, and λ, β ∈ [0, 1].

[0157] In this embodiment, through the decoupled aggregation exposure correction model, the detail perception and contrast perception have been greatly improved. As Figure 6 shown, due to the excessive temperature of the welding head, the overall image is overexposed, resulting in some circular spots inside the image being blurred, the details being unclear, and the contrast between the molten pool area and the background being low, resulting in the edge of the molten pool being unclear and the overall visual effect being poor. In Figure 7 after the de-exposure processing, the image details are more obvious, the spot contours are clearer, and at the same time, the edge contours of the molten pool are also more distinct.

[0158] Refer to Figure 8 , 9, 10, 11 The implementation steps of the S3 technical solution provided by the present invention are as follows:

[0159] S401. Feature extraction is performed on the molten pool image and the GASF image through the improved ResNet50 network. In the traditional ResNet50 network, as Figure 8 shown, the last fully connected layer is deleted, and the image features are finally output. Since the shape of the molten pool in the molten pool image is irregular, deformable convolution is used to replace some traditional convolution kernels. Deformable convolution introduces a learnable offset for each sampling point, enabling the convolution kernel to adaptively adjust the sampling area according to the local content of the input feature map.

[0160]

[0161] Among them, y(p0) represents the value of the output feature map of the traditional convolution kernel at position p0, y′(p0) represents the value of the output feature map of the deformable convolution kernel at position p0, R is the receptive field of the convolution kernel, p n represents the sampling position of the convolution kernel within the receptive field R, p0 represents a certain position on the output feature map, w(p n ) represents the weight of the convolution kernel at p n , and Δp n is the learnable offset.

[0162] In Figure 8 , in the second stage of the traditional ResNet50 network, the contour features have been initially revealed at this time, but the details are still blurred. The 3×3 convolution kernel in the second bottleneck block is replaced with deformable convolution, and the convolution kernel initially adapts to the extracted feature shape by dynamically learning the offset. In the third and fourth stages, all 3×3 convolution kernels are replaced with deformable convolution to extract all the deep feature changes, completing the change from adapting to the shape to screening features.

[0163] S402. Meanwhile, in order to meet the accuracy of adaptive feature extraction, the CBAM module is added. In the process of GASF image features, emphasis is placed on the channel attention mechanism. By performing global average pooling on the input feature map, the information of each channel is compressed into a scalar, and the weights of each channel are calculated through a fully connected layer, dynamically enhancing the key channels related to the periodic features of the time series and suppressing redundant or noisy channels.

[0164] M c (F) = σ(MLP(AvgPool(F)) + MLP(MaxPool(F))) (21)

[0165] Where F represents the input feature map, AvgPool(F) represents the average pooling operation of the input feature map F, MaxPool(F) represents the maximum pooling operation of the input feature map F, MLP(·) represents the multi-layer perceptron, which further processes the pooled features, and σ(·) is the Sigmoid function, mapping the output of the MLP to between [0,1], M c (F) represents the channel attention map, indicating the importance weights of each channel.

[0166] In the process of molten pool image feature extraction, emphasis is placed on the spatial attention mechanism. Global maximum pooling and average pooling are performed on the input feature map along the channel dimension, and the two feature maps are concatenated to generate a spatial weight map. The spatial weight map is multiplied with the original feature map position by position to highlight the key region features related to the dynamics of the molten pool in the molten pool image.

[0167] M s (F) = σ(Conv([AvgPool(F); MaxPool(F)])) (22)

[0168] Where F represents the input feature map, AvgPool(F) represents the average pooling operation of the input feature map F, MaxPool(F) represents the maximum pooling operation of the input feature map F, [AvgPool(F); MaxPool(F)] represents concatenating the results of average pooling and maximum pooling, Conv(·) represents a three-dimensional convolutional layer, which performs a convolutional operation on the concatenated features, σ(·) is the Sigmoid function, mapping the output of the MLP to between [0,1], M s (F) represents the spatial attention map, indicating the importance weights of each spatial position.

[0169] When inputting the GASF image as shown in Figure 5 and the molten pool image as shown in Figure 7 through as shown in Figure 8The improved feature extraction network shown combines deformable convolution for feature extraction. By adding the CBAM module in the third stage, not only can the excessive computational load caused by the large feature maps in the first two stages be avoided, but also the accurate recognition of key features of high-level semantics can be achieved. For GASF images, through Figure 8 the learned offset introduced by the deformable convolution in, the local feature differences caused by periodic changes in the GASF map can be accurately captured, and through Figure 8 in the channel attention mechanism of the CBAM module, the key channels closely related to the periodic features of the time series are strengthened, the noise channels are suppressed, and the sensitivity of the model to features such as periodicity is improved; for the molten pool image, the deformable convolution effectively extracts the key edges and contour features in the morphological changes of the molten pool, reduces information loss, and through Figure 8 when using the CBAM module, after the feature map channels are concatenated by global average pooling and max pooling through the spatial attention mechanism, the generated weight map more accurately focuses on the changing areas inside the molten pool, highlighting the dynamic features of the molten pool.

[0170] S403. A multi-modal double cross-attention fusion strategy is adopted. The dual-branch features extracted are concatenated through the double cross-attention mechanism shown in Figure 9 to form cross-modal features, and a bidirectional balance module is used to simultaneously learn the cross-attention maps from the two types of images. In the first branch, the query comes from the features of the molten pool image, while the key and value come from the audio features in the GASF map. The input images I and audio A are processed by the learnable weight matrices W i Q , W i K , W i V to obtain the query Q, key K, and value V.

[0171] Q = IW i Q (23)

[0172] K = AW i K (24)

[0173] V = AW i V (25)

[0174]

[0175] where W i Q , W i K , W i V are learnable weight matrices, and Q Iis the image query vector, K A is the audio key vector, V A is the audio value vector, Attention(Q I , K A , V A ) represents the final attention output.

[0176] Cross-attention calculates the similarity between image features and audio features, and then further calculates multi-head cross-attention to capture various features from multiple dimensions. Set the number of heads to h:

[0177]

[0178] Mulitihead = Concat(head1,..., head h )W h (28)

[0179] where head i represents the output of the i-th attention head, represents the image query vector of the i-th head, represents the audio key vector of the i-th head, V i A represents the audio value vector of the i-th head, W h represents the weight matrix.

[0180] Then, gradient vanishing is alleviated through residual connections and normalization layers to stabilize training, and linearization is performed through a feed-forward network and a rectified linear unit. After another residual connection and normalization, the first-branch features are finally output.

[0181] O = LayerNorm(I + Multihead(I)) (29)

[0182] FFN(O) = ReLu(OW1 + b1)W2 + b2 (30)

[0183] O I = LayerNorm(O + FFN(O)) (31)

[0184] where I represents the input image features, Multihead(I) represents the output image features of multi-head attention, LayerNorm represents layer normalization, + represents the residual connection, O represents the output features of the multi-head attention module, FFN(.) represents the mapping of the feed-forward neural network, W1, W2 represent the weight matrices of learnable linear transformations, b1, b2 represent the corresponding bias terms, ReLu represents the activation function to increase the non-linear feature expression ability, and O I represents the final output image features of the first branch.

[0185] Similarly, in the second branch, the query comes from audio features, while the key and value come from image features, and the output feature O of the second branch is also calculated. A . Finally, the output features of the two branches are concatenated, and then concatenated with the image global feature and the audio global feature, and sent to the classifier for classification.

[0186] Feature fusion Concat(O I ,O A ,I,A) (32) where O I represents the final output image feature of the first branch, O A represents the final output audio feature of the second branch, I represents the image global feature, and A represents the audio global feature.

[0187] In this embodiment, through the improved ResNet50 network as Figure 8 shown, the features such as the main molten pool shape after de-exposure in Figure 7 and the periodicity and color depth features in Figure 5 are extracted to lay the foundation for further fusion training. For Figure 7 , Figure 5 the key features in are trained and learned through the double-cross attention fusion strategy as Figure 9 shown, and the corresponding output features are obtained for the two branches respectively, and concatenated with the global features. The concatenated features of the 60 groups of test data are as Figure 10 shown. In this figure, there are 20 groups of different fused features for burn-through, full penetration, and incomplete penetration respectively. Project them into one-dimensional and two-dimensional spaces respectively to more intuitively make a judgment on the fused features. Finally, the data in the test samples are classified and tested. As Figure 11 shown, the blue columns are the predicted results of the test samples, and the broken lines are the actual results of the test samples. Finally, the 60 groups of sample data can be accurately classified, thus realizing the discrimination of the weld quality.

[0188] Obviously, the above embodiments are only examples clearly described and not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation manners here. And the obvious changes or modifications derived therefrom are still within the protection scope of this invention.

Claims

1. A multi-sensor weld penetration state recognition method with an improved attention mechanism, characterized in that: The steps include: S1, collecting welding process sound signals and collecting welding pool images; S2, using VMD technology to extract effective acoustic signals, and then using Gram's angular field method to visualize the acoustic signals; S3, de-exposure the overexposed area of the melt pool image through the decoupled aggregation exposure correction model; S4. By introducing channel attention and spatial attention mechanisms in sound and image feature extraction, deformable convolution adaptive sampling is used and combined with a double cross attention fusion strategy to fuse multi-source information, the penetration state classification is completed to determine the weld quality.

2. The multi-sensor weld penetration state recognition method with improved attention mechanism according to claim 1 is characterized in that: In step S1, high-strength low-carbon alloy steels of the same area but different thicknesses are welded using a laser welding device. An incomplete penetration test is performed on a 200 mm × 100 mm × 6 mm steel plate, a penetration test is performed on a 200 mm × 100 mm × 8 mm steel plate, and a penetration test is performed on a 200 mm × 100 mm × 10 mm steel plate. During the welding process, arc sound signals are detected by a microphone sensor. The detected signals are processed by a sound and vibration analyzer, displayed in the Labshop software, and stored. At the same time, a CCD sensor is used to capture the molten pool image, and a narrow-band filter and a protective mirror are added in front of the CCD lens to prevent spatter during welding. During the acquisition process, the CCD acquisition time is set by writing a LabVIEW program, and the collected molten pool image is transmitted to the image acquisition card, which is processed and stored in the PC.

3. The multi-sensor weld penetration state recognition method with improved attention mechanism according to claim 1 is characterized in that: The S2 specifically includes the following steps: S2.

1. The audio signal is represented as a discrete time series through sampling. By analyzing the time domain waveform of the audio signal, the amplitude envelope A(t) is calculated: Where y(t) is the extracted time series, is the derivative of the signal, which represents the rate of change of the signal; By calculating the amplitude envelope, the energy change trend of the audio signal is revealed. The low-frequency part contains the main energy of the signal and carries the main sound information. The high-frequency components in the signal are removed, the effective signal characteristics of the low frequency are retained, and the effective sound waveform is extracted. S2.

2. Decompose the time domain signal using VMD technology to obtain multiple intrinsic mode functions (IMFs), each of which represents the characteristics of the signal in different frequency bands: Among them, u k (t) represents the kth mode, ω k Indicates the corresponding center frequency, * indicates convolution, represents the partial derivative operation, δ(t) represents the unit impulse function; The constraints are: Among them, f(t) represents the decomposed signal, u k (t) represents the kth mode The penalty factor a and the Lagrange multiplier λ are introduced to transform the constraint into an unconstrained variational problem. The augmented Lagrange expression for solving the above objective function is: Where a is the penalty factor, represents the partial derivative operation, δ(t) represents the unit pulse function, u k (t) represents the kth mode, ω k represents the corresponding center frequency, f(t) represents the decomposed signal, and λ(t) is the Lagrange multiplier; The multiplication operator alternation algorithm is used to iteratively solve this problem, by continuously iteratively updating u k 、ω k and λ, the modal components and center frequencies are expressed as Lagrangian operators during the updating process: Where f(ω) is the Fourier transform of f(t), λ(ω) is the Fourier transform of λ(t), and u k (ω) is u k Fourier transform of (t), τ is the noise tolerance; Extract effective pulses from the decomposed modes, remove non-correlated modal noise, and extract the optimized signal most relevant to the welding sound state; S2.

3. The extracted key effective acoustic signal is a one-dimensional signal. In order to accurately fuse the features with the two-dimensional melt pool image, the GASF method is used to convert the extracted one-dimensional acoustic signal into a two-dimensional image. The extracted acoustic signal is taken as the original signal and normalized to the interval [-1, 1] to eliminate the amplitude difference: Where X={x1,x2,...,x N } is the original signal, x i ′ is the normalized value and x i ′∈[-1,1]. Then the normalized value x i ′ is converted to polar coordinate angle φ i : f i =arccos(x i ′) (9) among them i ∈[0,π]; Construct an N×N matrix where each element (i, j) represents the angle φ i and φ j The sum of the cosine values of , and finally the GASF graph is obtained; Where GASF(i,j) represents the element value of the matrix at (i,j), which corresponds to the grayscale value of the two-dimensional image.

4. The multi-sensor weld penetration state recognition method with improved attention mechanism according to claim 1, characterized in that: The S3 specifically includes the following steps: S3.

1. Exposure errors exist in the melt pool image captured by the sensor and acquisition card. An end-to-end exposure correction model is used to process overexposed areas. The model consists of a generator that includes an image encoder, a continuous residual block, and an image decoder. Contrast enhancement and detail recovery are decoupled in each convolution process. The decoupling operation adopts a local smoothness assumption and selects the central pixel of R in the local patch as the reference pixel, which is denoted as p. c , the pixel intensity at other locations can be expressed as the center pixel intensity and the deviation n i sum: x(p i )=x(p c )+n i (11) where x(p i ) represents the pixel intensity at any position within the local patch, x(p c ) represents the pixel intensity at the center of the local patch, n i is pixel p i Relative to p c deviation. According to formula (11), the convolution process is expressed as: where y(p c ) indicates that the convolution operation is performed at the center pixel p c Output, w(p i ) The convolution kernel is at pixel p i The weight at x(p i ) indicates p i The pixel intensity at ; From Equation (12), we can observe that in the traditional convolution feature response, the low-frequency response and the high-frequency response are mixed; the high-frequency convolution response is separated from the low-frequency convolution response by the center-around difference operation: where y h (p c ) indicates that the convolution operation is performed at the center pixel p c High frequency output, where y l (p c ) indicates that the convolution operation is performed at the center pixel p c The low frequency output, w(p i ) The convolution kernel is at pixel p i The weight at n i is pixel p i Relative to p c Deviation; Through center-surround difference operations, high and low frequency responses are suppressed respectively. The convolution kernel for high frequency response in the convolution operation is called detail perception kernel, and the convolution kernel for low frequency response is called contrast perception kernel. DA and CA are connected in parallel to replace the traditional convolution kernel. A dynamic adjustment coefficient is introduced on each branch to adjust the amplitude of the feature response for aggregation: Where s is the sigmoid activation function used to constrain the distribution of the adjustment coefficient from 0 to 1; through continuous training, the adjustment coefficient is continuously updated to balance the response amplitudes of CA and DA. S3.

2. After replacing traditional convolution kernels with DA and CA, the encoder in the generator extracts image features, passes them to successive residual blocks, edits the feature representation, and finally passes the modified features through the image decoder to produce the final output. A discriminator is also used to distinguish real samples from fake ones, and an adversarial loss representation is applied. It consists of several consecutive convolutional layers that downsample the image to produce the final output. It combines perceptual loss and pixel loss to ultimately improve image quality and achieve a de-exposure effect. L pixel =||y′-y||1 (17) L=L GAN (G,D)+λL pixel (G)+βL per (G) (18) Among them L per represents the perceptual loss, L pixel represents pixel loss, c i represents the weight coefficient of the i-th feature layer, y′ represents the generated image, y represents the target image, L i (y) represents the feature map of the target image at the i-th feature layer, L i (y′) represents the feature map of the generated image at the i-th feature layer, || ||1 represents the L1 norm, D represents the discriminator, G represents the generator, λ, β are coefficients, indicating the impact of each loss on the final loss, and λ, β∈[0,1].

5. The multi-sensor weld penetration state recognition method with improved attention mechanism according to claim 1, characterized in that: The S4 specifically includes the following steps: S4.

1. After obtaining the GASF image, feature extraction is performed on the melt pool image and the GASF image using an improved ResNet50 network. Due to the irregular shape of the melt pool in the melt pool image, deformable convolution is used to replace some traditional convolution kernels. Deformable convolution introduces a learnable offset for each sampling point, enabling the convolution kernel to adaptively adjust the sampling area based on the local content of the input feature map. Where y(p0) represents the value of the traditional convolution kernel output feature map at position p0, y′(p0) represents the value of the deformable convolution kernel output feature map at position p0, R is the receptive field of the convolution kernel, and p n represents the sampling position of the convolution kernel in the receptive field R, p0 represents a position on the output feature map, w(p n ) indicates that the convolution kernel is in p n The weight at Δp n is the learnable offset; S4.

2. To ensure the accuracy of adaptive feature extraction, a CBAM module was added. Channel attention and spatial attention mechanisms were introduced in sound and image feature extraction. The channel attention mechanism compresses each channel information into a scalar by performing global average pooling on the input feature map and calculates the weight of each channel through a fully connected layer. This dynamically enhances key channels related to the periodic characteristics of the time series and suppresses redundant or noisy channels. M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (21) Where F represents the input feature map, AvgPool(F) represents the average pooling operation of the input feature map F, MaxPool(F) represents the maximum pooling operation of the input feature map F, MLP(·) represents the multi-layer perceptron, which further processes the pooled features, σ(·) is the Sigmoid function, which maps the output of MLP to the range [0, 1]. c (F) represents the channel attention map, which indicates the importance weight of each channel; The spatial attention mechanism performs global maximum pooling and average pooling on the input feature map along the channel dimension, concatenating the two feature maps to generate a spatial weight map. The spatial weight map is multiplied position by position with the original feature map to highlight key regional features related to the melt pool dynamics in the melt pool image. M s (F)=σ(Conv([AvgPool(F);MaxPool(F)])) (22) Where F represents the input feature map, AvgPool(F) represents the average pooling operation of the input feature map F, MaxPool(F) represents the maximum pooling operation of the input feature map F, [AvgPool(F); MaxPool(F)] means concatenating the average pooling and maximum pooling results, Conv(·) represents a three-dimensional convolutional layer, which performs a convolution operation on the concatenated features, σ(·) is a Sigmoid function, which maps the output of MLP to the range [0, 1]. s (F) represents the spatial attention map, which indicates the importance weight of each spatial position; S4.3, a multimodal double cross attention fusion strategy is adopted; the extracted double-branch features are spliced through the multi-head attention mechanism to form cross-modal features and a bidirectional balancing module is used to simultaneously learn the cross attention mapping from the two images; in the first branch, quary comes from the melt pool image features, while key and value come from the audio features in the GASF graph; the input image I and audio A are represented by a learnable weight matrix W i Q ,W i K , W i V Operation to obtain query Q, key K, value V; Q2IW i Q (23) K=AW i K (24) V=AW i V (25) Where W i Q ,W i K , W i V is the learnable weight matrix, Q I is the image query vector, K A is the audio key vector, V A is the audio value vector, Attention(Q I ,K A ,V A ) represents the final attention output; Cross attention calculates the similarity between image features and audio features, and then further calculates multi-head cross attention to capture various features from multiple dimensions; the number of heads is set to h: Mulitihead=Concat(head1,...,head h )W h (28) Among them head i represents the output of the i-th attention head, represents the image query vector of the i-th head, represents the audio key vector of the i-th head, V i A represents the audio value vector of the i-th head, W h represents the weight matrix; Then, residual connections and normalization layers are used to alleviate gradient vanishing and stabilize training. Linearization is performed through a feedforward network and a linear correction unit, and finally the first branch features are output. In the second branch, quary comes from audio features, while key and value come from image features. The output features of the second branch are calculated in the same way. Finally, the output features of the two branches are spliced, and then spliced with the global image features and the global audio features, and sent to the classifier for classification.

Images