Battery charge state detection method based on deep learning and ultrasonic feature fusion

Abstract

The invention discloses a battery state-of-charge detection method based on deep learning and ultrasonic feature fusion. The method comprises the following steps: collecting and preprocessing battery ultrasonic signals and battery electrical parameters; extracting an ultrasonic amplitude; extracting and fusing the characteristics of the battery voltage, the surface temperature, the ultrasonic amplitude and the phase difference; the method comprises the following steps of: optimizing parameter values of a BiTCN-BiLSTM-Multi-Head Attention network on the basis of a WAA weighted average optimization algorithm, extracting basic characteristics related to the SOC of the battery in a characteristic matrix on the basis of a bidirectional time convolution network BiTCN, extracting long-term information related to the SOC of the battery in the basic characteristics on the basis of a BiLSTM network, extracting important characteristics related to the SOC of the battery in the long-term information on the basis of a Multi-Head Attention network, and calculating the SOC of the battery according to the extracted important characteristics. And outputting a predicted value of the SOC of the battery through the full connection layer. According to the invention, the estimation precision of the battery SOC can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of battery state of charge detection, and in particular to a battery state of charge detection method based on deep learning and ultrasonic feature fusion. Background Art

[0002] The state of charge (SOC) of a lithium battery represents the ratio of the remaining charge to the total capacity of the battery, expressed as a percentage. This directly reflects the current capacity of the lithium battery and plays an important role in its safety and performance. However, due to the complex chemical reactions within lithium batteries during operation, its state of charge cannot be directly measured using instruments. Instead, it can be indirectly predicted using parameters such as voltage, current, discharge current, and discharge time during battery operation. Different prediction algorithms have different predictive capabilities, which can affect the detection of battery SOC.

[0003] Currently, the ampere-hour integration method is commonly used to predict the SOC of lithium batteries. This method assumes that the battery capacity is constant and ignores errors caused by factors such as battery temperature and usage loss. The current SOC of the battery is calculated by integrating the current with time during the battery's operation. The ampere-hour integration method is easily affected by factors such as the initial capacity of the battery, accumulated errors, and internal losses. The prediction accuracy is very limited, and it is difficult to obtain high accuracy.

[0004] In addition, the existing technology also uses ultrasound to predict the SOC of lithium batteries, and uses ultrasound as an information medium to reflect the current SOC of the battery. For example, patent publication number CN117054527A is a device and method for non-destructive testing of lithium batteries based on ultrasonic resonance spectrum, which uses two ultrasonic probes to transmit and receive ultrasound respectively, and obtains ultrasonic resonance spectra based on transmitted waves and reflected waves at different positions of the lithium battery, from which information such as the amplitude of the transmitted waves at different frequency points is extracted, and the battery SOC is quantitatively estimated. This technical solution requires scanning and measuring different spatial positions on the surface of the lithium battery. The number of scanning points is large, which takes a certain amount of time and has a high time cost; the need to obtain the resonance spectrum of the lithium battery means that the ultrasonic probe needs to emit multiple ultrasonic waves of different frequencies, which takes a certain amount of time and has a high time cost; and relying solely on ultrasound as a carrier of battery SOC information, too few factors are considered, which will have an adverse effect on the accuracy of SOC warning;

[0005] For example, patent publication number CN118566748A discloses an online SOC estimation method for lithium batteries in a small satellite power system. This method establishes a temperature-dependent dual-polarization equivalent circuit model as an equivalent model for the internal parameters of the lithium battery, and uses this equivalent circuit model to estimate the SOC parameters of the lithium battery. This technical solution is highly dependent on the accuracy of parameters such as the internal chemical parameters and result parameters of the battery, and most of these parameters cannot be accurately obtained in real time, and the parameters of the equivalent model cannot be adjusted in real time. As a result, the method suffers from limited prediction accuracy and poor real-time performance. Summary of the Invention

[0006] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a battery state of charge detection method based on deep learning and ultrasonic feature fusion. Based on WAA-BiTCN-BiLSTM-Multi-Head Attention (weighted average algorithm-bidirectional temporal convolutional network-bidirectional long short-term memory network-multi-head attention mechanism), the method utilizes the phase difference characteristics of the waveform before and after the ultrasonic wave passes through the battery and the temperature characteristics of the surface during the battery operation. The battery state of charge detection algorithm is used to extract and analyze feature information, thereby obtaining the current state of charge parameters of the battery, thereby realizing real-time and high-precision detection of the battery state of charge parameters.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The present invention provides a battery state of charge detection method based on deep learning and ultrasonic feature fusion, comprising the following steps:

[0009] Collect battery ultrasonic signals and battery electrical parameters and perform data preprocessing;

[0010] Extracting the ultrasonic amplitude of the ultrasonic signal based on fast Fourier transform;

[0011] The battery voltage, surface temperature, ultrasonic amplitude, and phase difference are extracted to obtain the features of different modes. The features of different modes are weighted and fused, and the feature matrix is ​​obtained after dimensionality reduction processing.

[0012] Build a BiTCN-BiLSTM-Multi-Head Attention network and take the feature matrix as input;

[0013] The parameter values ​​in the BiTCN-BiLSTM-Multi-Head Attention network are optimized based on the WAA weighted average optimization algorithm. The basic features related to the battery SOC in the feature matrix are extracted based on the bidirectional temporal convolutional network BiTCN. The long-term information related to the battery SOC in the basic features is extracted based on the BiLSTM network. The important features related to the battery SOC in the long-term information are extracted based on the Multi-Head Attention network. Multiple important features are mapped to a one-dimensional space through a fully connected layer, and the predicted value of the battery SOC is output.

[0014] As a preferred technical solution, data preprocessing includes: using a least squares smoothing filter function to perform smoothing and noise reduction.

[0015] As a preferred technical solution, the parameter values ​​in the BiTCN-BiLSTM-Multi-HeadAttention network are optimized based on the WAA weighted average optimization algorithm, specifically including:

[0016] The parameters in the BiTCN-BiLSTM-Multi-Head Attention network include the number of BiLSTM units, learning rate, and regularization coefficient;

[0017] Initialize the candidate solution matrix of BiLSTM unit number, learning rate, and regularization coefficient;

[0018] Calculate the fitness values ​​of the candidate solutions for the number of BiLSTM units, learning rate, and regularization coefficient, sort the candidate solutions according to the fitness values, and select the top N Candidate The weighted average position of candidate solutions is calculated, where the top N Candidate The candidate solutions are expressed as:

[0019]

[0020] Among them, n represents the number of individuals participating in the calculation of weighted centers in the current iteration, nP represents the total population size set by the WAA weighted average optimization algorithm, it represents the current number of iterations; Max It Indicates the maximum number of iterations of the WAA weighted average optimization algorithm;

[0021] Perform location updates during the development phase;

[0022] In the exploration phase, the search is performed based on Levy Flight exploration and random search strategies;

[0023] Output the optimal values ​​of the parameters in the BiTCN-BiLSTM-Multi-Head Attention network.

[0024] As a preferred technical solution, location updating is performed during the development phase. The location updating strategies include:

[0025] The position update is performed based on the global and personal optimal positions, which is expressed as:

[0026]

[0027] The position update is based on the personal optimal position, which is expressed as:

[0028]

[0029] The position update is performed based on the global optimal position, which is expressed as:

[0030]

[0031] in, represents the value of the i-th parameter of the k-th individual at the t-th iteration, i = 1, 2, 3, respectively representing the number of BiLSTM units, learning rate, and regularization coefficient, μ i (t) represents the weighted center of the i-th parameter, which is calculated by weighting the i-th parameter values ​​of the first n better individuals, x i,GlobalBest (t) represents the global optimal solution of the i-th parameter, represents the i-th parameter in the k-th individual's historical optimal solution, w 11 、w 12 、w 13 、w 21 、w 22 、w 31 、w 32 Represents the corresponding random weight.

[0032] As a preferred technical solution, the basic features related to battery SOC in the feature matrix are extracted based on the bidirectional temporal convolutional network BiTCN, specifically including:

[0033] The bidirectional temporal convolutional network BiTCN processes the forward and backward time series information through the forward TCN network and the backward TCN network respectively. The feature matrix is ​​input into the forward TCN network in the positive time sequence and outputs the forward output. The feature matrix is ​​input into the backward TCN network in the reverse time sequence and outputs the backward output.

[0034] The forward output and reverse output are spliced ​​together to obtain the basic characteristics of the battery SOC in the forward and reverse directions.

[0035] As a preferred technical solution, the bidirectional temporal convolutional network BiTCN processes forward and backward time series information through the forward TCN network and the backward TCN network respectively, specifically including:

[0036] Calculate the dilation factor ds :

[0037] d s =sigmoid(W d ·h s +b d )

[0038] Among them, W d and b d is a learnable parameter, h s It is the feature representation of the feature matrix after a convolution layer, and sigmoid represents the sigmoid function;

[0039] The feature matrix is ​​input into the forward TCN network in the positive order of time. After being processed by multiple TCN blocks, the dilated causal convolution calculation is performed in each TCN block, which is specifically expressed as:

[0040]

[0041] Among them, F(s) represents the convolution result of the dilated causal convolution, f(i) is the weight of the convolution kernel, Represents the backtracking of the battery input parameter state at the historical moment, k is the convolution kernel size;

[0042] Input the convolution result F(s) of the dilated causal convolution into the activation function ReLU to obtain the activation value F of the convolution result act (s), after the activation value is subjected to residual connection and Dropout operation, the convolution result of the TCN convolution block is obtained, and the output results of all TCN blocks in the forward TCN network are superimposed to obtain the forward output F forward (s);

[0043] The structure of the reverse TCN network is the same as the forward TCN network. The feature matrix is ​​input into the backward TCN network in reverse time order, and the reverse output F is output. backward (s);

[0044] The forward output F forward (s) and reverse output F backward (s) The basic characteristics of the battery SOC in the forward and reverse directions are obtained by splicing.

[0045] As a preferred technical solution, the long-term information related to battery SOC in the basic features is extracted based on the BiLSTM network, specifically including:

[0046] The basic features are divided into multiple time steps, each time step contains an n-dimensional feature vector. The BiLSTM network includes a forward LSTM network and a backward LSTM network;

[0047] The basic features are input into the forward LSTM network from left to right, and the output h of each time step is t Depends on the current input x basic,t and the output h of the previous time step t-1 ;

[0048] The basic features are input into the backward LSTM network from right to left, and the output h′ of each time step is t Depends on the current input x basic,t and the output h′ of the next time step t+1 ;

[0049] The outputs of the forward LSTM network and the backward LSTM network at each time step are combined into an information sequence [h t ; h′ t ], the combined information sequence is output to the output layer to obtain long-term information related to the battery SOC.

[0050] As a preferred technical solution, in the forward LSTM network and the backward LSTM network, the calculation of the forget gate, input gate, candidate cell state, cell state update, and output gate is performed at each time step, specifically including:

[0051] Cell state variable C t What is stored is the basic features related to the battery SOC extracted from the bidirectional temporal convolutional network BiTCN at the current time t and all previous times, and processed through the forget gate:

[0052] f t =σ(W fx x basic,t +W fh h t-1 +b f )

[0053] Among them, f t is the output of the forget gate, σ is the Sigmoid activation function, x basic,t is the input vector of the LSTM network at the current time step t, i.e., the basic features related to the battery SOC, h t-1 is the hidden state of the previous time step t-1, W fx is the input x basic,t The weight matrix, W fh It is input h t-1 The weight matrix, b f is the bias term of the forget gate;

[0054] Control the flow of new information into the cell state through the input gate:

[0055] i t =σ(W ix x basic,t+W ig h t-1 +b i )

[0056] Among them, i t represents the activation vector of the input gate at time step t, σ represents the Sigmoid function, whose output range is between 0 and 1, indicating the degree of opening of the input gate, W ix Represents the weight matrix of input data to the input gate, W ih Represents the previous hidden state h t-1 The weight matrix to the input gate, b i represents the bias vector of the input gate;

[0057] The candidate states are represented as:

[0058]

[0059] Among them, W cx Represents the weight matrix input to the candidate state, W ch Represents the weight matrix from hidden state to candidate state, b c Bias vector representing the candidate state;

[0060] Update the cell state:

[0061]

[0062] Among them, f t Represents the forget gate vector, which determines the previous cell state C t-1 The retention ratio, C t-1 Indicates the previous cell state, i t represents the activation vector of the input gate at time step t;

[0063] The activation value of the output gate is:

[0064] o t =σ(W ox x t +W oh h t-1 +b o )

[0065] Among them, W ox Represents the weight vector from input to output gate, x t represents the input vector, W oh Represents the weight matrix from the hidden state to the output gate, b o Represents the bias vector of the output gate;

[0066] Filter the characteristic information related to battery SOC in the cell state and generate the hidden state:

[0067] ht =o t ⊙tanh(C t )

[0068] Among them, h t is the hidden state at the current time step t, o t is the activation value of the output gate, C t is the cell state at the current time step, ⊙ represents element-wise multiplication;

[0069] Define update gate z t and reset gate r t :

[0070] z t =σ(W z ·[h t-1 ;x t ]+b z )

[0071] r t =σ(W r ·[h t-1 ;x t ]+b r )

[0072] Among them, W z 、W r is the learnable weight matrix, b z 、b r is the bias term, σ is the sigmoid activation function;

[0073] The hidden state h is compressed by the compression mechanism t After compression, the hidden state after compression is expressed as:

[0074]

[0075] in, represents the compressed hidden state, tanh is the hyperbolic tangent activation function, W c represents the learnable weight matrix, b c represents the bias term;

[0076] Combine the gating mechanism and compression mechanism to update the hidden state h t :

[0077]

[0078] Where 1–z t Indicates the proportion of the hidden state retained at the previous moment, z t Indicates the ratio of updating the hidden state at the current moment;

[0079] The forward hidden state output is concatenated with the reverse hidden state output to obtain long-term information related to the battery SOC.

[0080] As a preferred technical solution, the Multi-Head Attention network is used to extract important features related to battery SOC from long-term information, specifically including:

[0081] Based on the multi-head attention mechanism, different weights are assigned to long-term information for training to obtain the weighted feature representation of attention. The weighted feature results of attention are input into the fully connected layer, and the fully connected layer outputs the predicted value of the battery SOC.

[0082] As a preferred technical solution, different weights are assigned to long-term information based on the multi-head attention mechanism for training. The weight distribution result is expressed as:

[0083] For the i-th attention head:

[0084]

[0085] The outputs of h attention heads are fused to obtain the final attention-weighted feature representation:

[0086] A=Concat(a 1 ,a 2 ,…,a h )W o

[0087] in, is the attention probability distribution function of the i-th attention head, is the attention weight matrix corresponding to the i-th attention head, h t is the hidden state at the current time step t, h s is the hidden state at the s-th time step, is the attention score of the i-th attention head, a i is the attention-weighted feature representation of the i-th attention head, W o is the weight matrix used to fuse multiple head outputs.

[0088] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0089] (1) Unlike the previous method of collecting ultrasonic information by using pulse waves, the present invention collects ultrasonic signals by using two continuous waves, extracts the amplitude and phase difference as acoustic parameters, and estimates the battery SOC by combining the battery voltage and surface temperature of the battery electrical parameters. The ultrasonic detection method applied to the physical surface of the battery does not cause changes in the internal components of the battery and is not affected by the internal structure of the battery itself. The experimental hardware requirements are not high during the detection, while the detection accuracy can be guaranteed, which has certain advantages in terms of cost and time.

[0090] (2) The present invention realizes real-time estimation of battery SOC based on the battery estimation model of WAA-BiTCN-BiLSTM-Multi-Head Attention network. The BiTCN network mainly captures local temporal dependencies, the BiLSTM network models long temporal dependencies on a longer time scale, and the multi-head attention mechanism can dynamically learn the relationship between different time points in the time series data. By combining these three, the battery estimation model can pay attention to local and global information at the same time, understand different patterns in the time series data from multiple perspectives, and help improve the estimation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 Schematic diagram of the process of the battery state of charge detection method based on deep learning and ultrasonic feature fusion of the present invention;

[0092] Figure 2 Schematic diagram of the overall architecture of the WAA-BiTCN-BiLSTM-Multi-Head Attention network of the present invention;

[0093] Figure 3 Schematic diagram of the network structure of the BiTCN residual block of the present invention;

[0094] Figure 4 (a) is a schematic diagram of the overall architecture of the LSTM network of the present invention;

[0095] Figure 4 (b) is a schematic diagram of the overall architecture of the BiLSTM network of the present invention. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0097] like Figure 1 As shown, this embodiment provides a battery state of charge detection method based on deep learning and ultrasonic feature fusion. This embodiment is preferably described using a lithium battery. An ultrasonic probe generates ultrasonic waves, which pass through the inside of the battery. The ultrasonic waves propagate inside the battery. Under different SOC states, the amplitude and phase difference of the ultrasonic waves will change. When the SOC of the lithium battery increases from small to large, the amplitude of the ultrasonic waves will first decrease, then increase and gradually tend to be constant. When the SOC is zero, its amplitude is the largest. When the SOC reaches the maximum capacity of the battery, its value is still less than the amplitude when the SOC is zero. When the SOC of the lithium battery increases from small to large, the ultrasonic phase difference will first increase and then gradually decrease.

[0098] Ultrasonic waves are received at a set location and smoothed and denoised using MATLAB's built-in least squares smoothing filter function (sgolayfilt()). Fast Fourier transform is then used to quickly extract the signal spectrum and obtain the ultrasonic amplitude. The cross-correlation function is used to obtain the signal phase difference. The lithium battery voltage is collected synchronously during charging and discharging, and the surface temperature of the lithium battery is collected using a thermocouple sensor. Finally, the lithium battery voltage, surface temperature, ultrasonic amplitude, and phase difference are input into the WAA-BiTCN-BiLSTM-Multi-Head Attention network to predict the lithium battery's state of charge (SOC).

[0099] The specific steps include:

[0100] S1: Lithium battery ultrasonic signal and battery electrical parameter acquisition and data preprocessing;

[0101] An ultrasonic probe and a thermocouple sensor are adhered to the surface of the battery. The battery is connected to a battery detection device to collect the lithium battery voltage. The thermocouple sensor is connected to a temperature detection instrument to collect the surface temperature of the lithium battery. One end of the ultrasonic probe is connected to an ultrasonic signal generator, and the other end is connected to a signal receiver to collect ultrasonic signals. Finally, the data collected by each device is transmitted to a computer for data processing, and a least squares smoothing filter function is used for smoothing and noise reduction.

[0102] S2: Extracting ultrasound amplitude based on fast Fourier transform (FFT);

[0103] Due to the influence of the experimental environment, the collected ultrasonic signals are subject to noise interference. To ensure the accuracy of the data, the time domain signal needs to be converted to the frequency domain to extract the signal data.

[0104] S3: The features of battery voltage, surface temperature, ultrasonic amplitude, and phase difference are processed independently and then fused to construct a feature matrix;

[0105] In this embodiment, the feature signals are independently processed by multimodal feature fusion technology, and the features of the ultrasonic amplitude A and phase difference Φ are extracted using a convolutional neural network (CNN), and the following are obtained:

[0106] F A =CNN(A)

[0107] F Φ =CNN(Φ)

[0108] The features of voltage V and temperature T are extracted using a recurrent neural network (RNN) to obtain:

[0109] F V =RNN(V)

[0110] F T =RNN(T)

[0111] Introducing weight coefficients α, β, γ, δ, satisfying α + β + γ + δ = 1, and performing weighted summation on the features of different modes, we get:

[0112] F fused =αF A +βF Φ +γF V +δF T

[0113] Among them, F A 、F Φ 、F V and F T are the characteristic representations of ultrasonic amplitude, phase difference, voltage and temperature, respectively, and F fused is the fused feature matrix.

[0114] Assumption F fused is a t×d matrix, where t is the total number of time steps and d is the dimension of the feature. To meet the input mode of BiTCN, F fused Perform dimensionality reduction to obtain the feature matrix F:

[0115] F=F fused W

[0116] Among them, F is the result after dimensionality reduction, and W is a d×4 weight matrix.

[0117] S4: As Figure 2 As shown, the lithium battery SOC is estimated based on the WAA-BiTCN-BiLSTM-Multi-Head Attention network;

[0118] The input feature matrix F is used to extract features from the input data using a bidirectional temporal convolutional network (BiTCN). This extracts basic features related to the SOC of the lithium battery parameters from the input data. The BiLSTM network is used to extract long-term information related to the lithium battery SOC from these basic features. The Attention network is used to extract important features related to the lithium battery SOC from this long-term information. Finally, a fully connected layer is used to map multiple important features into a one-dimensional space and output the predicted value of the lithium battery SOC.

[0119] S41: In this embodiment, the parameters of the BiTCN-BiLSTM-Multi-Head Attention network are optimized based on WAA to improve the accuracy of the BiTCN-BiLSTM-Multi-Head Attention network in predicting the SOC of lithium batteries. Specifically, the following steps are performed:

[0120] First, we use the WAA weighted average optimization algorithm to optimize the values ​​of three important parameters in the BiTCN-BiLSTM-Multi-Head Attention network architecture to ensure optimal algorithm performance. Specifically, we find the parameter values ​​that guarantee optimal performance for the entire network architecture. Only then can we input data into the network.

[0121] The three key parameters are the number of BiLSTM units, the learning rate, and the regularization coefficient. The number of BiLSTM units determines the output dimension of each LSTM layer. A larger number of units allows the model to capture more complex temporal patterns, but this also increases computational resource consumption and makes overfitting more likely. A smaller number of units can easily lead to loss of key information, resulting in reduced estimation accuracy.

[0122] The learning rate determines the step size of each gradient update of the model parameters, which directly affects the convergence speed and stability of the model. A larger learning rate leads to faster convergence, but may miss the optimal solution. A smaller learning rate leads to more precise parameter updates and more stable convergence, but it is slower and more likely to fall into a local optimum.

[0123] The role of the regularization coefficient is to suppress overfitting by penalizing model weights and balance model complexity with training data fitting ability. The larger the regularization coefficient, the more it can avoid overfitting, but may cause underfitting. The smaller the regularization coefficient, the more likely it is to cause overfitting.

[0124] The specific steps of this parameter optimization phase are as follows:

[0125] (1) Initialization

[0126] Initialize the candidate solution matrix for the three parameters of BiLSTM: number of units, learning rate, and regularization coefficient. The positions of the initial solutions for the above three parameters can be randomly generated using the following formula:

[0127] x i =rand·(UB i -LB i )+LB i ,i=1,2,3

[0128] Where rand is a random number between [0,1]; i=1, 2, and 3 represent three parameters: the number of BiLSTM units, the learning rate, and the regularization coefficient; UB i Indicates the upper bound of the i-th parameter; LB i represents the lower bound of the i-th parameter;

[0129] (2) Weighted average position calculation

[0130] 1) Calculate the fitness value of the candidate solution of the above three parameters Fitness(x i );

[0131] 2) Sort candidate solutions by fitness value;

[0132] 3) Select the top N Candidate The candidate solutions are used to calculate the weighted average position:

[0133]

[0134] Among them, n represents the number of individuals participating in the calculation of weighted centers in the current iteration; nP represents the total population size set by the WAA weighted average optimization algorithm; it represents the current number of iterations; Max It Indicates the maximum number of iterations of the WAA weighted average optimization algorithm.

[0135] (3) Search phase

[0136] In the iterative rounds of finding the optimal solution for the parameters, the following formula is used to determine the search stage of the current population:

[0137]

[0138] If K1 ≥ 0.5, the algorithm enters the local development stage; if K1 < 0.5, the algorithm enters the global exploration stage; α is a constant parameter used to adjust the balance between randomness and deterministic behavior during the search process.

[0139] (4) Development stage

[0140] During the development phase, the population updates its position according to the following three strategies:

[0141] 1) The first strategy:

[0142]

[0143] 2) The second strategy:

[0144]

[0145] 3) The third strategy:

[0146]

[0147] in, represents the value of the i-th parameter of the k-th individual (i=1, 2, 3 represent the parameters: number of BiLSTM units, learning rate, regularization coefficient) at the t-th iteration; μ i (t) represents the weighted center of the i-th parameter, which is calculated by weighting the i-th parameter values ​​of the first n better individuals; x i,GlobalBest (t) represents the global optimal solution of the i-th parameter; represents the i-th parameter in the k-th individual's historical optimal solution; w 11 ,w 12 ,…all represent random weights.

[0148] The three strategies are randomly selected according to the following formula:

[0149] K2=rand(3)

[0150] Where rand(3) is a random number between [1, 2, 3]. If K2 = 1, the first strategy is adopted to update based on the global and personal optimal positions. If K2 = 2, the second strategy is adopted to update based on the personal optimal position. If K2 = 3, the third strategy is adopted to update based on the global optimal position.

[0151] (5) Exploration stage

[0152] During the exploration phase, the following two strategies are used to balance global search and the ability to escape local optima:

[0153] 1) The first strategy - Levy Flight exploration:

[0154]

[0155] Where i=1, 2, and 3 represent the parameters: number of BiLSTM units, learning rate, and regularization coefficient, respectively; S i Represents the Levy step size of the i-th parameter, which follows the Levy distribution.

[0156] In some cases, the global optimum determined by the algorithm may be in an area around the global optimum, which is too far away from the ideal value of the global optimum. In this case, WAA faces the risk of converging to a local optimum when following the first strategy. To overcome this limitation, the second movement strategy is adopted to adjust the search space:

[0157] 2) The second strategy - random search:

[0158]

[0159] Among them, ub i ,lb i They represent the upper and lower bounds of the i-th parameter respectively; rand represents a random number uniformly distributed in [0,1].

[0160] The two strategies are randomly selected according to the following formula:

[0161] K3=rand

[0162] Among them, rand is a random number between [0,1];

[0163] 1) If K3 < 0.5, the first strategy, Levy Flight exploration, is selected to expand the search range and update the current solution.

[0164] 2) If K3>0.5, the second strategy of random search is selected to narrow the search range and update the current solution.

[0165] S42: BiTCN extracts basic features reflecting SOC;

[0166] like Figure 3 As shown in the figure, BiTCN consists of a forward TCN network and a backward TCN network with the same structure. Traditional TCN only performs forward convolution calculations on the input sequence, which can only extract the positive characteristics of the four input parameters of lithium battery voltage, surface temperature, ultrasonic amplitude and phase difference with respect to SOC, while ignoring the reverse implicit information. Therefore, this embodiment adopts a bidirectional temporal convolution structure to capture the hidden characteristics of lithium battery input parameters with respect to SOC in the forward and backward directions, and better obtain the basic characteristics of lithium battery voltage, surface temperature, ultrasonic amplitude and phase difference with respect to SOC.

[0167] In this embodiment, the bidirectional temporal convolutional structure (BiTCN) is an architecture that combines a temporal convolutional network (TCN) with bidirectional information transmission. It processes forward and backward time series information respectively through two independent TCN branches. The forward TCN branch is a standard TCN structure that processes and outputs in time order, processing sequences in positive time order; the backward TCN branch reverses the input sequence and inputs it into another TCN structure. Specifically, the lithium battery input data x is input into the BiTCN network, first passes through the forward TCN network in positive time order, and outputs a forward output after causal convolution and dilated convolution. Then, the lithium battery input data is input into another TCN network (called backward TCN) in reverse time order to output a reverse output. The forward output and backward output are then fused into the final output.

[0168] In order to effectively capture the basic characteristics of lithium battery input parameters regarding SOC, causal convolution requires more layers or a larger receptive field. Therefore, the bidirectional temporal convolutional network (BiTCN) uses dilated convolution to achieve a larger receptive field with fewer layers while maintaining the feature map dimension. The input sequence of the four parameters of the lithium battery (voltage, surface temperature, ultrasonic amplitude and phase difference) can be represented in matrix form:

[0169]

[0170] Among them, the subscripts 1, 2, …, τ represent the current time;

[0171] The input sequence x enters the BiTCN network and is processed by the forward TCN network and the reverse TCN network respectively, obtaining the forward output and the reverse output. The forward output and the reverse output are spliced ​​together to obtain the final BiTCN network output.

[0172] A small convolutional neural network is introduced to dynamically adjust the dilation factor according to the complexity of the input data. The dilation factor d is calculated using the following formula s :

[0173] d s =sigmoid(W d ·h s +b d )

[0174] Among them, W d and b d is a learnable parameter, h s is the input x s After a convolutional layer, the feature representation is restricted by the sigmoid function to the range [0, 1], ensuring that the expansion factor is a positive number.

[0175] The input sequence x is processed by multiple TCN blocks in the forward TCN network. In each TCN block, the dilated causal convolution is first calculated, and the calculation formula is as follows:

[0176]

[0177] Among them, F(s) represents the convolution result of the dilated causal convolution; f(i) is the weight of the convolution kernel; Represents the backtracking of the lithium battery input parameter state at the historical moment; k is the convolution kernel size; sd s i indicates the past direction; d s is the expansion factor, which represents the interval of convolution kernel weights in the time dimension;

[0178] Next, the dilated causal convolution result F(s) is input into the activation function ReLU to obtain the activation value F of the convolution result. act (s), finally, the activation value is subjected to residual connection and Dropout operation to obtain the convolution result of the TCN convolution block, and the output results of all TCN blocks in the forward TCN network are superimposed to obtain the output of the forward TCN network, that is, the forward output F forward (s);

[0179] The input sequence x is also processed by multiple TCN blocks in the reverse TCN network. Since the forward TCN network structure is the same as the reverse TCN network structure, the processing steps for the input sequence are also the same. The output of the reverse TCN network is the reverse output F backward(s);

[0180] Finally, the positive output F of the positive characteristic of the lithium battery SOC is forward (s) and the reverse output F of the basic characteristics of the reverse SOC of the lithium battery backward (s) are spliced ​​together to obtain the final output of the BiTCN network. This output can fully characterize the basic characteristics of the lithium battery SOC in both forward and reverse directions.

[0181] S43: BiLSTM extracts long-term information reflecting SOC

[0182] like Figure 4 (a)- Figure 4 As shown in (b), after BiTCN completes the extraction of basic features related to SOC, it outputs an n-dimensional vector containing the basic characteristic information of the lithium battery SOC. This n-dimensional feature vector is used as data and input into the BiLSTM bidirectional long short-term memory network to extract the long-term information that can reflect the lithium battery SOC. The specific steps are as follows:

[0183] 1) Data input

[0184] The output vector of BiTCN (i.e., the basic features related to the SOC of the lithium battery) is divided into multiple time steps, each of which contains an n-dimensional feature vector composed of the basic feature information related to the SOC extracted by BiTCN from the lithium battery voltage, surface temperature, ultrasonic amplitude and phase difference, i.e.

[0185] 2) Forward LSTM

[0186] The sequence is input into the forward LSTM network from left to right, and the output h at each time step is t Depends on the current input x basic,t and the output h of the previous time step t-1 .

[0187] 3) Backward LSTM

[0188] The same sequence is fed into the backward LSTM network from right to left, and the output h′ at each time step is t Depends on the current input x basic,t and the output h′ of the next time step t+1 .

[0189] 4) Information merging

[0190] The outputs of the forward and backward LSTM at each time step are combined as A complete sequence representation is formed. This sequence integrates the long-term information related to SOC in the lithium battery input parameters and can fully characterize the coordinated reflection of each input parameter on SOC.

[0191] 5) Output layer

[0192] The combined information sequence will be output to the output layer to output long-term information related to the lithium battery SOC.

[0193] The BiLSTM network structure used in this example consists of a forward LSTM and a backward LSTM. The two LSTM networks have the same structure, and each LSTM network calculates the following variables at each time step: forget gate, input gate, candidate cell state, cell state update, and output gate.

[0194] The specific calculation process under a single time step is as follows:

[0195] Cell state variable C t What is stored is the basic features related to the lithium battery SOC extracted from BiTCN at the current time t and all previous times, and then passed through the forget gate:

[0196] f t =σ(W fx x basic,t +W fh h t-1 +b f )

[0197] Among them, f t is the output of the forget gate, σ is the Sigmoid activation function, x basic,t is the input vector of the LSTM network at the current time step t, i.e., the basic features related to the lithium battery SOC, h t-1 is the hidden state of the previous time step t-1 (i.e., the output of LSTM), W fx is the input x basic,t The weight matrix, W fh It is input h t-1 The weight matrix, b f It is the bias term of the forget gate, which can adjust the activation threshold;

[0198] Determine how much of the cell state of the previous time step needs to be retained, and then control the flow of new information into the cell state through the input gate:

[0199] i t =σ(W ix x basic,t +W ih h t-1 +b i )

[0200] Among them, i tRepresents the activation vector of the input gate at time step t; σ represents the Sigmoid function, whose output range is between 0 and 1, indicating the degree of opening of the input gate, W ix Represents the weight matrix of input data to the input gate, W ih Represents the previous hidden state h t-1 The weight matrix to the input gate, b i represents the bias vector of the input gate;

[0201] Then combine the candidate states:

[0202]

[0203] Among them, W cx Represents the weight matrix input to the candidate state, W ch Represents the weight matrix from hidden state to candidate state, b c Bias vector representing the candidate state;

[0204] Update the cell state:

[0205]

[0206] Among them, f t Represents the forget gate vector, which determines the previous cell state C t-1 The retention ratio, C t-1 Indicates the previous cell state, i t represents the activation vector of the input gate at time step t;

[0207] Finally, the output gate:

[0208] o t =σ(W ox x t +W oh h t-1 +b o )

[0209] Among them, W ox Represents the weight vector from input to output gate; x t represents the input vector; W oh Represents the weight matrix from hidden state to output gate; b o Represents the bias vector of the output gate;

[0210] Filter the characteristic information related to the lithium battery SOC in the cell state and generate the hidden state:

[0211] h t =o t ⊙tanh(C t )

[0212] Among them, h tis the hidden state at the current time step t, o t is the activation value of the output gate, C t is the cell state at the current time step, ⊙ represents element-wise multiplication;

[0213] Gating and compression mechanisms are introduced to control the flow of information and reduce the dimension of the hidden state, thereby reducing memory usage and reducing computational complexity. Define two gates: Update gate z t and reset gate r t :

[0214] z t =σ(W z ·[h t-1 ;x t ]+b z )

[0215] r t =σ(W r ·[h t-1 ;x t ]+b r )

[0216] Where: W z 、W e is the learnable weight matrix, b z 、b r is the bias term, σ is the sigmoid activation function, and the output range is between [0,1].

[0217] The hidden state h is compressed t Compress to a lower dimension. Define the hidden state after compression

[0218]

[0219] Where: W c is the learnable weight matrix, b c is the bias term, tanh is the hyperbolic tangent activation function, and the output range is between [-1,1].

[0220] Combine the gating mechanism and compression mechanism to update the hidden state h t :

[0221]

[0222] Where: 1-z t Indicates the proportion of the hidden state retained at the previous moment, z t Indicates the ratio of updating the hidden state at the current moment.

[0223] Due to the structural consistency of the forward LSTM network and the backward LSTM network, the calculation process of each time step is consistent, that is, it follows the above calculation steps. In order to distinguish the outputs of the two network structures, the hidden state output of the forward LSTM network is recorded as the forward hidden state output h forward,t , which represents the forward long-term information related to the lithium battery SOC; the hidden state output of the backward LSTM network is recorded as the reverse hidden state output h backward,t , which represents the backward long-term information related to the lithium battery SOC. Next, the forward hidden state output and the reverse hidden state output are spliced, that is, h sum,t =[h forward,t ;h backward,t ].

[0224] Cain state h sum,t The output of the BiLSTM network contains long-term information reflecting the SOC, such as the lithium battery voltage, surface temperature, ultrasonic amplitude, and phase difference, and serves as the input of the subsequent multi-head attention mechanism layer.

[0225] S44: Extraction of important features reflecting SOC and output of SOC prediction values ​​through multi-head attention mechanism;

[0226] After BiLTSM completes the extraction of long-term information related to the SOC of lithium batteries, it inputs the extracted long-term information into the multi-head attention mechanism. The multi-head attention mechanism assigns different weights to the long-term feature information related to the SOC of lithium batteries through training to obtain the weighted feature representation of attention. Specifically, the input information is mapped to multiple subspaces (i.e., multiple "heads") through linear projection. Each subspace calculates attention independently to capture the association of information under different feature representations, and then integrates the attention results of each subspace through splicing operations, thereby increasing the influence of important feature information on the prediction results, making the model more expressive and flexible. Suppose the hidden state of BiLSTM output is h = (h1,h2,…,h τ ), the weight distribution of the multi-head attention mechanism is expressed as follows:

[0227] For the i-th attention head (i=1,2,…,h, h is the number of heads)

[0228]

[0229] The outputs of h attention heads are fused to obtain the final attention-weighted feature representation:

[0230] A=Concat(a 1 ,a 2 ,…,a h )W o

[0231] in, is the attention probability distribution function of the i-th attention head, is the attention weight matrix corresponding to the i-th attention head, h t is the hidden state at the current time step t, h s is the hidden state at the s-th time step, is the attention score of the i-th attention head, a i is the attention-weighted feature representation of the i-th attention head, W o This is the weight matrix used to fuse the outputs of multiple heads. Through this multi-head parallel computing and fusion approach, a more detailed and comprehensive weight distribution of long-term information is achieved, providing a richer feature representation for the subsequent fully connected layer output of the battery SOC prediction value.

[0232] After completing the calculation of the weighted feature representation A of the attention, A is input into a fully connected layer, which will output a one-dimensional vector, which is the predicted value of the lithium battery SOC.

[0233] The present invention uses acoustic parameters combined with battery electrical parameters to collect multiple aspects of characteristic information. By organically combining these two types of parameters with different dimensions but closely related to the lithium battery status, it can dig out more key characteristic information hidden in the operation process of the lithium battery, and then achieve efficient estimation of the lithium battery state of charge (SOC) based on this rich and multifaceted information. This method of combining multiple parameters for estimation has significant advantages over traditional estimation methods that rely only on a single type of parameter. It can more accurately grasp the true SOC of the lithium battery, greatly helping to improve the estimation accuracy of the lithium battery SOC, thereby providing more reliable data support for the safe, stable, and efficient use of lithium batteries in many application scenarios.

[0234] The present invention realizes real-time and high-precision SOC estimation of lithium batteries based on a deep learning model. The lithium battery input data is first input into BiTCN to obtain basic features related to the lithium battery SOC. The basic features are input into BiLSTM to obtain long-term information related to the lithium battery SOC. The long-term information is input into Multi-Head Attention to obtain important features of the lithium battery. After passing through a fully connected layer, the important features can output a one-dimensional vector, namely the SOC prediction value. In this model, BiLSTM provides powerful memory capabilities, TCN provides better local time series modeling, and self-attention can help the model make more accurate predictions by capturing global information. The combination of the three makes the model's processing of time series data more comprehensive and accurate.

[0235] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A battery state of charge detection method based on deep learning and ultrasonic feature fusion, characterized in that: The steps include: Collect battery ultrasonic signals and battery electrical parameters and perform data preprocessing; Extracting the ultrasonic amplitude of the ultrasonic signal based on fast Fourier transform; The battery voltage, surface temperature, ultrasonic amplitude, and phase difference are extracted to obtain the features of different modes. The features of different modes are weighted and fused, and the feature matrix is ​​obtained after dimensionality reduction processing. Build a BiTCN-BiLSTM-Multi-Head Attention network and take the feature matrix as input; The parameter values ​​in the BiTCN-BiLSTM-Multi-Head Attention network are optimized based on the WAA weighted average optimization algorithm. The basic features related to the battery SOC in the feature matrix are extracted based on the bidirectional temporal convolutional network BiTCN. The long-term information related to the battery SOC in the basic features is extracted based on the BiLSTM network. The important features related to the battery SOC in the long-term information are extracted based on the Multi-Head Attention network. Multiple important features are mapped to a one-dimensional space through a fully connected layer, and the predicted value of the battery SOC is output.

2. The battery state of charge detection method based on deep learning and ultrasonic feature fusion according to claim 1 is characterized in that: Data preprocessing includes: using the least squares smoothing filter function to smooth and reduce noise.

3. The battery state of charge detection method based on deep learning and ultrasonic feature fusion according to claim 1 is characterized in that: The parameter values ​​in the BiTCN-BiLSTM-Multi-Head Attention network are optimized based on the WAA weighted average optimization algorithm, specifically including: The parameters in the BiTCN-BiLSTM-Multi-Head Attention network include the number of BiLSTM units, learning rate, and regularization coefficient; Initialize the candidate solution matrix of BiLSTM unit number, learning rate, and regularization coefficient; Calculate the fitness values ​​of the candidate solutions for the number of BiLSTM units, learning rate, and regularization coefficient, sort the candidate solutions according to the fitness values, and select the top N Candidate The weighted average position of candidate solutions is calculated, where the top N Candidate The candidate solutions are expressed as: Among them, n represents the number of individuals participating in the calculation of weighted centers in the current iteration, nP represents the total population size set by the WAA weighted average optimization algorithm, it represents the current number of iterations; Max It Indicates the maximum number of iterations of the WAA weighted average optimization algorithm; Perform location updates during the development phase; In the exploration phase, the search is performed based on Levy Flight exploration and random search strategies; Output the optimal values ​​of the parameters in the BiTCN-BiLSTM-Multi-Head Attention network.

4. The battery state of charge detection method based on deep learning and ultrasonic feature fusion according to claim 3 is characterized in that: During the development phase, location updates are performed. The location update strategies include: The position update is performed based on the global and personal optimal positions, which is expressed as: The position update is based on the personal optimal position, which is expressed as: The position update is performed based on the global optimal position, which is expressed as: in, represents the value of the i-th parameter of the k-th individual at the t-th iteration, i = 1, 2, 3, respectively representing the number of BiLSTM units, learning rate, and regularization coefficient, μ i (t) represents the weighted center of the i-th parameter, which is calculated by weighting the i-th parameter values ​​of the first n better individuals, x i,GlobalBest (t) represents the global optimal solution of the i-th parameter, represents the i-th parameter in the k-th individual's historical optimal solution, w 11 、w 12 、w 13 、w 21 、w 22 、w 31 、w 32 Represents the corresponding random weight.

5. The battery state of charge detection method based on deep learning and ultrasonic feature fusion according to claim 1 is characterized in that: Based on the bidirectional temporal convolutional network BiTCN, the basic features related to battery SOC in the feature matrix are extracted, including: The bidirectional temporal convolutional network BiTCN processes the forward and backward time series information through the forward TCN network and the backward TCN network respectively. The feature matrix is ​​input into the forward TCN network in the positive time sequence and outputs the forward output. The feature matrix is ​​input into the backward TCN network in the reverse time sequence and outputs the backward output. The forward output and reverse output are spliced ​​together to obtain the basic characteristics of the battery SOC in the forward and reverse directions.

6. The battery state of charge detection method based on deep learning and ultrasonic feature fusion according to claim 5 is characterized in that: The bidirectional temporal convolutional network BiTCN processes forward and backward timing information through the forward TCN network and the backward TCN network respectively, specifically including: Calculate the dilation factor d s : d s =sigmoid(W d ·h s +b d ) Among them, W d and b d is a learnable parameter, h s It is the feature representation of the feature matrix after a convolution layer, and sigmoid represents the sigmoid function; The feature matrix is ​​input into the forward TCN network in the positive order of time. After being processed by multiple TCN blocks, the dilated causal convolution calculation is performed in each TCN block, which is specifically expressed as: Among them, F(s) represents the convolution result of the dilated causal convolution, f(i) is the weight of the convolution kernel, Represents the backtracking of the battery input parameter state at the historical moment, k is the convolution kernel size; Input the convolution result F(s) of the dilated causal convolution into the activation function ReLU to obtain the activation value F of the convolution result act (s), after the activation value is subjected to residual connection and Dropout operation, the convolution result of the TCN convolution block is obtained, and the output results of all TCN blocks in the forward TCN network are superimposed to obtain the forward output F forward (s); The structure of the reverse TCN network is the same as the forward TCN network. The feature matrix is ​​input into the backward TCN network in reverse time order, and the reverse output F is output. backward (s); The forward output F forward (s) and reverse output F backward (s) The basic characteristics of the battery SOC in the forward and reverse directions are obtained by splicing.

7. The battery state of charge detection method based on deep learning and ultrasonic feature fusion according to claim 1 is characterized in that: The BiLSTM network is used to extract long-term information related to battery SOC from basic features, including: The basic features are divided into multiple time steps, each time step contains an n-dimensional feature vector. The BiLSTM network includes a forward LSTM network and a backward LSTM network; The basic features are input into the forward LSTM network from left to right, and the output h of each time step is t Depends on the current input x basic,t and the output h of the previous time step t-1 ; The basic features are input into the backward LSTM network from right to left, and the output h of each time step is ′ t Depends on the current input x basic,t and the output h of the next time step ′ t+1 ; The outputs of the forward LSTM network and the backward LSTM network at each time step are combined into an information sequence [h t ;h ′ t ], the combined information sequence is output to the output layer to obtain long-term information related to the battery SOC.

8. The battery state of charge detection method based on deep learning and ultrasonic feature fusion according to claim 7, characterized in that: In the forward LSTM network and the backward LSTM network, the calculation of the forget gate, input gate, candidate cell state, cell state update, and output gate is performed at each time step, specifically including: Cell state variable C t What is stored is the basic features related to the battery SOC extracted from the bidirectional temporal convolutional network BiTCN at the current time t and all previous times, and processed through the forget gate: f t =σ(W fx x basic,t +W fh h t-1 +b f ) Among them, f t is the output of the forget gate, σ is the Sigmoid activation function, x basic,t is the input vector of the LSTM network at the current time step t, i.e., the basic features related to the battery SOC, h t-1 is the hidden state of the previous time step t-1, W fx is the input x basic,t The weight matrix, W fh It is input h t-1 The weight matrix, b f is the bias term of the forget gate; Control the flow of new information into the cell state through the input gate: i t =σ(W ix x basic,t +W ih h t-1 +b i ) Among them, i t represents the activation vector of the input gate at time step t, σ represents the Sigmoid function, whose output range is between 0 and 1, indicating the degree of opening of the input gate, W ix Represents the weight matrix of input data to the input gate, W ih Represents the previous hidden state h t-1 The weight matrix to the input gate, b i represents the bias vector of the input gate; The candidate states are represented as: Among them, W cx Represents the weight matrix input to the candidate state, W ch Represents the weight matrix from hidden state to candidate state, b c Bias vector representing the candidate state; Update the cell state: Among them, f t Represents the forget gate vector, which determines the previous cell state C t-1 The retention ratio, C t-1 Indicates the previous cell state, i t represents the activation vector of the input gate at time step t; The activation value of the output gate is: o t =σ(W ox x t +W oh h t-1 +b o ) Among them, W ox Represents the weight vector from input to output gate, x t represents the input vector, W oh Represents the weight matrix from the hidden state to the output gate, b o Represents the bias vector of the output gate; Filter the characteristic information related to battery SOC in the cell state and generate the hidden state: h t =o t ⊙tanh(C t ) Among them, h t is the hidden state at the current time step t, o t is the activation value of the output gate, C t is the cell state at the current time step, ⊙ represents element-wise multiplication; Define update gate z t and reset gate r t : z t =σ(W z ·[h t-1 ;x t ]+b z ) r t =σ(W r ·[h t-1 ;x t ]+b r ) Among them, W z 、W r is the learnable weight matrix, b z 、b r is the bias term, σ is the sigmoid activation function; The hidden state h is compressed by the compression mechanism t After compression, the hidden state after compression is expressed as: in, represents the compressed hidden state, tanh is the hyperbolic tangent activation function, W c represents the learnable weight matrix, b c represents the bias term; Combine the gating mechanism and compression mechanism to update the hidden state h t : Where 1–z t Indicates the proportion of the hidden state at the previous moment, z t Indicates the proportion of updating the hidden state at the current moment; The forward hidden state output is concatenated with the reverse hidden state output to obtain long-term information related to the battery SOC.

9. The battery state of charge detection method based on deep learning and ultrasonic feature fusion according to claim 1, characterized in that: The Multi-Head Attention network is used to extract important features related to battery SOC from long-term information, including: Based on the multi-head attention mechanism, different weights are assigned to long-term information for training to obtain the weighted feature representation of attention. The weighted feature results of attention are input into the fully connected layer, and the fully connected layer outputs the predicted value of the battery SOC.

10. The battery state of charge detection method based on deep learning and ultrasonic feature fusion according to claim 9, characterized in that: Based on the multi-head attention mechanism, different weights are assigned to long-term information for training. The weight distribution result is expressed as: For the i-th attention head: The outputs of h attention heads are fused to obtain the final attention-weighted feature representation: A=Concat(a 1 ,a 2 ,…,a h )W o in, is the attention probability distribution function of the i-th attention head, is the attention weight matrix corresponding to the i-th attention head, h t is the hidden state at the current time step t, h s is the hidden state at the s-th time step, is the attention score of the ith attention head, a i is the attention-weighted feature representation of the i-th attention head, W o is the weight matrix used to fuse multiple head outputs.

Images