Intelligent migration prediction method for service life of aero-engine bearing based on multi-source data fusion

Abstract

A multi-source data fused aero-engine bearing life intelligent migration prediction method comprises the following steps: adopting a bidirectional GRU network to depth time sequence degradation characteristics of multi-source data, designing a dynamic attention mechanism of degradation perception to realize adaptive fusion of multi-modal characteristics, and dynamically calculating each modal weight through a modal attention layer; meanwhile, a cross-working-condition transfer learning framework is constructed based on the maximum mean difference (MMD), and knowledge transfer under different working conditions is realized through minimizing the multi-source fusion data difference; the method is excellent in key indexes such as prediction precision, noise robustness and cross-working-condition adaptability.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft engine bearing life prediction, and in particular to an aircraft engine bearing life intelligent migration prediction method based on multi-source data fusion. Background Art

[0002] Industrial equipment health status monitoring and remaining life prediction (RUL) technology is a key link in ensuring the reliable operation of modern intelligent manufacturing systems and important transportation vehicles. Among them, data-driven RUL prediction methods have shown significant advantages in the field of industrial equipment health management compared to traditional mechanism-based models. Its core value lies in establishing a prediction model by mining the degradation laws in equipment operation data, rather than relying on prior physical knowledge. Data-driven RUL prediction adopts an end-to-end architecture, and can be mapped to the remaining service life based on the original monitoring signal of the target sensor, eliminating the need for manual feature extraction and reliance on expert knowledge in traditional methods. Therefore, it has gradually become a mainstream method in the field of life prediction of key components of aircraft engines (such as bearings, blades, etc.) ([1] Yang Hongfu, Jia Xiaoliang, Ren Shouwei. A review of data-driven aircraft engine fault diagnosis and prediction methods [J]. Aviation Precision Manufacturing Technology, 2016, 52(05): 6-9.).

[0003] Aircraft engines are critical aircraft components, their safety directly impacting the lives of passengers. Monitoring aircraft engines requires not only accurate but also comprehensive and multi-faceted monitoring. Therefore, life prediction methods based on multi-source data fusion have become a research hotspot in aircraft engine health management. Compared to traditional prediction methods, the core advantage of multimodal RLU prediction lies in its ability to fuse heterogeneous sensor data from multiple sources, enabling a more comprehensive and robust assessment of equipment degradation status. Multimodal fusion effectively overcomes the perceptual limitations of a single data source, demonstrating significant advantages in degradation feature complementarity, noise immunity, and long-term prediction stability. This collaborative prediction mechanism not only improves the accuracy of RLU estimation but also enhances the prediction system's adaptability to practical challenges such as changing operating conditions and missing data. It also enables more accurate identification of complex degradation patterns in aircraft engine bearings, thereby comprehensively enhancing the reliability and applicability of life prediction. Its core approach is to construct a life cycle health status representation system for equipment with spatiotemporal evolutionary characteristics through deep feature coupling of multi-source monitoring data and collaborative modeling of degradation trends.

[0004] In the prediction of the remaining life of aero-engine bearings, the traditional multi-source data fusion method ([1] Wang Daichao. Research on Rotating Machinery Fault Diagnosis Method Based on Multi-source Data Fusion [D]. Shandong University, 2022. DOI: 10.27272 / d.cnki.gshdu.2022.000916) faces three key bottlenecks: First, the static feature fusion strategy cannot capture the dynamic degradation coupling relationship between multi-modal data, resulting in the dilution or masking of key degradation features; Second, the existing methods lack the ability to model the evolution of the sensitivity of each mode during the entire life cycle of the bearing, making it difficult to adaptively adjust the feature contribution weights of different degradation stages such as normal wear, minor damage, and severe failure; Third, the traditional fusion framework lacks an effective processing mechanism for data distribution offset under variable operating conditions, resulting in a significant decrease in the prediction performance of the model when operating parameters such as speed and load change. This rigid fusion paradigm seriously restricts the synergistic efficiency potential of multi-source data in life prediction. Therefore, developing an intelligent fusion method with dynamic feature interaction mechanism, degradation stage perception capability and cross-working condition adaptability has become the core breakthrough direction for improving the accuracy of remaining life prediction of aircraft engine bearings. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to propose an intelligent migration prediction method for aircraft engine bearing life based on multi-source data fusion, which can combine the GRU network to model the time series, use the attention mechanism to fuse multi-source data, and perform migration prediction through maximum mean difference. It performs superiorly in key indicators such as prediction accuracy, noise robustness and cross-working condition adaptability.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] An intelligent migration prediction method for aircraft engine bearing life based on multi-source data fusion is proposed. The method adopts the temporal degradation characteristics of deep multi-source data using a bidirectional GRU network, designs a degradation-aware dynamic attention mechanism to achieve adaptive fusion of multimodal features, and the modal attention layer dynamically calculates the weights of each modality. At the same time, a cross-operating condition transfer learning framework is constructed based on the maximum mean difference (MMD), which realizes knowledge transfer under different operating conditions by minimizing the differences in multi-source fusion data.

[0008] A multi-source data fusion intelligent migration prediction method for aircraft engine bearing life includes the following steps:

[0009] Step 1: Acquire multi-modal original vibration monitoring data of the test bearing over its entire life cycle under multiple working conditions;

[0010] Step 2, data preprocessing: Select the full life data of the first n bearings in the source domain working condition data, and process them all into samples of length L and label them with corresponding remaining life as the training set TR As part of the training set T, the first n full-life bearing data of the target domain working condition are also selected and all processed into samples of length L without any annotation. R Finally, the full life data of the last bearing under the target domain working condition is selected and all processed into samples of length L as the test set T e ;

[0011] Step 3: Fault migration prediction model construction: A fault migration prediction model based on GRU gated recurrent units is constructed. Multi-source data signals are passed through a corresponding number of time series processing sub-networks (GRUs). After feature extraction by the GRU network, the outputs are concatenated in the feature dimension and then fused using a multimodal fusion method based on an attention mechanism. Finally, the output is input into a regressor through two fully connected layers to perform regression prediction of the remaining useful life.

[0012] Step 4: Set the training set T R The labeled source domain data and unlabeled target domain data are input into the fault migration prediction model for training, and the root mean square error and maximum averaged difference loss are used as optimization targets. Then, the gradient descent algorithm is used to update the model parameters θ; the root mean square error is shown in formula (1), the maximum averaged difference is shown in formula (2), and the gradient descent algorithm is shown in formula (3);

[0013]

[0014] Where: N is the number of samples, x i is the true value of the i-th sample, is the predicted value of the i-th sample;

[0015]

[0016] Among them H k It represents the RKHS space of the feature kernel k; P and Q represent two probability distributions, usually P is the source domain distribution and Q is the target domain distribution; E P is the expected operation on the source domain distribution P, E Q Similarly; φ is the feature mapping function, which maps the input data to the reproducing kernel Hilbert space H k ;x s and x t Represents the sample data of the source domain and target domain respectively;|||| Hk is the norm (distance metric) in the RKHS space; MMD k (P, Q) is used to measure the difference between the source domain distribution P and the target domain distribution Q. The subscript k indicates the calculation based on the kernel function k, and MMD is used only when P = Q. k (P,Q)=0;

[0017]

[0018] In the formula, θ represents the parameters of the model (weights and biases), which is usually a vector; α: learning rate, which controls the step size of each parameter update; L rmse : Root mean square error loss, used for regression tasks;

[0019] Step 5: Repeat step 4, iterate and update the optimized fault migration prediction model until the maximum number of iterations is reached, the training is completed, and the final fault migration prediction model is obtained;

[0020] Step 6: Test set T e , that is, the labeled target domain samples, are input into the final fault migration prediction model to obtain the root mean square error loss and mean relative error loss on the target domain working condition.

[0021] In step 3, each temporal sub-network uses a bidirectional GRU, with 2 stacking layers, 32 hidden layer dimensions, and 15 feature vector dimensions corresponding to each time step.

[0022] The specific method of the GRU gated recurrent unit in step 3 is as follows: the GRU network effectively captures the global relationship between the sequence data by introducing a reset gate and an update gate. t Used to calculate the previous moment h t-1 How much state needs to be retained, that is, how much of the current candidate state The degree of influence is expressed as follows:

[0023] r t =σ(W r ·[h t-1 ,x t ]+b r ) (4)

[0024] where h t-1 is the state at the previous moment, x t is the input information at the current moment, b r is the weight bias, W r is the weight matrix of the reset gate, and σ is the Sigmoid activation function, which is used to convert the output value to between 0 and 1;

[0025] Update gate z t The role of calculating candidate states The influence of the hidden state h(t-1) on the current hidden state h(t) is calculated as follows:

[0026] z t =σ(W z ·[ht-1 ,x t ]+b z ) (5)

[0027] Where W z is the weight matrix of the update gate, b z is the weight bias of the update gate;

[0028]

[0029] Where ⊙ is the element-by-element multiplication, that is, the Hadamard product. If r t ≈0, ignore historical information and only use the current input x t ;

[0030]

[0031] z t Control the ratio of the new and old states: z t ≈1: Use the new state h t Replace the old state; z t ≈0: keep the old state h t-1 .

[0032] The multimodal fusion method based on the attention mechanism in step 3 is to assign different importance weights to different parts of the input sequence by using the attention mechanism, so that the model focuses on the information most relevant to the current output. The attention mechanism has the following parts:

[0033]

[0034] Where Q is the query matrix (Query); K is the key matrix (Key); V is the value matrix (Value); before calculating the front attention score, the query matrix Q, key matrix K, and value matrix V are linearly transformed as shown below;

[0035] Q=QW Q ,K=KW K ,V=VW V (9)

[0036] Where W Q 、W K 、W V is the learnable linear transformation matrix of the attention head, Q, K, V are the query, key, and value of the attention head after the linear transformation;

[0037] The attention score measures the correlation between the query matrix Q and the key matrix K and is normalized into a weight through Softmax. The attention score is then used to assign weights to the value matrix V. The final calculation result is as follows:

[0038]

[0039] where d k is the dimension of the key, It is a scaling factor, which prevents the gradient from disappearing due to the explosion of the dot product value. Softmax is a normalization function.

[0040] Compared with the prior art, the advantages of the present invention are as follows:

[0041] The present invention proposes an intelligent migration prediction method for aircraft engine bearing life based on multi-source data fusion. Compared with the traditional single sensor monitoring method, the present invention integrates multimodal data to construct a more comprehensive bearing health status degradation characterization system throughout its life. In addition, an improved GRU (Gated Recurrent Unit) network is used as the core timing modeling tool. Compared with traditional RNN and LSTM networks, it has a simpler network structure and lower computational complexity while maintaining long-term memory capabilities. At the same time, a feature fusion module based on the attention mechanism is incorporated, which can adaptively weight the importance of different data sources, significantly improving the completeness and reliability of feature expression. Compared with the fixed-weight fusion method, it has stronger robustness to abnormal working conditions and noise interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flowchart of the method of the present invention.

[0043] Figure 2 This is a structural diagram of the fault migration prediction model of the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below with reference to the embodiments and accompanying drawings.

[0045] Reference Figure 1 A multi-source data fusion intelligent migration prediction method for aircraft engine bearing life includes the following steps:

[0046] Step 1: Acquire multi-modal original vibration monitoring data of the test bearing over its entire life cycle under multiple working conditions;

[0047] Step 2, data preprocessing: Select the full life data of the first four bearings in the source domain working condition data, and process them all into samples of length L and label them with corresponding remaining life as the training set T R As part of the training set T, the first four full-life bearing data of the target domain working condition are also selected and all processed into samples of length L without any annotation. R Finally, the full life data of the last bearing under the target domain working condition is selected and all processed into samples of length L as the test set T e ;

[0048] Step 3: Fault migration prediction model construction: refer to Figure 2 , a fault migration prediction model based on GRU gated recurrent unit is constructed, so that multi-source data signals pass through the corresponding number of time series processing sub-networks GRU respectively. Each time series sub-network uses bidirectional GRU, and the number of stacking layers is 2, the hidden layer dimension is set to 32, and the feature vector dimension corresponding to each time step is set to 15; after feature extraction through the GRU network, the outputs of all time steps are spliced ​​on the feature dimension, and then weighted fusion is performed using a multimodal fusion method based on the attention mechanism. Then, the weighted feature vectors of all time steps are added together to obtain a comprehensive feature vector expressing all time steps of the entire sequence. The dimension of this comprehensive feature vector is twice the feature dimension of each time step output by the previous GRU; finally, the comprehensive feature vector is input into the fully connected layer FC1, and then the remaining useful life is regressed and predicted through the regressor;

[0049] In order to perform time series modeling on the full life data of bearings, GRU gated recurrent unit is used to extract features from multi-source data. The GRU network mainly captures the global relationship between the sequence data by introducing reset gate and update gate. The reset gate and update gate are obtained by linearly transforming the current input information and the hidden state at the previous moment. The reset gate r t Used to calculate the previous moment h t-1 How much state needs to be retained, that is, how much of the current candidate state The degree of influence can be expressed as follows:

[0050] r t =σ(W r ·[h t-1 ,x t ]+b r ) (4)

[0051] where h t-1 is the state at the previous moment, x t is the input information at the current moment, b r is the weight bias, W r is the weight matrix of the reset gate, and σ is the Sigmoid activation function, which is used to convert the output value to between 0 and 1;

[0052] Update gate z t The role of calculating candidate states The influence of the hidden state h(t-1) on the current hidden state h(t) is calculated as follows:

[0053] z t =σ(W z ·[h t-1 ,x t ]+b z) (5)

[0054] Where W z is the weight matrix of the update gate, b z is the weight bias of the update gate;

[0055] Get the reset gate r t After that, we can continue to calculate the hidden state h at the previous moment t-1 Candidate status Contribution, candidate status It is mainly used to calculate the current hidden state h t , candidate state The calculation process is as follows:

[0056]

[0057] Where ⊙ is the element-by-element multiplication, that is, the Hadamard product. If r t ≈0, then the candidate state needs to ignore historical information and only use the current input x t ;

[0058] Finally, update the gate z t Calculate the current hidden state h t , z t Determine the hidden state h at the previous moment t-1 and candidate status Hide the current state h t The contribution degree is calculated as follows:

[0059]

[0060] where z t Control the ratio of the old and new states: when z t ≈1: Use the new state h t Replace the old state; when z t ≈0: keep the old state h t-1 ;

[0061] The current hidden state h t That is, it is output at the current moment and participates in the calculation of the hidden state at the next moment;

[0062] In addition to time series modeling, features extracted from multi-source data must be fused. This is achieved by using a multimodal fusion method based on an attention mechanism. Specifically, the attention mechanism assigns different importance weights to different parts of the input sequence, allowing the model to focus on the information most relevant to the current output. The attention mechanism mainly consists of the following parts:

[0063]

[0064] Where Q is the query matrix (Query), which represents the target position that needs to be generated (such as the hidden state of the decoder); K is the key matrix (Key), which represents the input sequence (such as the hidden state of the encoder); V is the value matrix (Value), which is usually the same as the Key, but can also be independent (such as the encoder state after a linear transformation); before calculating the front attention score, the query matrix Q, key matrix K, and value matrix V are linearly transformed as shown below;

[0065] Q=QW Q ,K=KW K ,V=VW V (9)

[0066] Where W Q 、W K 、W V is the learnable linear transformation matrix of the attention head, Q, K, V are the query, key, and value of the attention head after the linear transformation;

[0067] The attention score measures the correlation between the query matrix Q and the key matrix K. In modal fusion, it is used to measure the importance of the eigenvectors corresponding to all time steps of the GRU output. It is normalized into weights through Softmax. The attention score is then used to assign weights to the value matrix V. The final calculation result is as follows:

[0068]

[0069] where d k is the dimension of the key, It is a scaling factor, which is used to prevent the gradient from disappearing due to the explosion of the dot product value. Softmax is a normalization function.

[0070] Step 4: In order to enable the network to achieve both the remaining useful life prediction and the cross-operating condition migration prediction, the training set T R The labeled source domain data and the unlabeled target domain data are input into the prediction model for training, and the root mean square error and the maximum mean difference loss are simultaneously used as optimization targets. Then the gradient descent algorithm is used to update the model parameters θ; the root mean square error is shown in formula (1), the maximum mean difference is shown in formula (2), and the gradient descent algorithm is shown in formula (3);

[0071]

[0072] Where: N is the number of samples, x i is the true value of the i-th sample, is the predicted value of the i-th sample;

[0073]

[0074] Among them H k It represents the RKHS space of the feature kernel k; P and Q represent two probability distributions, usually P is the source domain distribution and Q is the target domain distribution; E P is the expected operation on the source domain distribution P, E Q Similarly; φ is the feature mapping function, which maps the input data to the reproducing kernel Hilbert space H k ;x s and x t Represents the sample data of the source domain and target domain respectively;|||| Hk is the norm (distance metric) in the RKHS space; MMD k (P, Q) is used to measure the difference between the source domain distribution P and the target domain distribution Q. The subscript k indicates the calculation based on the kernel function k, and MMD is used only when P = Q. k (P,Q)=0;

[0075]

[0076] In the formula, θ represents the parameters of the model (weights and biases), which is usually a vector; α: learning rate, which controls the step size of each parameter update; L rmse : Root mean square error loss, used for regression tasks;

[0077] Step 5: Repeat step 4, iterate and update the optimized fault migration prediction model until the maximum number of iterations is reached, the training is completed, and the final fault migration prediction model is obtained;

[0078] Step 6: Test set T e , that is, the labeled target domain samples, are input into the final fault migration prediction model to obtain the root mean square error loss and mean relative error loss on the target domain working condition.

[0079] The following takes the remaining useful life prediction of the XJTU-SY rolling bearing accelerated life experiment data set as an example to verify the effectiveness of the present invention.

[0080] The XJTU-SY rolling bearing accelerated life test dataset contains the full lifecycle vibration signals of 15 rolling bearings under three operating conditions, as shown in Table 1. The damaged locations of each bearing are indicated. The experimental platform for this dataset consists of an AC motor, motor speed controller, rotating shaft, support bearings, hydraulic loading system, and test bearings. This allows for conducting accelerated life tests on various rolling and sliding bearings under different operating conditions, acquiring full lifecycle monitoring data for the test bearings.

[0081] Table 1.3 working conditions

[0082]

[0083] The Xi'an Jiaotong University pump cyclic failure experimental data set uses two types of modal data: radial horizontal vibration x and axial vibration y. In this data set, the full life information of 15 bearings is shown in Table 2 below:

[0084] Table 2.4 working conditions

[0085]

[0086] Table 2.4 working conditions (continued)

[0087]

[0088] Experimental parameter settings: XJTU-SY rolling bearing accelerated life experiment data set: batch size B s =1024, number of cycles N e Depending on the specific migration task settings, the learning rate L r =0.00005.

[0089] The following comparison demonstrates the effectiveness of the present invention on the XJTU-SY rolling bearing accelerated life test dataset. The multimodal data fusion migration method based on the attention mechanism is compared with a migration method that uses only single modal data x and y without data fusion. The task is set as follows: T12 indicates that the source domain is 1 and the target domain is 2, and the diagnosis is transferred from source domain 1 to target domain 2. The comparison results are shown in Table 3.

[0090] Table 3 shows that the proposed method outperforms single-modal methods without multimodal data fusion in all transfer tasks. For tasks T12, T13, and T23, the diagnostic accuracy of the multimodal data fusion method decreased by 7.14, 0.71, and 1.49, respectively, compared to predictions without multimodal fusion. The diagnostic accuracy of the multimodal data fusion method decreased by 3.22, 0.96, and 3.46, respectively, compared to the single-modal x method without fusion. The diagnostic accuracy of the multimodal data fusion method decreased by 5.12, 3.72, and 2.78, respectively, compared to the single-modal y method without fusion. The mean relative error decreased by 0.07, 0.15, and 0.03, respectively. This indicates that the intelligent transfer prediction method for aircraft engine bearing life based on multi-source data fusion can effectively capture the multimodal correlation characteristics of bearing faults and establish a more comprehensive system for characterizing bearing health degradation throughout its life. Furthermore, an improved GRU network was used as the core time series modeling tool, resulting in a simpler network structure and lower computational complexity. A feature fusion module based on an attention mechanism was also incorporated to adaptively weight the importance of different data sources, significantly improving the completeness and reliability of feature representation. Finally, transfer learning based on maximum mean difference was used to achieve efficient prediction of the remaining useful life of aircraft engine bearing components across multiple operating conditions.

[0091] Table 3. Comparison experimental results of multimodal fusion and single modality x,y without fusion

[0092]

Claims

1. A multi-source data fusion intelligent migration prediction method for aircraft engine bearing life, characterized by: By adopting the temporal degradation characteristics of deep multi-source data of the bidirectional GRU network, a degradation-aware dynamic attention mechanism is designed to realize the adaptive fusion of multimodal features. The modal attention layer dynamically calculates the weights of each modality. At the same time, a cross-working condition transfer learning framework is constructed based on the maximum mean difference (MMD), which realizes knowledge transfer under different working conditions by minimizing the differences in multi-source fusion data.

2. The method for intelligent migration prediction of aircraft engine bearing life based on multi-source data fusion according to claim 1 is characterized in that: The following steps are involved: Step 1: Acquire multi-modal original vibration monitoring data of the test bearing over its entire life cycle under multiple working conditions; Step 2, data preprocessing: Select the full life data of the first n bearings in the source domain working condition data, and process them all into samples of length L and label them with corresponding remaining life as the training set T R As part of the training set T, the first n full-life bearing data of the target domain working condition are also selected and all processed into samples of length L without any annotation. R Finally, the full life data of the last bearing under the target domain working condition is selected and all processed into samples of length L as the test set T e ; Step 3: Fault migration prediction model construction: A fault migration prediction model based on GRU gated recurrent units is constructed. Multi-source data signals are passed through a corresponding number of time series processing sub-networks (GRUs). After feature extraction by the GRU network, the outputs are concatenated in the feature dimension and then fused using a multimodal fusion method based on an attention mechanism. Finally, the output is input into a regressor through two fully connected layers to perform regression prediction of the remaining useful life. Step 4: Set the training set T R The labeled source domain data and unlabeled target domain data are input into the fault migration prediction model for training, and the root mean square error and maximum averaged difference loss are used as optimization targets. Then, the gradient descent algorithm is used to update the model parameters θ; the root mean square error is shown in formula (1), the maximum averaged difference is shown in formula (2), and the gradient descent algorithm is shown in formula (3); Where: N is the number of samples, x i is the true value of the i-th sample, is the predicted value of the i-th sample; Among them H k It represents the RKHS space of the feature kernel k; P and Q represent two probability distributions, P is the source domain distribution, and Q is the target domain distribution; E P is the expected operation on the target domain distribution P, E Q Similarly; φ is the feature mapping function, which maps the input data to the reproducing kernel Hilbert space H k ;x s and x t Represent the sample data of the source domain and target domain respectively;|||| Hk is the norm in the RKHS space, i.e., the distance metric; MMD k (P, Q) is used to measure the difference between the source domain distribution P and the target domain distribution Q. The subscript k indicates the calculation based on the kernel function k, and MMD is used only when P = Q. k (P,Q)=0; In the formula, θ represents the parameters of the model, namely weights and biases, which is a vector; α: learning rate, which controls the step size of each parameter update; L rmse : Root mean square error loss, used for regression tasks; Step 5: Repeat step 4, iterate and update the optimized fault migration prediction model until the maximum number of iterations is reached, the training is completed, and the final fault migration prediction model is obtained; Step 6: Test set T e , that is, the labeled target domain samples, are input into the final fault migration prediction model to obtain the root mean square error loss R and mean relative error loss M on the target domain working condition.

3. The method according to claim 2, wherein: In step 3, each temporal sub-network uses a bidirectional GRU, with 2 stacking layers, 32 hidden layer dimensions, and 15 feature vector dimensions corresponding to each time step.

4. The method according to claim 2, characterized in that The specific method of the GRU gated recurrent unit in step 3 is as follows: the GRU network effectively captures the global relationship between the sequence data by introducing a reset gate and an update gate. t Used to calculate the previous moment h t-1 How much state needs to be retained, that is, how much of the current candidate state The degree of influence is expressed as follows: r t =σ(W r ·[h t-1 ,x t ]+b r ) (4) where h t-1 is the state at the previous moment, x t is the input information at the current moment, b r is the weight bias, W r is the weight matrix of the reset gate, and σ is the Sigmoid activation function, which is used to convert the output value to between 0 and 1; Update gate z t The role of calculating candidate states The influence of the hidden state h(t-1) on the current hidden state h(t) is calculated as follows: z t =σ(W z ·[h t-1 ,x t ]+b z ) (5) Where W z is the weight matrix of the update gate, b z is the weight bias of the update gate; Where ⊙ is the element-by-element multiplication, that is, the Hadamard product. If r t ≈0, ignore historical information and only use the current input x t ; z t Control the ratio of the new and old states: z t ≈1: Use the new state h t Replace the old state; z t ≈0: keep the old state h t-1 .

5. The method according to claim 2, characterized in that The multimodal fusion method based on the attention mechanism in step 3 is to assign different importance weights to different parts of the input sequence by using the attention mechanism, so that the model focuses on the information most relevant to the current output. The attention mechanism has the following parts: Where Q is the query matrix (Query); K is the key matrix (Key); V is the value matrix (Value); before calculating the front attention score, the query matrix Q, key matrix K, and value matrix V are linearly transformed as shown below; Q=QW Q ,K=KW K ,V=VW V (9) Where W Q 、W K 、W V is the learnable linear transformation matrix of the attention head, Q, K, V are the query, key, and value of the attention head after the linear transformation; The attention score is a measure of the correlation between the query matrix Q and the key matrix K, which is normalized into a weight through Softmax; The attention scores are then used to assign weights to the value matrix V, and the final calculation results are as follows: where d k is the dimension of the key, It is a scaling factor, which prevents the gradient from disappearing due to the explosion of the dot product value. Softmax is a normalization function.

Images