Missile fault detection method based on dynamic attention mechanism multi-modal information fusion

Abstract

The application discloses a kind of missile fault detection methods of multi-modal information fusion based on dynamic attention mechanism, mainly solve the problem of low detection rate in prior art, and missile can be damaged to a certain extent.The implementation scheme is as follows: collecting missile sound signal and measuring physical parameters;Obtain training sample set and test sample set;A visual feature model including residual neural network and a physical information feature model of multilayer perception are constructed;Dynamic attention flow between modes is constructed;Attention flow in dynamic mode is constructed;The state detection of missile is realized by the fusion of two kinds of attention flow.The application improves the accuracy and convenience of missile detection, and can be used for missile health management, fault prediction and the like.

Description

TECHNICAL FIELD

[0001] The application relates to a missile fault detection method based on dynamic attention mechanism and multi-modal information fusion, and belongs to the technical field of missile health management. BACKGROUND

[0002] In recent years, with the development of science and technology and the improvement of weapon level, weapon researchers pay more and more attention to missile health management. The running state of a missile is different from that of a general device, and an abnormal running state usually indicates the occurrence of a certain fault, such as device wear, device corrosion, device fracture and device aging. Therefore, missile health management is of great significance for early fault detection, maintenance and replacement of missiles.

[0003] At present, according to the difference of the used signal type, missile fault detection can be divided into two categories: an invasive physical quantity detection method and a monitoring method based on non-invasive monitoring signals. The invasive physical quantity detection method can obtain necessary related physical parameters, such as intuitive visual inspection. However, this method needs to be measured invasively, and the disassembly and other operations in the detection process cause a certain degree of damage to the missile itself. The non-invasive monitoring signal method only needs to attach sensors to the outside of the missile to collect vibration or sound signals. Although this method is low in economic cost, the vibration signal feature knowledge is insufficient and limited by the precision of the collection instrument, so that the recognition efficiency of the vibration signal-based fault detection method is low.

[0004] Therefore, it is of great significance for missile health management to find a fault detection technology that combines the advantages of the two detection methods and has small errors. SUMMARY

[0005] The application aims at the above-mentioned prior art and provides a missile fault detection method based on dynamic attention mechanism and multi-modal information fusion, which is used to solve the technical problem of difficult missile fault detection operation in the prior art.

[0006] To achieve the above-mentioned purpose, the technical solution adopted by the application comprises the following steps:

[0007] (1) collecting missile sound signals and measuring physical parameters;

[0008] n sound sensors are used to sample at a sampling frequency f s n missile state sound signals with a length of T are collected, and vibration sensors with the same sampling frequency as the sound sensors are used to collect the upper end vibration intensity v1 and the lower end vibration intensity v2 of the missile to be measured, n>=1, T>=4687500, f s >=1500Hz;

[0009] (2) Obtain a training sample set and a test sample set:

[0010] (2a) Take N signal segments from the original missile state sound signal in the order of acquisition, with a length of w and a step of s, to obtain n groups of signal segments, each containing N signal segments,

[0011] (2b) Perform short-time Fourier transform on the n sound signals to convert them into time-frequency graphs;

[0012] (2c) Take the time-frequency graph and the corresponding sample-label pair composed of the real missile state as a sample, and all samples form a sample set with a capacity of n x N. Randomly select 80% of the samples in the sample set as the training sample set, and the remaining 20% of the samples form the test sample set;

[0013] (3) Construct a visual and physical information feature extraction model;

[0014] Construct a visual feature model including a residual neural network and a physical information feature model including a multilayer perceptron, wherein the residual neural network includes a plurality of residual modules stacked with each other, for extracting regional visual features of the missile state The multilayer perceptron includes a fully connected network layer composed of a plurality of neurons stacked in sequence, for extracting physical information features where I is the input image after short-time Fourier transform, P is the physical information corresponding to I, θ ResNet represents the visual feature extraction model parameter, θ MLP represents the physical information feature extraction model parameter;

[0015] (4) Construct a dynamic attention flow between modalities:

[0016] (4a) Inspired by the Transformer model, use W Q , W V , W K matrix to convert the input visual feature and physical information feature into R Q , R V , P Q , P V and

[0017] R K = Linear(R; θ RK ), PK = Linear(R; θ PK ),

[0018] RQ = Linear(R; θ RQ ), P Q = Linear(P; θ PQ ),

[0019] R V = Linear(R; θ RV ), P V = Linear(P; θ PV ).

[0020] where Linear denotes a fully connected network layer with parameter θ, and dim denotes the common dimension of the two converted features;

[0021] (4b) The original attention weight for the physical feature information to aggregate into each visual feature is obtained by calculating the inner product R Q P K between each pair of visual features R Q and physical information main features P K T Then, the original weight is normalized by the square root of dim and a nonlinear function to obtain the information flow InterMAF R←P from physical information to visual features and the information flow InterMAF R→P from visual features to physical information, respectively;

[0022]

[0023]

[0024] where dim is the common dimension of the two converted features, R Q P K is the inner product between the visual feature Query vector R Q and the physical information main feature Key vector P K , and P Q R K is the inner product between the physical information main feature Query vector P Q and the visual feature Key vector R K .

[0025] (4c) The two bidirectional InterMAF matrices capture the importance between each visual feature and physical information, and then obtain the updated visual features R update and physical features P update ;

[0026] R update = InterMAF R←P × P V ,

[0027] P update = InterMAF R→P × R V .

[0028] where, P V and R V are the unweighted information flows, InterMAF R←P is the information flow from physical information to visual features, and InterMAF R→P is the information flow from visual features to physical information.

[0029] (4d) After obtaining the updated visual and physical features, they are concatenated with the original visual feature R and physical feature P, and a fully connected layer is used to transform the concatenated features to obtain the interaction output between different modalities.

[0030] R = Linear([R, R update ] T ; θ RT ),

[0031] P = Linear([P, P update ] T ; θ PT ).

[0032] (5) Constructing attention flow within dynamic modes:

[0033] (5a) To utilize information from other modes, the visual features are averaged along the index dimension of visual features and the physical information features are averaged along the index dimension of physical information. Then, the averaged set of features of both modes are transformed into a two-dimensional feature vector to match R Q , R K and P Q , P K . Next, the two-dimensional feature vector of each mode is controlled by a sigmoid nonlinear function σ(·) to produce a control adjustment to the other mode;

[0034] G R→P = σ(Linear(Avg_Pool(R) ; θ RP ),

[0035] G R←P = σ(Linear(Avg_Pool(P) ; θ PP ).

[0036] where R is the original visual feature, P is the original physical feature, θ RP and θ PP are model parameters, and GR←P and G R→P These are respectively the control of physical features by visual features and the control of visual features by physical features.

[0037] (5b) Both modal flows are affected by the conditional gating vector from the other modal flow when calculating the Q and V matrices;

[0038]

[0039]

[0040] Where ⊙ represents element-wise multiplication at corresponding positions, G R←P and G R→P These represent the control of physical features by visual features and the control of visual features by physical features, respectively. R Q For visual feature query vectors, R K P is the visual feature key vector. Q For the physical feature query vector, P K This is the physical feature key vector.

[0041] (5c) By calculating each pair of visual features Main characteristics of physical information Inner product between The raw attention weights used to summarize physical feature information into each visual feature are obtained. Then, the raw weights are normalized using the square root of "dim" and a nonlinear function to obtain the information flow DyIntralMAF within the dynamic pattern of physical information. R←R Information flow within visual feature dynamic patterns (DyIntralMAF) P←P ;

[0042]

[0043]

[0044] (5d) The dynamic information flow within the two modalities passes through the original features P in their respective modalities. V and R V Update using residuals;

[0045] R = Linear(R + R) update ;θ RD ),

[0046] P = Linear(P + P) update ;θ PD ).

[0047] in

[0048] R update = DyIntralMAF R←R x R V ,

[0049] P update = DyIntralMAF P←P x P V .

[0050] DyIntralMAF R←R is the information flow of the missile physical information in the dynamic mode of the missile, DyIntralMAF P←P is the information flow of the missile visual feature in the dynamic mode, R is the original visual feature of the missile, P is the original physical feature of the missile, R V is the Value vector of the original visual feature of the missile, P V is the Value vector of the original physical feature of the missile;

[0051] (6) The visual feature and the physical feature processed in the modal and the inter-modal are point multiplied, and the probability of different states of the missile is obtained through a softmax function to realize missile state detection.

[0052] Compared with the prior art, the present application has the following advantages:

[0053] 1、The missile fault detection method based on dynamic attention mechanism multi-modal information fusion of the present application includes a visual feature model of a residual neural network and a physical information feature model of a multi-layer perception machine, labels corresponding to a time-frequency graph of a missile sound signal and measuring missile physical parameters are used, regional visual features of the time-frequency graph of the missile sound signal are extracted by convolution on the time-frequency graph of the missile sound signal, and the extracted regional visual features are fused with the physical information features proposed by the multi-layer perception machine, so that multi-modal information fusion from the missile sound signal to the missile physical parameters is completed, which fundamentally provides guarantee for improving the precision of missile fault detection.

[0054] 2、The application establishes the dynamic attention flow between the missile sound signal and the missile physical parameter mode through the dynamic attention mechanism, learns to capture the importance between the missile sound signal characteristics and the missile physical parameter characteristics, and transmits the information flow between the two modes according to the learned importance weight and aggregated characteristics, so as to update each missile physical parameter characteristic and missile sound signal time-frequency graph area characteristic, ensure that the information flow process can identify the cross-modal relationship between the missile sound time-frequency graph visual area and the missile physical parameter, and avoid the problem of inaccurate missile detection based on a single information source. In addition, the dynamic attention flow in the respective mode of the missile sound signal and the missile physical parameter based on the dynamic attention mechanism is a supplement to the dynamic attention flow between the mode of the missile sound signal and the missile physical parameter, which further improves the accuracy of the missile fault detection. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a flowchart of the missile fault detection method based on the dynamic attention mechanism of the multi-modal information fusion of the application. DETAILED DESCRIPTION

[0056] In order to clearly understand the technical solutions of the application, the detailed structure will be described in the following description. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0057] The following will be combined with the drawings Figure 1 The application will be further described in detail.

[0058] Embodiment 1

[0059] The missile fault detection method based on the dynamic attention mechanism of the multi-modal information fusion of the application, with reference to the drawings Figure 1 , includes the following steps:

[0060] (1) Collecting missile sound signals and measuring physical parameters;

[0061] Using n sound sensors to sample at a sampling frequency f s Collecting n missile state sound signals with a length of T of the missile to be tested, and using a vibration sensor with the same sampling frequency as the sound sensor to collect the upper end vibration intensity v1 and the lower end vibration intensity v2 of the missile to be tested, n≥1, T≥4687500, f s ≥1500Hz;

[0062] (2) Obtaining training sample set and test sample set:

[0063] (2a) Take the original missile state sound signal with w as length and s as step, and arrange the N signal segments in sequence after cutting the signal segments, to obtain n groups of signal segments containing N signal segments,

[0064] (2b) Perform short-time Fourier transform on the n sound signals to convert them into time-frequency graphs;

[0065] (2c) Take the time-frequency graph and the corresponding sample-label pair composed of the real missile state as a sample, and all samples form a sample set with a capacity of n x N, randomly select 80% of the samples in the sample set as a training sample set, and the remaining 20% of the samples form a test sample set;

[0066] (3) Construct a visual and physical information feature extraction model;

[0067] The visual feature model includes a residual neural network, and the residual neural network includes a plurality of residual modules stacked with each other, for extracting regional visual features of the missile state The multi-layer perceptron includes a fully connected network layer composed of a plurality of neurons stacked in sequence, for extracting physical information features where I is the input image after short-time Fourier transform, P is the physical information corresponding to I, θ ResNet represents the visual feature extraction model parameter, θ MLP represents the physical information feature extraction model parameter;

[0068] (4) Construct a dynamic attention flow between modalities:

[0069] (4a) Inspired by the Transformer model, use W Q , W V , W K matrix respectively converts the input visual feature and physical information feature to R Q , R V , P Q , P V and

[0070] R K =Linear(R;θ RK ), P K =Linear(R;θ PK ),

[0071] R Q =Linear(R;θRQ ), P Q = Linear(P; 0 PQ ),

[0072] R V = Linear(R; 0 RV ), PV = Linear(P; 0 PV ).

[0073] where “Linear” denotes a fully connected network layer with parameter 0

[0074] (4b) Obtain the original attention weight for the physical feature information to each visual feature by calculating the inner product R Q P K between each pair of visual feature R Q and physical information main feature P K ; T Then, the original weight is normalized by the square root of “dim” and a nonlinear function to obtain the information flow InterMAF R←P from physical information to visual feature and InterMAF R→P from visual feature to physical information, respectively;

[0075]

[0076]

[0077] where dim is the common dimension of the two converted features, R Q P K is the inner product between the visual feature Query vector R Q and the physical information main feature Key vector P K , P Q R K is the inner product between the physical information main feature Query vector P Q and the visual feature Key vector R K .

[0078] (4c) Two bidirectional InterMAF matrices capture the importance between each visual feature and physical information, and then obtain the updated visual feature R update and physical feature P update ;

[0079] R update = InterMAF R←P x P V ,

[0080] Pupdate = InterMAF R→P × R V .

[0081] where, where, P V and R V are unweighted information flows, InterMAF R←P is the information flow from physical information to visual features, InterMAF R→P is the information flow from visual features to physical information.

[0082] (4d) After obtaining the updated visual and physical features, they are concatenated with the original visual feature R and physical feature P, and a fully connected layer is used to transform the concatenated features to obtain the interaction output between different modalities;

[0083] R = Linear ([R, R update ] T ; θ RT ),

[0084] P = Linear ([P, P update ] T ; θ PT ).

[0085] (5) Constructing attention flow within dynamic modes:

[0086] (5a) To utilize information from other modes, the visual features are averaged along the index dimension of visual features and the physical information features are averaged along the index dimension of physical information. Then, the averaged set of features of both modes are transformed into a two-dimensional feature vector to match R Q , P K and P Q , P K . Next, the two-dimensional feature vector of each mode is controlled by a sigmoid nonlinear function σ(·) to produce a control adjustment to the other mode;

[0087] G R→P = σ (Linear (Avg_Pool (R) ; θ RP ),

[0088] G R←P = σ (Linear (Avg_Pool (P) ; θ PP ).

[0089] where, R is the original visual feature P is the original physical feature, θ RP and θ PP are model parameters, G R←P and G R→PThese are respectively the control of physical features by visual features and the control of visual features by physical features.

[0090] (5b) Both modal flows are affected by the conditional gating vector from the other modal flow when calculating the Q and V matrices;

[0091]

[0092]

[0093] Where ⊙ represents element-wise multiplication at corresponding positions, G R←P and G R→P These represent the control of physical features by visual features and the control of visual features by physical features, respectively. R Q For visual feature query vectors, R K P is the visual feature key vector. Q For the physical feature query vector, P K This is the physical feature key vector.

[0094] (5c) By calculating each pair of visual features Main characteristics of physical information Inner product between The raw attention weights used to summarize physical feature information into each visual feature are obtained. Then, the raw weights are normalized using the square root of "dim" and a nonlinear function to obtain the information flow DyIntralMAF within the dynamic pattern of physical information. R←R Information flow within visual feature dynamic patterns (DyIntralMAF) P←P ;

[0095]

[0096]

[0097] (5d) The dynamic information flow within the two modalities passes through the original features P in their respective modalities. V and R V Update using residuals;

[0098] R = Linear(R + R) uPdate ;θ RD ),

[0099] P = Linear(P + P) update ;θ PD ).

[0100] in

[0101] R updata=DyIntralMAF R←R ×R V ,

[0102] P uPdate =DyIntralMAF P←P ×P V .

[0103] DyIntralMAF R←R is the information flow in the dynamic mode of the missile physical information, DyIntralMAF P←P is the information flow in the dynamic mode of the missile visual feature, R is the original visual feature of the missile, P is the original physical feature of the missile, R V is the Value vector of the original visual feature of the missile, P V is the Value vector of the original physical feature of the missile;

[0104] (6) The visual feature and the physical feature processed in the mode and between the modes are point multiplied, and the probability of different states of the missile is obtained through a softmax function to realize missile state detection.

[0105] Test Example 1

[0106] The technical effects of the present application are further described in combination with a simulation experiment;

[0107] 1. Simulation conditions and contents

[0108] The data used in the experiment includes 356 missile sound signals collected by a sound sensor and corresponding missile physical information collected by a vibration sensor. The simulation platform is an Inter Core i7-6900K CPU with a main frequency of 3.20 GHz, an Nvidia GTX1080Ti graphics card, 32.0 GB of memory, a Windows 10 operating system, a Pytorch deep learning platform of version 1.10, and a Python 3.8 version development platform.

[0109] The method is used to train a missile failure rate detection model based on dynamic attention mechanism and multi-modal information fusion for each missile sound signal, and to obtain the estimated state of each missile sound signal test set sample. The failure detection method of residual convolutional neural network ResNet-18 is used to obtain the estimated state of each missile signal test sample, and the missile state estimation accuracy of the two methods is calculated respectively.

[0110] The calculation formula of the missile state estimation accuracy is:

[0111]

[0112] wherein, J is the total number of samples of the missile signal test set; yj and y′ j ...

[0113] 2. Simulation Result Analysis

[0114] The table below shows the accuracy of missile state estimation of our method and the residual convolutional neural network ResNet-18 method (hereinafter referred to as ResNet-18) on the above 71 (356×20%) missile signal test sets.

[0115] As can be seen from Table 1, the missile state estimation accuracy of our method on the test set of 71 missile signals is 0.9356, while the missile state estimation accuracy of ResNet-18 on the test set of 71 missile signals is 0.8889. Obviously, our method has a higher estimation accuracy than ResNet-18.

[0116] The experimental results and analysis above demonstrate that, within the framework of supervised learning, the missile failure rate detection model based on dynamic attention mechanism multimodal information fusion can better fuse the two types of multimodal information—missile sound signals and missile physical parameters—by constructing dynamic attention flows between and within the modes of missile sound signals and missile physical parameters, respectively, and then calculating the missile state estimation results, thereby effectively improving the accuracy of missile state estimation.

[0117] Table 1 Comparison of missile fault detection results between our method and the ResNet-18 algorithm.

[0118]

Claims

1. A missile fault detection method based on dynamic attention mechanism and multi-modal information fusion, characterized in that, The method comprises the following steps: 1) Collecting missile sound signals and measuring physical parameters: n sound sensors are used to sample at a frequency f s n missile state sound signals of a missile to be tested with a length of T are collected, and a vibration sensor with the same sampling frequency as the sound sensor is used to collect the upper end vibration intensity v1 and the lower end vibration intensity v2 of the missile to be tested, n≥1, T≥4687500, f s ≥1500Hz; 2) Obtaining a training sample set and a test sample set: (2a) Cutting N signal segments from the original missile state sound signal in the order of collection with a length of w and a step of s, and then sequentially arranging the N signal segments to obtain n groups of signal segments each containing N signal segments; (2b) Converting the n sound signals into time-frequency diagrams through short-time Fourier transform; (2c) Taking the time-frequency diagram and the corresponding sample-label pair composed of the real missile state as a sample, and taking all samples to form a sample set, wherein the sample set has a capacity of n x N, and randomly selecting 80% of the samples in the sample set as a training sample set and the remaining 20% of the samples as a test sample set; 3) Constructing a visual and physical information feature extraction model; Constructing a visual feature model comprising a residual neural network and a physical information feature model comprising a multilayer perceptron; 4) Constructing a dynamic attention flow between modalities: (4a) Inspired by the Transformer model, W Q ,W K ,W V The matrix respectively converts the input visual feature R and physical information feature P into P Q , P V and (4b) Compute the inner product R Q between the physical information main feature P K and each visual feature R Q P K T to obtain the original attention weight for the physical feature information aggregation to each visual feature; then, the original weight is normalized by the square root of dim and a nonlinear function to obtain the information flow InterMAF R←P from the physical information to the visual feature and the information flow InterMAF R→P from the visual feature to the physical information, respectively; (4c) obtaining updated visual features R update and physical features P update ; (4d) obtaining updated visual features R update After obtaining the visual features R and the physical features P, they are spliced with the original visual features R and the physical features P, and a fully connected layer is used for converting the spliced features to obtain the interaction output between different modalities. 5) Constructing an attention flow within a dynamic mode; (5a) In order to utilize information from other modes, the visual features are averaged along the index dimension of the visual features, and the physical information features are averaged along the index dimension of the physical information; Then, the average set features of both modes are converted into a two-dimensional feature vector to match R Q , R K and P Q , P K； Next, the two-dimensional feature vector of each mode is controlled by a Sigmoid non-linear function σ(·) to generate control adjustments to the other mode; (5b) When calculating the Q and V matrices, the influence of the conditional gating vector of the other mode flow is applied; (5c) Obtain the updated visual feature Query vector for each pair by calculating the inner product between the updated visual feature Query vector and the updated physical information Key vector Obtain the original attention weight for the physical feature information aggregation to each visual feature; then, normalize the original weight by the square root of "dim" and a nonlinear function to obtain the information flow DyIntralMAF within the dynamic mode of physical information to physical information R←R and the information flow DyIntralMAF within the dynamic mode of visual feature to visual feature P←P ;​​ (5d) Dynamic information flow within two modalities through the original features P within each modality V and R V Update by residual 6) The visual features and the physical features processed within and between modalities are multiplied, and the probabilities of different states of the missile are obtained through a softmax function to realize missile state detection.

2. The method of claim 1, wherein the method of missile fault detection based on dynamic attention mechanism of multi-modal information fusion is characterized in that In step (2a), the n groups of signal segments each containing N signal segments are obtained as follows: Wherein, T is the length of the missile to be measured, w is the length, s is the step, and n is the number of missiles.

3. The method of claim 1, wherein the method is based on a dynamic attention mechanism. In step 3), the visual feature model comprising a residual neural network and the physical information feature model comprising a multilayer perceptron are constructed, and the composition and functions of each module are as follows: The residual neural network comprises a plurality of residual modules stacked with each other, and is used for extracting regional visual features R of the missile state; The multi-layer perception comprises full connection network layers composed of multiple neurons stacked in sequence, for extracting physical information features P; R = ResNet(I; θ ResNet ) P = MLP(P; 0 MLP ) where I is the input image after short-time Fourier transform, P is the physical information corresponding to I, θ ResNet denotes the visual feature extraction model parameter, θ MLP denotes the physical information feature extraction model parameter.

4. The method of claim 1, wherein the method is a method of detecting a missile fault based on multi-modal information fusion of a dynamic attention mechanism. the input visual feature R and the physical information feature P are respectively converted into P Q , P V and R K = Linear(R; θ RK ), P K = Linear(R; θ PK ), R Q = Linear(R; θ RQ ), P Q = Linear(P; θ PQ ), R V = Linear(R; θ RV ), P V = Linear(P; θ PV ). Wherein, Linear represents a fully connected network layer with parameters θ, and dim represents the common dimension of the two converted features; The information flow InterMAF of obtaining physical information to visual features in step (4b) R←P and visual features to physical information to information flow InterMAF R→P : where dim is the common dimension of the two conversion features, R Q P K is the inner product between the visual feature Query vector R Q and the physical information main feature Key vector P K . Q R K is the inner product between the physical information main feature Query vector P Q and the visual feature Key vector R K .

5. The method of claim 1, wherein the method is based on a dynamic attention mechanism. The updated visual feature R is obtained in step (4c) update and physical feature P update ; the formula is as follows: R update = InterMAF R←P x P V P update = InterMAF R→P x R V wherein, wherein, P V and R V are unweighted information flows, InterMAF R←P is the information flow from physical information to visual features, InterMAF R→P is the information flow from visual features to physical information.

6. The method of claim 1, wherein the method of missile fault detection based on dynamic attention mechanism of multi-modal information fusion is characterized in that In step (4d), the interaction output between different modalities is obtained as follows: R = Linear([R, R update ] T ; theta RT ), R = Linear([R, R update ] T ; θ PT ), where R update is the updated visual feature, P update is the updated physical feature, R is the original visual feature P is the original physical feature, and Θ RT and Θ PT are model parameters.

7. The method of claim 1, wherein the method further comprises: In step (5a), the two modes are controlled with each other, and the formula is as follows: In step (5b), the influence of the conditional gating vector of the other mode flow is applied, and the formula is as follows: G R→P = σ(Linear(Avg_Pool(R); θ RP )), G R←P = σ(Linear(Avg_Pool(P); θ PP )). where R is the original visual feature, P is the original physical feature, and Θ RP and Θ PP are model parameters, G R←P and G R→P are the control of visual feature on physical feature and the control of physical feature on visual feature, respectively.

8. The method of claim 1, wherein the method is a method of detecting a missile fault based on multi-modal information fusion of a dynamic attention mechanism. In step (5d), the update is performed through the residual, and the formula is as follows: wherein, ⊙ represents the multiplication of elements in the corresponding positions, G R←P and G R→P are the control of visual features to physical features and the control of physical features to visual features, respectively; R Q is the visual feature Query vector, R K is the visual feature Key vector, P Q is the physical feature Query vector, P K is the physical feature Key vector.

9. The method of claim 1, wherein the method further comprises: Information flow DyIntraIMAF within the dynamic mode of physical information to physical information in step (5c) R←R and visual feature to visual feature dynamic mode DyIntraIMAF P←P The formula is as follows: wherein, is an inner product between the updated visual feature Query vector and the updated visual feature main feature Key vector is an inner product between the updated physical feature Query vector and the updated physical feature main feature Key vector ​​ 10. The method of claim 1, wherein the method further comprises: Wherein R = Linear(R + R update ; Θ RD ), P = Linear (P + P update ; θ PD ) ​ R update = DyIntralMAF R←R × R V , P uPdate = DyIntralMAF P←P x P V DyIntralMAF R←R DyIntralMAF is the information flow within the dynamic model of the missile physical information P←P R is the missile raw visual features, P is the missile raw physical features, R V R is the Value vector of the missile raw visual features, P V P is the Value vector of the missile raw physical features.

Images