Abnormal maneuvering action detection and recognition method based on multi-modal spatio-temporal feature fusion

Abstract

The application relates to the technical field of flight data intelligent processing, and provides an abnormal maneuvering action detection and recognition method based on multi-modal space-time feature fusion, which comprises the following steps: collecting flight parameter data of an airplane to form sequence data; adopting a BP neural network model to recognize each frame of the sequence data and identify whether each frame is an abnormal action point; connecting the identified abnormal action points to form a complete abnormal maneuvering action; performing stretching conversion preprocessing on the data of the abnormal maneuvering action; inputting the preprocessed data of the abnormal maneuvering action into an abnormal maneuvering action recognition model based on multi-modal space-time feature fusion; and identifying and detecting the type of the abnormal maneuvering action through the abnormal maneuvering action recognition model. The multi-modal abnormal maneuvering action type recognition method provided by the application considers a more extensive range of maneuvering action feature mining, better highlights strong correlation features of a category target, and improves the maneuvering action recognition rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of flight data intelligent processing, and particularly relates to an abnormal maneuver action detection and recognition method based on multi-modal spatio-temporal feature fusion. BACKGROUND

[0002] In the flight training process, the abnormal maneuver action usually refers to that the pilot cannot follow the standard control method or is affected by the airflow and does not reach the standard of the maneuver action training. The abnormal maneuver action analysis is the key to finding the pilot's control method deficiency and improving the quality of flight training, and therefore, the pilot pays special attention to the abnormal maneuver action in the training. The flight parameter data records the parameters in the whole flight process, but does not label the trained abnormal maneuver action. In the actual training review, the abnormal maneuver action extraction mainly relies on the pilot's memory to locate the time point and review with the aid of three-dimensional visualization software. This method has large errors, is time-consuming, and it is difficult to locate the abnormal action again for review and summary after the memory disappears. At present, the abnormal maneuver action recognition is usually extracted and recognized by the field experts based on some fixed rules, but the flight parameter data is large in quantity, high in dimension and high in redundancy, the action recognition efficiency is low and inaccurate, and a large amount of manual cost is wasted.

[0003] Moreover, it is worth noting that the flight parameter data is particularly long time series data, and the flight parameter data of one flight is about one hour, and there are about 20 frames of sampling data in each second, and the dimension of each frame can reach more than 100. The actual training maneuver action data of each flight is less, and most of the other data is redundant data. How to quickly and accurately extract the time segment corresponding to the abnormal maneuver action from a large amount of flight parameter data is a technical problem to be solved at present.

[0004] Moreover, the traditional scheme is also difficult to identify the type of abnormal maneuver action, and the main reasons for specific identification difficulties include: (1) some maneuver actions are similar and difficult to classify. For example, the difference between the large-speed roll and the four-point roll is that the four-point roll requires a 1-second pause after each 90-degree roll, and the other state data and the entry of the two are consistent. (2) Due to the difference in the pilot's control, the difference in the objective conditions of the weather, and other reasons, the flight parameter data of the same maneuver action has certain differences, that is, the sequence characteristics of the same maneuver action are different. (3) The abnormal maneuver action is usually non-standard maneuver action data, which is similar to part of the normal adjustment action data in flight (the adjustment action is made to quickly reach the entry condition of the course objective due to the demand of the training course). SUMMARY

[0005] The present application aims to solve at least one of the technical problems in the background art, and provides an abnormal maneuver action detection and recognition method based on multi-modal spatio-temporal feature fusion.

[0006] To achieve the above object, the application provides an abnormal maneuver detection and recognition method based on multi-modal spatio-temporal feature fusion, comprising:

[0007] Collecting flight parameter data of an airplane to form sequence data during operation;

[0008] Recognizing each frame of the sequence data by using a BP neural network model to determine whether each frame is an abnormal action point;

[0009] Connecting the recognized abnormal action points to form a complete abnormal maneuver;

[0010] Stretching and converting the data of the abnormal maneuver for pretreatment;

[0011] Inputting the pretreated data of the abnormal maneuver into an abnormal maneuver recognition model based on multi-modal spatio-temporal feature fusion to recognize and detect the type of the abnormal maneuver by the abnormal maneuver recognition model.

[0012] According to one aspect of the application, the BP neural network model is composed of an input layer, a hidden layer and an output layer, the BP neural network model adopts two hidden layers, the number of neurons of each hidden layer is 20, the data of the input layer is roll angle, pitch angle, normal acceleration, longitudinal acceleration, relative air pressure height, lifting speed, Mach number, total atmospheric temperature, longitudinal angle of the control column and force arm adjustment position, and the output of the output layer is whether an action point, wherein 0 represents a non-action point and 1 represents an action point.

[0013] According to one aspect of the application, the connecting of the recognized abnormal action points to form a complete abnormal maneuver comprises:

[0014] Connecting an abnormal maneuver segment: taking a certain abnormal action point as the center, searching forward and backward respectively, when searching forward, finding the last data point with a pitch angle less than 15 degrees and a roll angle less than 30 degrees as the starting point of the action segment, when searching backward, finding the first data point with a pitch angle less than 15 degrees, a roll angle less than 30 degrees and a maneuver_flag of 0 as the ending point of the action segment, and the flight parameter data between the starting point and the ending point as the abnormal maneuver segment;

[0015] Connecting the whole abnormal maneuver segment, calculating the interval time of each abnormal maneuver segment, and connecting the abnormal maneuver segments with an interval time less than 5 seconds again to form a complete abnormal maneuver.

[0016] According to one aspect of the application, the stretching and converting the data of the abnormal maneuver for pretreatment comprises:

[0017] The data of the abnormal maneuver action is stretched by using a multi-dimensional interpolation mode and a k-neighbor interpolation method, so that the data length is 1024 and the dimension is unchanged.

[0018] The data of the abnormal maneuver action after stretching is converted into a data dimension of 32*32*10, and there are 10 channels, each corresponding to a flight parameter feature, and the feature map size of each channel is 32*32.

[0019] According to an aspect of the present application, the multi-modal spatio-temporal feature fusion abnormal maneuver action recognition model mines abnormal maneuver action features from two modalities of time and space, and screens more important features in different modalities through a cross-modal attention mechanism, and designs a progressive modal reinforcement unit to enhance the modal feature information that the attention mechanism focuses on through the progressive modal reinforcement unit to obtain an enhanced feature vector.

[0020] According to an aspect of the present application, the multi-modal spatio-temporal feature fusion abnormal maneuver action recognition model mines abnormal maneuver action features from two modalities of time and space, and includes:

[0021] The spatial features of the abnormal maneuver action data are extracted by using a convolutional neural network.

[0022] The time features of the abnormal maneuver action data are extracted by using a bidirectional long short-term memory network.

[0023] According to an aspect of the present application, the abnormal maneuver action recognition model splices the obtained enhanced feature vector to obtain a vector used for maneuver action recognition, and then performs linear transformation by using two fully connected layers, and the Relu function is used as an activation function in the two fully connected layers.

[0024] The type of the final maneuver action recognition is obtained by using a softmax function.

[0025] To achieve the above-mentioned purpose, the present application also provides an abnormal maneuver action detection and recognition system based on multi-modal spatio-temporal feature fusion, which includes:

[0026] A flight parameter data acquisition module acquires flight parameter data of a running aircraft to form sequence data;

[0027] An abnormal action point recognition module identifies each frame of the sequence data by using a BP neural network model to identify whether each frame is an abnormal action point;

[0028] An abnormal maneuver action forming module connects the identified abnormal action points to form a complete abnormal maneuver action;

[0029] A data preprocessing module performs stretching and conversion preprocessing on the data of the abnormal maneuver action.

[0030] The abnormal maneuver action type recognition detection module inputs the preprocessed abnormal maneuver action data into the multi-modal spatiotemporal feature fusion abnormal maneuver action recognition model, and detects the type of the abnormal maneuver action through the abnormal maneuver action recognition model.

[0031] To achieve the above object, the application further provides an electronic device, comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the computer program is executed by the processor to implement the multi-modal spatiotemporal feature fusion based abnormal maneuver action detection and recognition method.

[0032] To achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the multi-modal spatiotemporal feature fusion based abnormal maneuver action detection and recognition method.

[0033] According to the scheme of the application, the application is based on the design idea of machine learning, and the detection accuracy is continuously improved through the iterative training method, and the simple neural network model is designed, so that the efficiency is higher in the model prediction application stage.

[0034] The application firstly considers mining abnormal maneuver action features from two modalities of time and space, and screens more important features in different modalities through the cross-modality attention mechanism, and designs a progressive modality reinforcement unit to further enhance the modality feature information concerned by the attention mechanism. Compared with other machine learning related maneuver action recognition methods, the multi-modal abnormal maneuver action type recognition method proposed by the application considers a wider range of maneuver action feature mining, better highlights the strong correlation features with the category target, and improves the maneuver action recognition rate. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 a flow chart of the multi-modal spatiotemporal feature fusion based abnormal maneuver action detection and recognition method according to an embodiment of the application is schematically shown;

[0036] Figure 2 a progressive modality reinforcement unit principle diagram for embodiment 1. DETAILED DESCRIPTION

[0037] The content of the application will now be discussed with reference to exemplary embodiments. It should be understood that the discussed embodiments are only for better understanding and thus implementing the content of the application by those of ordinary skill in the art, and are not intended to imply any limitation on the scope of the application.

[0038] As used herein, the term "includes" and its variants are to be read as open-ended terms that mean "including, but not limited to." The term "based on" is to be construed as "based at least in part on." The terms "one implementation" and "an implementation" are to be construed as "at least one implementation."

[0039] Figure 1 A flow chart schematically representing a method of abnormal maneuver action detection and recognition based on multi-modal spatio-temporal feature fusion according to an implementation of the present application. As shown in Figure 1 In this implementation, the method of abnormal maneuver action detection and recognition based on multi-modal spatio-temporal feature fusion includes:

[0040] Collecting flight parameter data of an aircraft to form sequential data during operation;

[0041] Using a BP neural network model to recognize each frame of the sequential data to determine whether each frame is an abnormal action point;

[0042] Connecting the recognized abnormal action points to form a complete abnormal maneuver action;

[0043] Stretching and converting the data of the abnormal maneuver action for preprocessing;

[0044] Inputting the preprocessed data of the abnormal maneuver action into a model of abnormal maneuver action recognition based on multi-modal spatio-temporal feature fusion to recognize and detect the type of the abnormal maneuver action through the model of abnormal maneuver action recognition.

[0045] Further, according to an implementation of the present application, the BP neural network model is composed of an input layer, a hidden layer, and an output layer. The BP neural network model uses two hidden layers, each of which has 20 neurons. The data of the input layer are roll angle, pitch angle, normal acceleration, longitudinal acceleration, relative air pressure height, lifting speed, Mach number, total atmospheric temperature, longitudinal angle of the control stick, and force arm adjustment position. The output of the output layer is whether an action point, in which 0 represents a non-action point and 1 represents an action point.

[0046] Further, according to an implementation of the present application, connecting the recognized abnormal action points to form a complete abnormal maneuver action includes:

[0047] Connecting abnormal maneuver action segment: taking an abnormal action point as the center, searching forward and backward respectively, when searching forward, finding the last data point with a pitch angle less than 15 degrees and a roll angle less than 30 degrees as the starting point of the action segment (this data point is regarded as the beginning of the abnormality), when searching backward, finding the first data point with a pitch angle less than 15 degrees, a roll angle less than 30 degrees and maneuver_flag (maneuver_flag is 0) as the ending point of the action segment (the model predicts each data point, and when the prediction is 0, it represents a non-maneuver point, and when the prediction is 1, it represents a maneuver point. When searching backward, the first 0 represents the ending point of the abnormal maneuver), and the flight parameter data between the starting point and the ending point as the abnormal maneuver action segment;

[0048] Connecting abnormal maneuver action segment, calculating the interval time of each abnormal maneuver action segment, and connecting the abnormal maneuver action segments with an interval time less than 5 seconds again to form a complete abnormal maneuver action.

[0049] Further, according to an embodiment of the present application, the data of the abnormal maneuver action is preprocessed by stretching and converting, including:

[0050] The data of the abnormal maneuver action is stretched by using a multi-dimensional interpolation method and a k-nearest neighbor interpolation method, so that the data length is 1024 and the dimension is unchanged;

[0051] The data of the stretched abnormal maneuver action is converted to a data dimension of 32x32x10, a total of 10 channels, each channel corresponding to a flight parameter feature, and the feature map size of each channel being 32x32.

[0052] Further, according to an embodiment of the present application, the multi-modal spatio-temporal feature fusion abnormal maneuver action recognition model mines abnormal maneuver action features from two modalities of time and space, and filters more important features in different modalities through a cross-modal attention mechanism, and designs a progressive modal reinforcement unit to enhance the modal feature information that the attention mechanism focuses on through the progressive modal reinforcement unit to obtain an enhanced feature vector.

[0053] Further, according to an embodiment of the present application, the multi-modal spatio-temporal feature fusion abnormal maneuver action recognition model mines abnormal maneuver action features from two modalities of time and space, including:

[0054] The spatial features of the data of the abnormal maneuver action are extracted by using a convolutional neural network;

[0055] The time features of the data of the abnormal maneuver action are extracted by using a bidirectional long short-term memory network.

[0056] Further, according to an embodiment of the present application, the abnormal maneuver action recognition model splices the obtained enhanced feature vectors to obtain a vector finally used for maneuver action recognition, and then adopts two full connection layers for linear transformation, and a Relu function is adopted as an activation function in the two full connection layers.

[0057] The final maneuver action recognition type is obtained through a softmax function.

[0058] According to the above scheme of the present application, in order to extract the flight parameter data segment of the abnormal maneuver action, based on the existing complete flight record and the corresponding action label, the labeled data points and the data points with large overload are taken as abnormal action points, the BP neural network is used for training to obtain the data of all abnormal action points in the entire flight record, and the flight parameter data action point merging algorithm is designed according to the actual maneuver action characteristics, the divergent action points are connected into an action segment, and are taken as a next step abnormal maneuver action type recognition sample.

[0059] The abnormal maneuver action is the attitude and trajectory of the aircraft in space exhibited by the pilot in continuous time, in order to better recognize the abnormal maneuver action type, according to the performance characteristics of the abnormal maneuver action, the present application provides an abnormal maneuver action recognition method based on multi-modal spatio-temporal feature fusion, which simultaneously mines features in time and space two modes, and simultaneously introduces a cross-modal attention mechanism and a progressive modal reinforcement unit for better enhancement of features.

[0060] According to the above scheme of the present application, based on the design idea of machine learning, the detection accuracy is continuously improved through the method of iterative training, and through the designed simple neural network model, the efficiency is higher in the model prediction application stage.

[0061] The present application firstly considers mining abnormal maneuver action features from time and space two modes, and screens more important features in different modes through a cross-modal attention mechanism, and simultaneously designs a progressive modal reinforcement unit to further enhance the modal feature information concerned by the attention mechanism. Compared with other machine learning related maneuver action recognition methods, the multi-modal abnormal maneuver action type recognition method provided by the present application considers a more extensive range of maneuver action feature mining, better highlights the strong correlation features with the category target, and improves the maneuver action recognition rate.

[0062] Further, in order to achieve the above object, the present application also provides an abnormal maneuver action detection and recognition system based on multi-modal spatio-temporal feature fusion, comprising:

[0063] A flight parameter data acquisition module acquires flight parameter data of the aircraft to form sequence data;

[0064] An abnormal action point recognition module adopts a BP neural network model to recognize each frame of the sequence data to recognize whether each frame is an abnormal action point;

[0065] An abnormal maneuver action forming module connects the recognized abnormal action points to form a complete abnormal maneuver action;

[0066] A data preprocessing module performs stretching conversion preprocessing on the data of the abnormal maneuver action;

[0067] An abnormal maneuver action type recognition and detection module inputs the preprocessed data of the abnormal maneuver action into a multi-modal spatiotemporal feature fusion abnormal maneuver action recognition model, and detects the type of the abnormal maneuver action through the abnormal maneuver action recognition model.

[0068] The above-mentioned aircraft abnormal maneuver action detection system according to the present application can realize the above-mentioned multi-modal spatiotemporal feature fusion based abnormal maneuver action detection and recognition method, and the specific process steps are as described above, and will not be repeated here.

[0069] Further, in order to achieve the above-mentioned purpose, the present application further provides an electronic device, which comprises a processor, a memory, and a computer program stored on the memory and executable on the processor, and the computer program realizes the above-mentioned multi-modal spatiotemporal feature fusion based abnormal maneuver action detection and recognition method when executed by the processor.

[0070] Further, in order to achieve the above-mentioned purpose, the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program realizes the above-mentioned multi-modal spatiotemporal feature fusion based abnormal maneuver action detection and recognition method when executed by a processor.

[0071] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments, and it should be understood that the specific embodiments described herein are only one of the best embodiments of the present application, which are used to explain the present application and do not limit the protection scope of the present application, and all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0072] Embodiment 1

[0073] S1. A multi-modal spatiotemporal feature fusion based abnormal maneuver action detection and recognition method, comprising the following steps:

[0074] A BP neural network model is adopted to recognize each frame of the sequence data to recognize whether each frame is an abnormal action point:

[0075] To avoid the waste of manpower and inaccuracy of artificial rule, the neural network algorithm is used to identify the maneuver. Considering the realizability and the efficiency of the algorithm, the simple BP neural network model is used to identify whether each frame of the sequence data is a maneuver point. The BP neural network model is generally composed of an input layer, a hidden layer and an output layer, and its complexity is determined by the input layer node parameters, the output layer node parameters, the number of nodes and the number of layers of the hidden layer and other parameters. The BP neural network model of the embodiment adopts two hidden layers, each hidden layer has 20 neurons, and the input layer data is 10-dimensional data (the input layer data is determined according to expert experience and actual test) including roll angle, pitch angle, normal acceleration, longitudinal acceleration, relative air pressure height, lifting speed, Mach number, total atmospheric temperature, longitudinal angle of the control stick and force arm adjustment position, and the output layer is whether a maneuver point (0 represents a non-manipulation point, and 1 represents a maneuver point).

[0076] S2. Connecting the identified abnormal maneuver points to form a complete abnormal maneuver:

[0077] After obtaining the abnormal maneuver points in the cruise phase of the complete flight record, due to the discontinuity of the abnormal maneuver points, it is necessary to connect these abnormal maneuver points to form a complete maneuver. The overall idea of the abnormal maneuver point connection algorithm is as follows: first, connect the abnormal maneuver segments. Take a certain abnormal maneuver point as the center, search forward and backward respectively. When searching forward, find the last data point with a pitch angle less than 15 degrees and a roll angle less than 30 degrees as the starting point of the maneuver segment, and when searching backward, find the first data point with a pitch angle less than 15 degrees, a roll angle less than 30 degrees and a maneuver_flag of 0 as the ending point of the maneuver segment, and the flight parameter data between the starting point and the ending point as the maneuver segment. Second, connect the complete maneuver. Calculate the interval time of each maneuver segment, and connect the maneuver segments with an interval time less than 5 seconds into a complete abnormal maneuver.

[0078] S3. Stretching and converting preprocessing of the data of the abnormal maneuver:

[0079] To better use the model for calculation, the abnormal maneuver obtained by the above abnormal maneuver connection algorithm is stretched so that the data length is 1024 and the dimension is unchanged (10-dimensional features). The stretching adopts a multi-dimensional interpolation method, and the interpolation method adopts a k-nearest neighbor interpolation method. In terms of spatial model, the flight parameter data is converted into a data dimension of 32x32x10, that is, there are 10 channels, each channel corresponds to a flight parameter feature, and each channel is a 32x32 map.

[0080] S4. inputting the preprocessed abnormal maneuver data into a multi-modal spatio-temporal feature fusion abnormal maneuver recognition model, and recognizing the type of abnormal maneuver through the abnormal maneuver recognition model:

[0081] The abnormal maneuver recognition model based on multi-modal spatio-temporal feature fusion solves the long-term dependence relationship of flight sequence data by using a bidirectional long short-term memory network (BiLSTM), effectively captures the time correlation in the sequence data, and learns the relationship between the flight state features at a certain time and the flight state features before and after the time. At the same time, the flight time sequence data is converted into image feature data, and a convolutional neural network (CNN) is used to extract spatial local features, effectively capturing the spatial dependence between different positions. Then, the common and single-modal enhanced features in time and space are formed by progressive modal reinforcement. Specifically, a common message pool is introduced, which sends general messages to each mode, and enhances the characteristics of each mode through cross-modal attention mechanism. In turn, the message pool also collects enhanced features from each mode and makes them produce enhanced common messages. Through the repeated cycle process, the common information and modal features can be gradually enhanced, and finally the common features and modal features are used for the final action classification.

[0082] Specifically, the S4 step includes:

[0083] (1) Spatial modal feature extraction:

[0084] CNN is a method of producing more abstract representations by convolution operation on input, and then extracting local features of input. CNN has a prominent performance in image processing and text classification. When using CNN to process time series data, it can also extract local features of data in time series.

[0085] In order to extract the spatial features of flight sequence data, three layers of convolution are designed in this embodiment, using 3x3 convolution kernel, applying small receptive field to each convolution layer to capture detailed information and reduce the number of parameters. In particular, the number of filters in the first convolution layer is set to 32, the second convolution layer is doubled (64) to increase the feature learning ability, and the third layer of convolution is 128. Each layer of convolution reduces the number of parameters in the network through max pooling to avoid overfitting phenomenon and improve the calculation speed and running efficiency of the model. The specific formula is as follows:

[0086] ;

[0087] wherein, is the output of the next layer, is the input feature map, d m is the dimension of the image, W and H are the width and height of the image respectively, k mis the size of the convolution kernel, Maxpool is the max pooling layer, and Conv is the convolution function. To subsequently fuse the spatial and temporal dimensions, the image dimension is converted to the same dimension as the temporal feature by a fully connected layer (FFN). The specific formula is as follows:

[0088] ;

[0089] (2) Temporal modality feature extraction:

[0090] A bidirectional long short-term memory network (BiLSTM) is composed of a forward LSTM and a backward LSTM. To construct context information, the BiLSTM is used to extract features on the time sequence. The specific formula is as follows:

[0091] ;

[0092] wherein, is the network parameter in the BiLSTM model, is the sequence length, represents the single modality feature dimension, which is the same as the spatial feature dimension.

[0093] (3) Multi-modal information fusion:

[0094] After extraction by the spatial modality model (CNN) and the temporal modality model (BiLSTM), single modality features are obtained, denoted by z. The two types of modality features are fused, denoted by z, wherein:

[0095] ;

[0096] wherein, is the fusion mechanism, which is completed by the cross-modal attention mechanism and the progressive modality reinforcement mechanism described below.

[0097] (4) Cross-modal attention mechanism:

[0098] The final abnormal maneuver action recognition can take into account both spatial feature information and temporal feature information. However, directly splicing single modality feature information is not an optimal solution, and the asynchrony between modalities cannot make full use of the maneuver action information contained in different modalities when recognizing maneuver actions. To further enable single modality feature information to contain higher-order information about maneuver action recognition, the cross-modal attention mechanism described below is used to further fuse single modality feature information.

[0099] Cross-modal attention strengthens the information of the target modality by learning the directional attention between the target and source modalities. ​

[0100] For example, define a data sequence representing the source modality, a data sequence representing the target modality, where Similar to the attention mechanism, the cross-modal attention unit includes query, keys and values, which are defined as , , The cross-modal attention of a head can be represented as:

[0101] ;

[0102] where, is the attention weight of the data sequence of the source modality and the data sequence of the target modality, represents the concatenation of the data sequence of the source modality and the data sequence of the target modality in the same dimension.

[0103] To further enhance the representation of the importance of the source modality features by the cross-modal attention mechanism, the embodiment uses multi-head cross-modal attention, which focuses on the cross-modal interaction between the source modality features through the model, and the target modality is strengthened, , and finally the output of the strengthened feature is , that is, the product of the multi-head attention weight and .

[0104] ,

[0105] (5) Progressive modality enhancement unit:

[0106] In order to enhance the modality feature information, so that the single modality feature contains high-order information about the maneuvering action, a message pool is introduced to exchange information with each modality. As shown in Figure 2 , the message pool can send public messages to each modality in order to enhance its features through cross-modal attention. In turn, it also uses these features to update the information in the message pool to obtain the latest fusion information. Therefore, in this way, the public information and the feature information of the modality are gradually complementary. In addition, a dynamic filtering mechanism is introduced in the progressive modality enhancement unit to dynamically determine the passing ratio of the enhanced features. The message pool promotes cross-modal information flow and encourages cross-modal attention mechanisms to explore element-level dependency relationships between spatial features and temporal features.

[0107] As shown in Figure 2 , in the progressive modality enhancement unit, the public information and the spatial modality feature , the temporal modality feature The internal correlation of the modal elements is gradually complementary. Specifically, each layer includes two modal enhancement units and a message update module for updating information in the message pool. represents a message update module, represents a modal enhancement unit of a corresponding modal.

[0108] The modal enhancement unit takes and as input and outputs enhanced feature information , where is the feature output enhanced by the cross-modal attention mechanism, is the feature splicing of the spatial modal feature and the temporal single-modal feature , and i is the time. The specific representation is as follows:

[0109] ;

[0110] Specifically, the modal enhancement unit includes two enhancement branches, the first one is a self-attention branch, and the second one is a cross-modal attention branch. The specific formula is as follows:

[0111] ;

[0112] wherein and LN represent multi-head attention operation and layer normalization operation, and the enhanced feature and is processed by the following dynamic filtering mechanism:

[0113] ;

[0114] The passing ratio of each branch can be dynamically determined by the learnable parameters and , and this operation can filter information generated by incorrect cross-modal interaction.

[0115] The enhanced feature obtained by the message update module will be used to enhance the common information in the previous modal enhancement unit layer, and the message update module takes and as input and outputs enhanced common information :

[0116] ;

[0117] Specifically, the message update module includes two modal enhancement units, and each module enhances the common message by a modal enhancement. denote the corresponding modal enhancement units, wherein , in , by focusing on the elements of the modal are enhanced:

[0118] ;

[0119] Through the self-attention operation on , the interaction between all related modalities can be obtained, so that can be integrated into through an attention layer, and then the following operations are performed on it:

[0120] ;

[0121] The attention layer can dynamically control the information passed in , and generate information-rich public information. Finally, is passed through a feedforward layer with a skip connection to obtain the output .

[0122] (6) Abnormal maneuver action classification function

[0123] The obtained enhanced feature vector is spliced to obtain the final vector used for maneuver action recognition, and finally two fully connected layers are used for linear transformation, and the ReLU function is used as the activation function in the two fully connected layers. The calculation process is as follows:

[0124] ;

[0125] Finally, the final maneuver action recognition class is obtained through the softmax function, and the model is optimized using the mean square loss function. The calculation process is as follows:

[0126] ;

[0127] Where, n represents the number of batch samples, i represents the sample serial number, represents the real label, represents the predicted label.

[0128] Those skilled in the art will recognize that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and equipment can be referred to the corresponding process in the foregoing method implementation, and will not be repeated here.

[0130] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0131] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs.

[0132] In addition, the functional modules in the embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0133] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods for transmitting / receiving implemented by the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various media that can store program codes.

[0134] The above description is merely preferred embodiments of the present application and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

[0135] It should be understood that the size of the serial number of each step in the summary and embodiments of the present application does not absolutely mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

Claims

1. An abnormal maneuver detection and recognition method based on multi-modal spatio-temporal feature fusion, characterized in that, The method comprises the following steps: Collecting flight parameter data of an aircraft, which forms a sequence of data during operation of the aircraft; Using a BP neural network model to identify each frame of the sequence data to determine whether each frame is an abnormal action point; Connecting the identified abnormal action points to form a complete abnormal maneuver action; Stretching and converting the data of the abnormal maneuver action for pretreatment; Inputting the pretreated data of the abnormal maneuver action into a multi-modal spatio-temporal feature fusion abnormal maneuver action recognition model to identify and detect the type of the abnormal maneuver action through the abnormal maneuver action recognition model; The multi-modal spatio-temporal feature fusion abnormal maneuver action recognition model extracts abnormal maneuver action features from two modalities of time and space, and filters more important features in different modalities through a cross-modal attention mechanism. Meanwhile, a progressive modal reinforcement unit is designed to enhance the modal feature information focused by the attention mechanism, and an enhanced feature vector is obtained; The multi-modal spatio-temporal feature fusion abnormal maneuver action recognition model extracts abnormal maneuver action features from two modalities of time and space, which comprises: Using a convolutional neural network to extract spatial features of the data of the abnormal maneuver action; Using a bidirectional long short-term memory network to extract time features of the data of the abnormal maneuver action; The abnormal maneuver action recognition model concatenates the obtained enhanced feature vector to obtain a vector used for maneuver action recognition, and then performs linear transformation through two fully connected layers, wherein a Relu function is used as an activation function in the two fully connected layers; The final maneuver action recognition type is obtained through a softmax function.

2. The abnormal maneuver detection and recognition method based on multi-modal spatio-temporal feature fusion according to claim 1, characterized in that, The BP neural network model comprises an input layer, a hidden layer and an output layer. The BP neural network model uses two hidden layers, each of which has 20 neurons. The data of the input layer includes roll angle, pitch angle, normal acceleration, longitudinal acceleration, relative air pressure height, lifting speed, Mach number, total atmospheric temperature, longitudinal angle of the control column and force arm adjustment position. The output layer outputs whether it is an action point, wherein 0 represents a non-action point and 1 represents an action point.

3. The abnormal maneuver detection and recognition method based on multi-modal spatio-temporal feature fusion according to claim 1, characterized in that, The method for connecting the identified abnormal action points to form a complete abnormal maneuver action comprises the following steps: Connecting abnormal maneuver action segments: taking a certain abnormal action point as the center, searching forward and backward respectively, finding the last data point with a pitch angle less than 15 degrees and a roll angle less than 30 degrees as the starting point of the action segment when searching forward, and finding the first data point with a pitch angle less than 15 degrees, a roll angle less than 30 degrees and a maneuver_flag of 0 as the ending point of the action segment when searching backward. The flight parameter data between the starting point and the ending point is the abnormal maneuver action segment; Connecting the abnormal maneuver action segments, calculating the interval time of each abnormal maneuver action segment, and connecting the abnormal maneuver action segments with an interval time less than 5 seconds to form a complete abnormal maneuver action.

4. The abnormal maneuver detection and recognition method based on multi-modal spatio-temporal feature fusion according to claim 1, characterized in that, The method for stretching and converting the data of the abnormal maneuver action for pretreatment comprises the following steps: Using a multi-dimensional interpolation method and a k-nearest neighbor interpolation method to stretch the data of the abnormal maneuver action, so that the data length is 1024 and the dimension remains unchanged. The data of the abnormal maneuver after stretching is converted into a data dimension of 32*32*10, and there are 10 channels, each corresponding to a flight parameter feature, and the feature map size of each channel is 32*32.

5. An abnormal maneuver detection and recognition system based on multi-modal spatio-temporal feature fusion, characterized in that, The method comprises: a flight parameter data acquisition module, which acquires flight parameter data of an aircraft running to form sequence data; an abnormal action point identification module, which uses a BP neural network model to identify each frame of the sequence data to identify whether each frame is an abnormal action point; an abnormal maneuver formation module, which connects the identified abnormal action points to form a complete abnormal maneuver; a data preprocessing module, which performs stretching conversion preprocessing on the data of the abnormal maneuver; an abnormal maneuver type identification and detection module, which inputs the preprocessed data of the abnormal maneuver into a multi-modal spatio-temporal feature fusion abnormal maneuver identification model, and identifies and detects the type of the abnormal maneuver through the abnormal maneuver identification model; The multi-modal spatio-temporal feature fusion abnormal maneuver identification model mines abnormal maneuver features from two modalities of time and space, and filters more important features in different modalities through a cross-modal attention mechanism, and designs a progressive modal reinforcement unit to enhance the modal feature information focused by the attention mechanism through the progressive modal reinforcement unit to obtain an enhanced feature vector. The multi-modal spatio-temporal feature fusion abnormal maneuver identification model mines abnormal maneuver features from two modalities of time and space, comprising: a convolutional neural network is used to extract spatial features of the data of the abnormal maneuver; a bidirectional long short-term memory network is used to extract time features of the data of the abnormal maneuver; The abnormal maneuver identification model concatenates the obtained enhanced feature vector to obtain a vector used for maneuver identification, and then performs linear transformation through two fully connected layers, and uses a Relu function as an activation function in the two fully connected layers; The final maneuver identification type is obtained through a softmax function.

6. An electronic device, characterized by The computer program is stored on the computer readable storage medium and is executed by the processor to implement the multi-modal spatio-temporal feature fusion based abnormal maneuver detection and identification method according to any one of claims 1-4.

7. A computer readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executed by the processor to implement the multi-modal spatio-temporal feature fusion based abnormal maneuver detection and identification method according to any one of claims 1-4.

Images