A method for detecting anomalies in eVTOL flight control system based on multimodal sensing

Abstract

This invention discloses a method for detecting eVTOL flight control anomalies based on multimodal perception, comprising the following steps: S1, collecting and preprocessing multimodal perception data during eVTOL flight to construct a fused feature sequence; S2, using a sliding window to truncate the fused feature sequence and extracting the state representation sequence using a temporal convolutional network; S3, constructing a hybrid prediction model to generate a predicted state sequence based on the state representation sequence; S4, using the SimSiam method to perform feature perturbation and order transformation on the state representation sequence and the predicted state sequence to generate a difference feature vector; S5, constructing a covariance matrix based on the difference feature vector and calculating the Mahalanobis distance, identifying abnormal time steps according to a set distance threshold; S6, generating eVTOL flight control anomaly detection results based on the abnormal time steps. This invention integrates multimodal perception and contrastive learning, possessing high accuracy, fast response, and strong generalization ability.

Description

Technical Field

[0001] This invention relates to the field of eVTOL flight control system technology, and in particular to a method for detecting abnormal eVTOL flight control status based on multimodal perception. Background Technology

[0002] With the development of urban air mobility technology, electric vertical takeoff and landing (eVTOL) aircraft, with their advantages of low noise, zero emissions, and flexible operation, have shown broad application prospects in scenarios such as passenger commuting, logistics transportation, and emergency response. In order to ensure the flight safety of eVTOL in complex environments and improve the system's response capability under abnormal conditions, the flight control system has placed higher demands on its ability to perceive and identify flight status anomalies.

[0003] In existing technologies, eVTOL flight control status monitoring largely relies on single-modal data sources, such as inertial measurement units or radar ranging modules, and uses simple moving averages or threshold judgment methods to detect state anomalies. These methods are sensitive to sensor noise, have insufficient response to nonlinear fluctuations, and are difficult to cope with sudden state changes in highly dynamic flight. In addition, some studies have attempted to introduce long short-term memory networks or one-dimensional convolutional networks for state sequence modeling, but training is only performed under single-modal conditions, making it difficult to integrate multi-source heterogeneous data. This results in insufficient ability of the model to identify abnormal behavior, leading to high false alarm and false negative rates.

[0004] On the other hand, although some studies have proposed introducing contrastive learning or Mahalanobis distance into the flight control anomaly identification process, there are still the following shortcomings: First, existing contrastive learning methods mostly rely on labeled data for feature alignment training, making it difficult to achieve unsupervised or weakly supervised difference modeling during flight; Second, although discrimination methods such as Mahalanobis distance have mathematical rationality, if there is a lack of effective feature perturbation mechanisms and structured difference construction methods, they often cannot accurately locate abnormal changes in local time steps, thus limiting their practicality.

[0005] Furthermore, in the process of anomaly identification, traditional methods generally use fixed rules or template matching to make judgments, which cannot dynamically adjust the identification strategy in combination with the high-dimensional features of the actual flight state. This results in a low accuracy rate for the final anomaly classification and makes it difficult to provide interpretable scoring feedback.

[0006] Therefore, how to provide a method for detecting abnormal eVTOL flight control status based on multimodal perception is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] One objective of this invention is to propose an anomaly detection method for eVTOL flight control status based on multimodal perception. This invention fully integrates multi-source perception data such as video images, inertial measurement, radar ranging, and motor parameters. It combines temporal convolutional networks, hybrid prediction models, and SimSiam contrastive learning technology to construct an analytical framework that can finely model changes in flight status. The invention details the entire process from multimodal feature extraction, status prediction, difference measurement to anomaly identification and classification. It has the advantages of high detection accuracy, fast response speed, strong model generalization ability, and adaptability to complex flight environments.

[0008] A method for detecting anomalies in eVTOL flight control status based on multimodal sensing according to an embodiment of the present invention includes the following steps: S1. Acquire multimodal perception data of eVTOL during flight and preprocess it, then construct a fusion feature sequence according to time steps; S2. Based on a time window of a set length, the fused feature sequence is truncated by sliding. A time convolutional network containing a multi-layer dilated convolutional structure is used to perform causal convolution and residual connection operations on the fused features in each time window to generate a state representation sequence. S3. Construct a hybrid prediction model consisting of a convolutional neural network and a random forest algorithm. Perform multi-scale convolution operations on the state representation sequence to extract local change features. Based on the extraction results, perform multi-step trend inference to generate a predicted state sequence. S4. The SimSiam method is used to perform feature perturbation and order transformation operations on the state representation sequence and the predicted state sequence at time steps respectively. The perturbation results and the corresponding transformation results are then differentially processed to generate difference feature vectors. S5. Construct a covariance matrix based on all differential eigenvectors, calculate the Mahalanobis distance of the differential eigenvectors based on the covariance matrix, and identify abnormal time steps based on a set distance threshold. S6. Perform anomaly identification processing on the differential feature vectors of abnormal time steps, generate anomaly type sequences and scoring sequences, and combine them in time alignment to generate eVTOL flight control status anomaly detection results.

[0009] Optionally, the multi-source sensing data includes video image sequences, inertial measurement data, radar ranging data, motor operating parameters, control surface signals, and wind speed sensor data, and the abnormal time step represents a time node with a potential flight control state abnormality.

[0010] Optionally, the preprocessing specifically includes: AI vision technology is used to perform grayscale transformation and Gaussian filtering operations on video image sequences in sequence. Based on the filtering results, corner features in the image are extracted by the Harris corner detection method. The pixel intensity changes of corner features in consecutive image frames are identified by the inter-frame difference method to extract image change features. Inertial measurement data, radar ranging data, motor operating parameters, control surface control signals and wind speed sensor data are processed using Min-Max normalization, and linear interpolation is used to time-align the normalization results with the video image sequence. The normalized results, after aligning the image change features with the time sequence, are concatenated according to the feature dimension and arranged in chronological order to form a fused feature sequence.

[0011] Optionally, S2 specifically includes: S21. Construct a fixed-length sliding time window for the fused feature sequence to form multiple fixed-length sequence segments; S22. A temporal convolutional network containing multi-layer dilated convolutional structures is used to perform layer-by-layer feature extraction. Causal convolution operations are performed on sequence segments in each convolutional layer to generate convolutional response sequences. S23. Perform residual connection operation on the convolutional response sequence, extract the feature vector of each time step in the convolutional response sequence, calculate the mean of all feature values ​​in each feature vector as the fusion coefficient of the corresponding time step, and perform element-wise weighted fusion of the convolutional response sequence and the corresponding sequence segment to generate intermediate feature representation. S24. Concatenate all intermediate feature representations according to time steps to construct a state representation sequence.

[0012] Optionally, S3 specifically includes: S31. Construct a hybrid prediction model consisting of a convolutional neural network and a random forest algorithm. Under a preset convolutional scale, the convolutional neural network performs multi-scale convolution operations on the state representation sequence to extract the local change features corresponding to each time step. S32. Perform feature splitting, feature aggregation, and regression scoring operations on all locally changing features using the random forest algorithm; S33. Perform multi-step trend inference based on the regression score results of each time step, perform sequence expansion processing on the regression score results of consecutive time steps, and generate trend inference results, wherein the trend inference results represent the trend of state changes in multiple future time steps. S34. Arrange the trend inference results sequentially according to time steps to construct the predicted state sequence.

[0013] Optionally, the processing of the state representation sequence by the hybrid prediction model specifically includes: Multiple one-dimensional convolutional kernels are set in the convolutional neural network. One-dimensional convolution operations are performed on the state representation sequence at different convolutional scales. Batch normalization is performed on the convolutional responses at each scale, and the ReLU function is used for nonlinear activation to generate a set of convolutional features. Perform feature concatenation and average pooling operations on the convolutional feature set to extract the local change features corresponding to each time step; The random forest algorithm is used to perform feature splitting operation on all local change features to construct multiple regression decision trees. In each regression decision tree, the nodes are divided based on the criterion of minimizing the squared error. The mean value of all feature values ​​corresponding to the local change features falling into the same leaf node is calculated to generate the regression score value for the corresponding time step. All regression scores of each regression decision tree are arranged in chronological order to form a score time series, and moving average and trend extraction operations are performed to generate trend inference results. The predicted state sequence is formed by combining all the trend inference results according to time steps.

[0014] Optionally, the process of constructing the regression decision tree specifically includes: Set the maximum tree depth and minimum number of splits for each regression decision tree, initialize the root node, and randomly sample from all local variation features according to a set ratio to generate a sample feature set; Perform a candidate partitioning operation on the sample feature set of the current node, randomly select different feature dimensions and partitioning thresholds to construct multiple candidate partitioning schemes, and calculate the partitioning error corresponding to each candidate partitioning scheme; The candidate partitioning scheme with the smallest partitioning error is selected as the partitioning criterion for the current node. Based on the partitioning criterion, the sample feature set is divided into a left node set and a right node set, which are respectively used as the left child node and right child node of the root node. Recursively execute candidate splitting operations on the two child nodes as new root nodes until the node depth reaches the maximum tree depth or the number of sample features in the current node is lower than the minimum number of splits. Nodes that no longer meet the splitting conditions are marked as leaf nodes, thus completing the construction of the regression decision tree.

[0015] Optionally, S4 specifically includes: S41. Match the state representation sequence and the predicted state sequence by time step, and extract the state representation vector and the predicted state vector at each time step to form a feature pair containing the time index. S42. The SimSiam method is used to perform feature perturbation processing on the state representation vector of each feature pair. Based on the preset occlusion ratio, a pseudo-random number generator is used to select the feature dimension that needs to be zeroed, and the value of the corresponding feature dimension in the state representation vector is set to zero to generate the occlusion feature vector. S43. Perform an order transformation on the predicted state vectors of each feature pair, keeping the values ​​of each feature dimension unchanged, rearranging the order of all feature dimensions, and generating a transformed feature vector. S44. Perform difference processing on the occlusion feature vector and the corresponding transformed feature vector to generate the difference feature vector.

[0016] Optionally, S5 specifically includes: S51. Extract the values ​​of each feature dimension of all differential feature vectors in sequence according to the time step, calculate the average value of each feature dimension in all time steps, and calculate the average value of the product of the differences between all feature dimensions based on the difference between the average value and the value of each feature dimension, and construct the covariance matrix. S52. Based on the covariance matrix, perform distance calculation on the differential feature vectors at each time step, extract the Mahalanobis distance of each differential feature vector relative to the covariance matrix, and form a Mahalanobis distance sequence arranged by time step. S53. Based on the preset distance threshold, the Mahalanobis distance sequence is processed to identify and filter out time steps whose Mahalanobis distance is greater than the distance threshold as abnormal time steps.

[0017] Optionally, S6 specifically includes: S61. Extract the differential feature vectors corresponding to all abnormal time steps, and perform sorting and feature standardization operations. S62. Perform type determination processing on the standardized differential feature vector according to the preset recognition rules, and extract the anomaly type label and score value; S63. Combine all anomaly type labels and score values ​​in chronological order to generate anomaly type sequence and score sequence, and align them with the anomaly time step to generate eVTOL flight control status anomaly detection results.

[0018] The beneficial effects of this invention are: First, this invention introduces a multimodal perception mechanism, fusing video images, inertial measurement, radar ranging, motor operating parameters, control surface signals, and wind speed sensor data. This effectively solves the problems of existing methods' strong reliance on single sensors and incomplete state representation, significantly improving the comprehensiveness and robustness of flight control state perception. Combined with image feature extraction and multi-source data alignment processing in the preprocessing stage, a high-quality input foundation is provided for subsequent state modeling.

[0019] Secondly, this invention constructs a state modeling structure composed of a temporal convolutional network and a hybrid prediction model, which can accurately capture local dynamic change trends during eVTOL flight. By introducing the SimSiam method to compare and learn the state representation sequence with the predicted state sequence, the model's sensitivity to state deviations and anomalous features is effectively enhanced, significantly improving the accuracy and stability of difference extraction. Simultaneously, Mahalanobis distance combined with the covariance matrix is ​​used to identify anomalous time steps of the discrepancies, demonstrating strong anomaly detection capabilities.

[0020] Finally, based on preset identification rules, this invention completes the anomaly type determination and scoring sequence generation, realizing the automatic identification and classification output of eVTOL flight control status anomalies. This process not only improves the intelligence and interpretability of anomaly detection but also quantifies the degree of anomaly, providing support for subsequent safety control strategies. The overall system possesses advantages such as high detection accuracy, fast response speed, clear algorithm structure, and adaptability to complex flight environments, verifying the significant application value of this invention in actual flight control anomaly detection tasks. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0022] Figure 1 The flowchart shows a method for detecting abnormal eVTOL flight control status based on multimodal sensing proposed in this invention. Figure 2 This is a flowchart of the state representation and prediction sequence generation process for an eVTOL flight control state anomaly detection method based on multimodal sensing proposed in this invention. Figure 3 This is a flowchart of the anomaly identification process of an eVTOL flight control state anomaly detection method based on multimodal perception proposed in this invention. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0024] refer to Figure 1-3 A method for detecting abnormal eVTOL flight control status based on multimodal perception includes the following steps: S1. Acquire multimodal perception data of eVTOL during flight and preprocess it, then construct a fusion feature sequence according to time steps; S2. Based on a time window of a set length, the fused feature sequence is truncated by sliding. A time convolutional network containing a multi-layer dilated convolutional structure is used to perform causal convolution and residual connection operations on the fused features in each time window to generate a state representation sequence. S3. Construct a hybrid prediction model consisting of a convolutional neural network and a random forest algorithm. Perform multi-scale convolution operations on the state representation sequence to extract local change features. Based on the extraction results, perform multi-step trend inference to generate a predicted state sequence. S4. The SimSiam method is used to perform feature perturbation and order transformation operations on the state representation sequence and the predicted state sequence at time steps respectively. The perturbation results and the corresponding transformation results are then differentially processed to generate difference feature vectors. S5. Construct a covariance matrix based on all differential eigenvectors, calculate the Mahalanobis distance of the differential eigenvectors based on the covariance matrix, and identify abnormal time steps based on a set distance threshold. S6. Perform anomaly identification processing on the differential feature vectors of abnormal time steps, generate anomaly type sequences and scoring sequences, and combine them in time alignment to generate eVTOL flight control status anomaly detection results.

[0025] In this embodiment, the multi-source sensing data includes video image sequences, inertial measurement data, radar ranging data, motor operating parameters, control surface signals, and wind speed sensor data. The abnormal time step represents a time node with a potential flight control state abnormality.

[0026] In this embodiment, the preprocessing specifically includes: AI vision technology is used to perform grayscale transformation and Gaussian filtering operations on video image sequences in sequence. Based on the filtering results, corner features in the image are extracted by the Harris corner detection method. The pixel intensity changes of corner features in consecutive image frames are identified by the inter-frame difference method to extract image change features. Inertial measurement data, radar ranging data, motor operating parameters, control surface control signals and wind speed sensor data are processed using Min-Max normalization, and linear interpolation is used to time-align the normalization results with the video image sequence. The normalized results, after aligning the image change features with the time sequence, are concatenated according to the feature dimension and arranged in chronological order to form a fused feature sequence.

[0027] In this embodiment, the process of extracting image change features specifically includes: A weighted fusion method is used to perform grayscale transformation on each frame of the video image sequence, and the color regions of the image are weighted and added together according to a fixed weight coefficient to generate a grayscale image. A Gaussian filtering operation is performed on a grayscale image. The grayscale image is processed pixel by pixel using a filter template of a preset size. The value of each pixel is replaced with the weighted average value of the corresponding neighboring pixels to obtain a smooth image sequence. The neighborhood refers to a local set of pixels that are expanded outwards by a fixed order within the spatial coordinate range with any pixel in the grayscale image as the center. The weight coefficient of the weighted average value is determined by the spatial distance between the pixel and the center of the neighborhood and the grayscale difference. Harris corner detection is performed on each frame of smooth image. The gradient information of grayscale image in the horizontal and vertical directions is calculated to construct image gradient matrix. The corner response value of each pixel position is calculated based on image gradient matrix. Pixel positions with corner response values ​​greater than all corner response values ​​in the corresponding neighborhood are selected and filtered according to a set threshold to construct corner feature set. Perform inter-frame difference operation on the corner feature set of two consecutive frames, calculate the change amplitude of corner response value at the same spatial position, and select significant difference regions as image change features according to the preset change threshold.

[0028] In this embodiment, S2 specifically includes: S21. Construct a fixed-length sliding time window for the fused feature sequence to form multiple fixed-length sequence segments; S22. A temporal convolutional network containing multiple dilated convolutional structures is used to perform layer-by-layer feature extraction. At each convolutional layer, a causal convolution operation is performed on the sequence segment to generate a convolutional response sequence, specifically including: Construct a temporal convolutional network containing multiple dilated convolutional structures, with each convolutional layer having an independent dilation rate parameter and kernel size; Each sequence segment is input into the temporal convolutional network in chronological order, and one-dimensional causal convolution operations are performed sequentially through each convolutional layer to ensure that the current time step depends only on the feature information of the historical time steps. ReLU activation and batch normalization are performed on the convolution output at each time point; The convolution results output from each convolutional layer are concatenated according to time steps to construct a convolutional response sequence; S23. Perform residual connection operation on the convolutional response sequence, extract the feature vector of each time step in the convolutional response sequence, calculate the mean of all feature values ​​in each feature vector as the fusion coefficient of the corresponding time step, and perform element-wise weighted fusion of the convolutional response sequence and the corresponding sequence segment to generate intermediate feature representation. S24. Concatenate all intermediate feature representations according to time steps to construct a state representation sequence.

[0029] In this embodiment, S3 specifically includes: S31. Construct a hybrid prediction model consisting of a convolutional neural network and a random forest algorithm. Under a preset convolutional scale, the convolutional neural network performs multi-scale convolution operations on the state representation sequence to extract the local change features corresponding to each time step. S32. Perform feature splitting, feature aggregation, and regression scoring operations on all locally changing features using the random forest algorithm; S33. Perform multi-step trend inference based on the regression score results of each time step, perform sequence expansion processing on the regression score results of consecutive time steps, and generate trend inference results, wherein the trend inference results represent the trend of state changes in multiple future time steps. S34. Arrange the trend inference results sequentially according to time steps to construct the predicted state sequence.

[0030] In this embodiment, the processing of the state representation sequence by the hybrid prediction model specifically includes: Multiple one-dimensional convolutional kernels are set in the convolutional neural network. One-dimensional convolution operations are performed on the state representation sequence at different convolutional scales. Batch normalization is performed on the convolutional responses at each scale, and the ReLU function is used for nonlinear activation to generate a set of convolutional features. Perform feature concatenation and average pooling operations on the convolutional feature set to extract the local change features corresponding to each time step; The random forest algorithm is used to perform feature splitting operation on all local change features to construct multiple regression decision trees. In each regression decision tree, the nodes are divided based on the criterion of minimizing the squared error. The mean value of all feature values ​​corresponding to the local change features falling into the same leaf node is calculated to generate the regression score value for the corresponding time step. All regression scores of each regression decision tree are arranged in chronological order to form a score time series, and moving average and trend extraction operations are performed to generate trend inference results. The predicted state sequence is formed by combining all the trend inference results according to time steps.

[0031] In this embodiment, the process of constructing the regression decision tree specifically includes: Set the maximum tree depth and minimum number of splits for each regression decision tree, initialize the root node, and randomly sample from all local variation features according to a set ratio to generate a sample feature set; Perform a candidate partitioning operation on the sample feature set of the current node, randomly select different feature dimensions and partitioning thresholds to construct multiple candidate partitioning schemes, and calculate the partitioning error corresponding to each candidate partitioning scheme; The candidate partitioning scheme with the smallest partitioning error is selected as the partitioning criterion for the current node. Based on the partitioning criterion, the sample feature set is divided into a left node set and a right node set, which are respectively used as the left child node and right child node of the root node. Recursively execute candidate splitting operations on the two child nodes as new root nodes until the node depth reaches the maximum tree depth or the number of sample features in the current node is lower than the minimum number of splits. Nodes that no longer meet the splitting conditions are marked as leaf nodes, thus completing the construction of the regression decision tree.

[0032] In this embodiment, S4 specifically includes: S41. Match the state representation sequence and the predicted state sequence by time step, and extract the state representation vector and the predicted state vector at each time step to form a feature pair containing the time index. S42. The SimSiam method is used to perform feature perturbation processing on the state representation vector of each feature pair. Based on the preset occlusion ratio, a pseudo-random number generator is used to select the feature dimension that needs to be zeroed, and the value of the corresponding feature dimension in the state representation vector is set to zero to generate the occlusion feature vector. S43. Perform an order transformation on the predicted state vectors of each feature pair, keeping the values ​​of each feature dimension unchanged, rearranging the order of all feature dimensions, and generating a transformed feature vector. S44. Perform a difference processing on the occlusion feature vector and the corresponding transformed feature vector to generate a difference feature vector, specifically including: For each time step, the occlusion feature vector and the corresponding transformation feature vector are subtracted element by element along the feature dimension, and the numerical difference of each feature dimension is extracted. Squaring all numerical differences at each time step according to the feature dimension to form an initial set of differences; Perform a numerical summation operation on the initial set of differences, and arrange the summation results in chronological order to generate a difference feature vector.

[0033] In this embodiment, S5 specifically includes: S51. Extract the values ​​of each feature dimension of all differential feature vectors in sequence according to the time step, calculate the average value of each feature dimension in all time steps, and calculate the average value of the product of the differences between all feature dimensions based on the difference between the average value and the value of each feature dimension, and construct the covariance matrix. S52. Based on the covariance matrix, perform distance calculation on the differential feature vectors at each time step, extract the Mahalanobis distance of each differential feature vector relative to the covariance matrix, and form a Mahalanobis distance sequence arranged by time step. S53. Based on the preset distance threshold, the Mahalanobis distance sequence is processed to identify and filter out time steps whose Mahalanobis distance is greater than the distance threshold as abnormal time steps.

[0034] In this embodiment, the calculation process of the Mahalanobis distance specifically includes: Perform the inverse matrix operation on the covariance matrix, and add an identity matrix of the same dimension to the right side of the covariance matrix to form a concatenated matrix; Recursively process each row of the concatenated matrix, scaling the first non-zero element of the current row to 1, using the column containing the scaled element as the reference column, and performing addition and subtraction operations on the current row and all non-current rows of the concatenated matrix until all elements in the reference columns of all non-current rows are 0; When the covariance matrix part on the left side of the spliced ​​matrix is ​​transformed into the identity matrix, the recursive processing stops, and the right side of the processed spliced ​​matrix is ​​extracted as the inverse covariance matrix. The right side part represents the identity matrix part in the spliced ​​matrix before the recursive processing. Calculate the difference between each differential feature vector and the mean of all feature dimensions, and construct the difference vector; Perform a row and column swap operation on the difference vector, and perform matrix multiplication on the swap result and the covariance inverse matrix to generate an intermediate vector; Perform a dot product operation between the intermediate vector and the difference vector, and then perform square root processing on the dot product result to obtain the Mahalanobis distance at that time step. Record all Mahalanobis distances in chronological order to construct a Mahalanobis distance sequence.

[0035] In this embodiment, S6 specifically includes: S61. Extract the differential feature vectors corresponding to all abnormal time steps, and perform sorting and feature standardization operations. S62. Perform type determination processing on the standardized differential feature vector according to the preset recognition rules, and extract the anomaly type label and score value; S63. Combine all anomaly type labels and score values ​​in chronological order to generate anomaly type sequence and score sequence, and align them with the anomaly time step to generate eVTOL flight control status anomaly detection results.

[0036] Example 1: To verify the feasibility of this invention in practice, it was applied to the flight control status monitoring and anomaly detection task during the flight of an eVTOL (electric vertical takeoff and landing) aircraft. The main objective is to address the shortcomings of existing methods, such as insufficient multi-source sensor data fusion capabilities, large state prediction errors, and low anomaly identification accuracy. In actual flight missions, eVTOLs need to execute multi-stage flight commands in complex environments. Their flight control systems face continuously changing loads, wind speed disturbances, and control surface response lags, making them highly susceptible to hidden anomalies. Traditional methods based on single sensors or fixed rules struggle to detect abnormal trends in flight control status in a timely manner, resulting in high false positive rates and long detection delays, severely hindering the safe and efficient operation of the aircraft.

[0037] To address the aforementioned problems, this invention constructs a flight control anomaly detection process based on multimodal perception. First, the image acquisition unit acquires continuous video image frames during flight, while simultaneously acquiring triaxial acceleration and angular velocity output from the inertial measurement unit. The radar ranging module obtains environmental distance information, and the flight control system records motor speed, current, voltage, control surface status, and wind speed sensor data. In the preprocessing stage, the video image sequence undergoes grayscale conversion and Gaussian filtering, and corner features are extracted. Combined with other data sources, Min-Max normalization and linear interpolation are used for time-series alignment, ultimately constructing a time-synchronized, format-consistent fused feature sequence.

[0038] Subsequently, this invention employs a multi-layered dilated convolutional temporal convolutional network to perform causal convolution and residual connection operations on the fused feature sequence, thereby extracting a state representation sequence with historical dependencies. In the state modeling stage, a convolutional neural network is used to extract multi-scale local change features, and a random forest algorithm is combined to perform state trend regression inference, generating a predicted state sequence that corresponds one-to-one with the time steps of the state representation sequence. In the state comparison stage, the SimSiam method is used to perturb and transform the state representation vector and the predicted state vector respectively, generating a difference feature vector through differencing, and using the covariance matrix to calculate the Mahalanobis distance to identify potential abnormal time steps. Finally, this invention performs type determination and scoring calculation on the difference features of abnormal time steps according to preset identification rules, forming a complete anomaly detection result.

[0039] To evaluate the practical effectiveness of this method, the anomaly detection capabilities of three schemes on 14 flight mission samples in the same aircraft mission were compared and analyzed, including: the method of this invention, comparative method one (traditional rule-based thresholding method), and comparative method two (single-modal LSTM model). Evaluation metrics for each scheme included anomaly recognition accuracy, average detection latency, false alarm rate, false negative rate, and computation time.

[0040] The specific comparison data is shown in the table below: Table 1 Comparison of eVTOL flight control system anomaly detection performance results

[0041] As shown in Table 1, this invention demonstrated stable performance across all 14 flight missions, achieving an average accuracy of 96.1%, significantly outperforming traditional rule-based methods (60.1%) and the single-modal LSTM model (75.3%). Furthermore, this invention maintained an average detection latency of approximately 110ms, exhibiting excellent real-time response capabilities. Regarding false positives and false negatives, this invention achieved virtually no false negatives, and its false positive rate was kept at a significantly lower level, far superior to other methods.

[0042] In summary, the anomaly detection process constructed based on this invention can effectively integrate multi-source sensing data and accurately identify abnormal states in the eVTOL flight control process, demonstrating significant practical value and promising prospects for application in complex flight scenarios.

[0043] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for detecting anomalies in eVTOL flight control status based on multimodal sensing, characterized in that, Includes the following steps: S1. Acquire multimodal perception data of eVTOL during flight and preprocess it, then construct a fusion feature sequence according to time steps; S2. Based on a time window of a set length, the fused feature sequence is truncated by sliding. A time convolutional network containing a multi-layer dilated convolutional structure is used to perform causal convolution and residual connection operations on the fused features in each time window to generate a state representation sequence. S3. Construct a hybrid prediction model consisting of a convolutional neural network and a random forest algorithm. Perform multi-scale convolution operations on the state representation sequence to extract local change features. Based on the extraction results, perform multi-step trend inference to generate a predicted state sequence. S4. The SimSiam method is used to perform feature perturbation and order transformation operations on the state representation sequence and the predicted state sequence at time steps respectively. The perturbation results and the corresponding transformation results are then differentially processed to generate difference feature vectors. S5. Construct a covariance matrix based on all differential eigenvectors, calculate the Mahalanobis distance of the differential eigenvectors based on the covariance matrix, and identify abnormal time steps based on a set distance threshold. S6. Perform anomaly identification processing on the differential feature vectors of abnormal time steps, generate anomaly type sequences and scoring sequences, and combine them in time alignment to generate eVTOL flight control status anomaly detection results.

2. The method for detecting eVTOL flight control anomalies based on multimodal sensing according to claim 1, characterized in that, The multi-source sensing data includes video image sequences, inertial measurement data, radar ranging data, motor operating parameters, control surface signals, and wind speed sensor data. The abnormal time step represents a time node with a potential flight control status abnormality.

3. The method for detecting eVTOL flight control anomalies based on multimodal sensing according to claim 1, characterized in that, The preprocessing specifically includes: AI vision technology is used to perform grayscale transformation and Gaussian filtering operations on video image sequences in sequence. Based on the filtering results, corner features in the image are extracted by the Harris corner detection method. The pixel intensity changes of corner features in consecutive image frames are identified by the inter-frame difference method to extract image change features. Inertial measurement data, radar ranging data, motor operating parameters, control surface control signals and wind speed sensor data are processed using Min-Max normalization, and linear interpolation is used to time-align the normalization results with the video image sequence. The normalized results, after aligning the image change features with the time sequence, are concatenated according to the feature dimension and arranged in chronological order to form a fused feature sequence.

4. The method for detecting eVTOL flight control anomalies based on multimodal sensing according to claim 1, characterized in that, S2 specifically includes: S21. Construct a fixed-length sliding time window for the fused feature sequence to form multiple fixed-length sequence segments; S22. A temporal convolutional network containing multi-layer dilated convolutional structures is used to perform layer-by-layer feature extraction. Causal convolution operations are performed on sequence segments in each convolutional layer to generate convolutional response sequences. S23. Perform residual connection operation on the convolutional response sequence, extract the feature vector of each time step in the convolutional response sequence, calculate the mean of all feature values ​​in each feature vector as the fusion coefficient of the corresponding time step, and perform element-wise weighted fusion of the convolutional response sequence and the corresponding sequence segment to generate intermediate feature representation. S24. Concatenate all intermediate feature representations according to time steps to construct a state representation sequence.

5. The method for detecting eVTOL flight control anomalies based on multimodal sensing according to claim 1, characterized in that, S3 specifically includes: S31. Construct a hybrid prediction model consisting of a convolutional neural network and a random forest algorithm. Under a preset convolutional scale, the convolutional neural network performs multi-scale convolution operations on the state representation sequence to extract the local change features corresponding to each time step. S32. Perform feature splitting, feature aggregation, and regression scoring operations on all locally changing features using the random forest algorithm; S33. Perform multi-step trend inference based on the regression score results of each time step, perform sequence expansion processing on the regression score results of consecutive time steps, and generate trend inference results, wherein the trend inference results represent the trend of state changes in multiple future time steps. S34. Arrange the trend inference results sequentially according to time steps to construct the predicted state sequence.

6. The method for detecting eVTOL flight control anomalies based on multimodal perception according to claim 5, characterized in that, The specific processing steps of the hybrid prediction model for the state representation sequence include: Multiple one-dimensional convolutional kernels are set in the convolutional neural network. One-dimensional convolution operations are performed on the state representation sequence at different convolutional scales. Batch normalization is performed on the convolutional responses at each scale, and the ReLU function is used for nonlinear activation to generate a set of convolutional features. Perform feature concatenation and average pooling operations on the convolutional feature set to extract the local change features corresponding to each time step; The random forest algorithm is used to perform feature splitting operation on all local change features to construct multiple regression decision trees. In each regression decision tree, the nodes are divided based on the criterion of minimizing the squared error. The mean value of all feature values ​​corresponding to the local change features falling into the same leaf node is calculated to generate the regression score value for the corresponding time step. All regression scores of each regression decision tree are arranged in chronological order to form a score time series, and moving average and trend extraction operations are performed to generate trend inference results. The predicted state sequence is formed by combining all the trend inference results according to time steps.

7. The method for detecting eVTOL flight control anomalies based on multimodal sensing according to claim 6, characterized in that, The process of constructing the regression decision tree specifically includes: Set the maximum tree depth and minimum number of splits for each regression decision tree, initialize the root node, and randomly sample from all local variation features according to a set ratio to generate a sample feature set; Perform a candidate partitioning operation on the sample feature set of the current node, randomly select different feature dimensions and partitioning thresholds to construct multiple candidate partitioning schemes, and calculate the partitioning error corresponding to each candidate partitioning scheme; The candidate partitioning scheme with the smallest partitioning error is selected as the partitioning criterion for the current node. Based on the partitioning criterion, the sample feature set is divided into a left node set and a right node set, which are respectively used as the left child node and right child node of the root node. Recursively execute candidate splitting operations on the two child nodes as new root nodes until the node depth reaches the maximum tree depth or the number of sample features in the current node is lower than the minimum number of splits. Nodes that no longer meet the splitting conditions are marked as leaf nodes, thus completing the construction of the regression decision tree.

8. The method for detecting eVTOL flight control anomalies based on multimodal sensing according to claim 1, characterized in that, S4 specifically includes: S41. Match the state representation sequence and the predicted state sequence by time step, and extract the state representation vector and the predicted state vector at each time step to form a feature pair containing the time index. S42. The SimSiam method is used to perform feature perturbation processing on the state representation vector of each feature pair. Based on the preset occlusion ratio, a pseudo-random number generator is used to select the feature dimension that needs to be zeroed, and the value of the corresponding feature dimension in the state representation vector is set to zero to generate the occlusion feature vector. S43. Perform an order transformation on the predicted state vectors of each feature pair, keeping the values ​​of each feature dimension unchanged, rearranging the order of all feature dimensions, and generating a transformed feature vector. S44. Perform difference processing on the occlusion feature vector and the corresponding transformed feature vector to generate the difference feature vector.

9. The method for detecting eVTOL flight control anomalies based on multimodal sensing according to claim 1, characterized in that, S5 specifically includes: S51. Extract the values ​​of each feature dimension of all differential feature vectors in sequence according to the time step, calculate the average value of each feature dimension in all time steps, and calculate the average value of the product of the differences between all feature dimensions based on the difference between the average value and the value of each feature dimension, and construct the covariance matrix. S52. Based on the covariance matrix, perform distance calculation on the differential feature vectors at each time step, extract the Mahalanobis distance of each differential feature vector relative to the covariance matrix, and form a Mahalanobis distance sequence arranged by time step. S53. Based on the preset distance threshold, the Mahalanobis distance sequence is processed to identify and filter out time steps whose Mahalanobis distance is greater than the distance threshold as abnormal time steps.

10. The method for detecting eVTOL flight control anomalies based on multimodal perception according to claim 1, characterized in that, S6 specifically includes: S61. Extract the differential feature vectors corresponding to all abnormal time steps, and perform sorting and feature standardization operations. S62. Perform type determination processing on the standardized differential feature vector according to the preset recognition rules, and extract the anomaly type label and score value; S63. Combine all anomaly type labels and score values ​​in chronological order to generate anomaly type sequence and score sequence, and align them with the anomaly time step to generate eVTOL flight control status anomaly detection results.

Images