Abnormity detection method for unmanned aerial vehicle multi-sensor system and storage medium

Abstract

The invention discloses an anomaly detection method for an unmanned aerial vehicle multi-sensor system and a storage medium, and belongs to the technical field of unmanned aerial vehicle anomaly detection. The method comprises the following steps: receiving an exception analysis task; inputting the unmanned aerial vehicle flight data into an unmanned aerial vehicle anomaly analysis model to obtain anomaly analysis information of an anomaly analysis task; the unmanned aerial vehicle anomaly analysis model comprises a multi-modal feature extraction network, a known anomaly recognition module and an unknown anomaly detection module; the multi-modal feature extraction network is used for feature extraction; the known anomaly identification module is used for identifying known anomaly types; the unknown anomaly detection module is used for detecting unknown anomaly; and the unknown anomaly detection module is used for identifying the type of the unknown anomaly in combination with a preset unknown anomaly identification algorithm under the condition that the unknown anomaly is detected. By using the technical scheme provided by the invention, fine-grained distinguishing of known anomalies and unknown anomalies of the multi-sensor system of the unmanned aerial vehicle can be realized, and the accuracy of anomaly detection of the multi-sensor system of the unmanned aerial vehicle is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) anomaly detection, and particularly to an anomaly detection method and a storage medium for a multi-sensor system of UAVs. Background Art

[0002] With the continuous development of UAV technology, UAV technology can be integrated with many cutting-edge technologies such as electronic information and artificial intelligence, which not only greatly increases the functions of UAVs, but also significantly improves their working efficiency. Currently, UAVs are widely used in many civilian fields due to their convenience, flexibility, and cost-effectiveness advantages, including target detection, wireless communication, agricultural remote sensing, disaster management, intelligent transportation systems, etc. Although UAVs have great application potential, they still face serious safety problems. A UAV is a complex physical system composed of many components such as sensors, engines, and wings. Once these mechanical parts fail, it will also lead to the failure or even crash of the UAV flight, causing serious consequences. In order to timely detect various anomalies that occur during the operation of UAVs and respond quickly, UAV anomaly detection technology has emerged and gradually become a research hotspot in the UAV field.

[0003] The purpose of anomaly detection is to identify abnormal behaviors or patterns in data, which is an important prerequisite for subsequent implementation of corrective measures. During the flight of a UAV, a large amount of flight data will be generated. These data are usually collected by various sensors and are important indicators reflecting the performance of the UAV such as stability, maneuverability, and flight efficiency. Therefore, anomaly detection for UAV flight data is an effective measure to ensure UAV flight safety. Currently, there are many studies on UAV anomaly detection, and common detection methods can be divided into three categories, including knowledge-based methods, model-based methods, and data-driven methods. Knowledge-based methods design anomaly detection rules and algorithms relying on the professional skills and knowledge of UAV experts. This method has very good detection effects when the UAV system is relatively simple or the failure modes are clearly distinguishable. However, these methods rely heavily on prior knowledge and have very limited flexibility. Model-based methods first comprehensively model the physical characteristics of UAVs, and then use the established model to estimate the behavior of UAVs. By comparing the difference between the estimated value and the actual value, it is judged whether the UAV is abnormal. Although model-based methods are robust and can be adjusted according to specific fields and dataset types, it is very difficult to establish an accurate physical model of UAVs in practice.

[0004] Compared with knowledge-based and model-based methods, data-driven methods do not require modeling the complex physical characteristics of drones, nor do they rely on a large amount of prior knowledge, and can make full use of drone flight data. Therefore, data-driven methods have attracted the attention of researchers and become the mainstream means of drone anomaly detection. Data-driven methods aim to analyze the statistical characteristics or distributions of drone flight data and identify abnormal data that is different from the normal data distribution. Among numerous data-driven methods, due to the rapid development of computer science and artificial intelligence, data-driven methods based on deep learning have become increasingly prominent in the field of drone anomaly detection. These methods can effectively capture the complexity and variability in drone flight data.

[0005] Although data-driven methods based on deep learning have achieved good detection results in drone anomaly detection, there are still many problems to be solved. Drone anomaly detection methods based on deep learning can generally be divided into two categories, including supervised learning methods and unsupervised learning methods.

[0006] Existing supervised learning methods can only detect known drone anomaly patterns. However, drones have complex structures and variable operating environments. During the initial deployment, abnormal states that have never occurred may occur. Once unknown anomalies appear, existing models will be powerless. And existing unsupervised learning methods can only achieve binary classification of normal / abnormal and cannot distinguish more finely what kind of anomaly has occurred. This has hindered the subsequent targeted maintenance of drones. Therefore, more reliable solutions are needed. Summary of the Invention

[0007] The purpose of the present invention is to overcome the deficiencies in the prior art and provide an anomaly detection method and storage medium for a drone multi-sensor system, which can improve the accuracy and efficiency of anomaly detection in the drone multi-sensor system.

[0008] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0009] On the one hand, the present invention provides an anomaly detection method for a drone multi-sensor system, and the method includes:

[0010] Receiving an anomaly analysis task sent by a user;

[0011] Based on the anomaly analysis task, obtaining drone flight data collected by the drone multi-sensor system;

[0012] Inputting the drone flight data into a drone anomaly analysis model to obtain anomaly analysis information corresponding to the anomaly analysis task;

[0013] Among them, the UAV anomaly analysis model includes a multi-modal feature extraction network, a known anomaly recognition module, and an unknown anomaly detection module; the multi-modal feature extraction network is used for feature extraction, and the multi-modal feature extraction network is obtained by adjusting the parameters of the initial multi-modal feature extraction network based on the triplet margin total loss, and the triplet margin total loss is determined based on the triplet set in combination with the triplet loss function; the known anomaly recognition module is used to recognize known anomaly types; the unknown anomaly detection module is used to detect unknown anomalies; when the unknown anomaly detection module detects an unknown anomaly, it is combined with a preset unknown anomaly recognition algorithm to identify the unknown anomaly type.

[0014] In some possible implementation manners, the anomaly analysis task includes a known anomaly analysis task and an unknown anomaly analysis task;

[0015] Inputting the UAV flight data into the UAV anomaly analysis model to obtain the anomaly analysis information corresponding to the anomaly analysis task includes:

[0016] When the anomaly analysis task is a known anomaly analysis task and an unknown anomaly analysis task, inputting the UAV flight data into the multi-modal feature extraction network for feature extraction to obtain flight multi-modal feature data;

[0017] Based on the known anomaly recognition module, performing known anomaly recognition processing on the flight multi-modal feature data to obtain the known anomaly type information of the flight multi-modal feature data;

[0018] Based on the unknown anomaly detection module, performing unknown anomaly detection processing on the flight multi-modal feature data to obtain the unknown anomaly detection information of the flight multi-modal feature data;

[0019] Based on the unknown anomaly detection information and the preset unknown anomaly recognition algorithm, performing unknown anomaly recognition processing on the unknown anomaly detection information to obtain the unknown anomaly type information of the flight multi-modal feature data;

[0020] Based on the known anomaly type information and the unknown anomaly type information, determining the anomaly analysis information of the anomaly analysis task.

[0021] In some possible implementation manners, the multi-modal feature extraction network includes a time series feature extraction network, an attitude feature extraction network, and a feature fusion network; the UAV flight data includes time series data and attitude data;

[0022] Inputting the UAV flight data into the multi-modal feature extraction network for feature extraction to obtain flight multi-modal feature data includes:

[0023] Input the time series data into the time series feature extraction network for time series feature extraction to obtain time series features;

[0024] Input the pose data into the pose feature extraction network for pose feature extraction to obtain pose features;

[0025] Input the time series features and the pose features into the feature fusion network for feature fusion processing to obtain flight multi-modal feature data.

[0026] In some possible implementation manners, the multi-modal feature extraction network is constructed in the following manner:

[0027] Train an initial multi-modal feature extraction network including an initial time series feature extraction network, an initial pose feature extraction network, and an initial feature fusion network to obtain an initial multi-modal feature extraction network;

[0028] Obtain multi-modal flight sample data;

[0029] Input the multi-modal flight sample data into the initial multi-modal feature extraction network for feature extraction processing to obtain flight sample data features;

[0030] Perform triplet extraction processing on the flight sample data features to obtain a triplet set;

[0031] Based on the triplet set, combine a triplet loss function to determine the total triplet margin loss;

[0032] Based on the total triplet margin loss, adjust the parameters of the initial multi-modal feature extraction network to obtain a multi-modal feature extraction network.

[0033] In some possible implementation manners, the performing triplet extraction processing on the flight sample data features to obtain a triplet set includes:

[0034] Traverse each flight data feature in the flight sample data features;

[0035] When any flight data feature is traversed, determine the flight data feature as the current anchor data;

[0036] Determine multiple positive flight data with the same abnormal type as the current anchor data in the flight sample data features;

[0037] Based on the distance between the current anchor data and each positive flight data, determine the maximum positive distance;

[0038] Determine multiple negative flight data with different abnormal types from the current anchor data in the flight sample data features;

[0039] Determine the minimum negative distance based on the distance between the current anchor data and each piece of the negative flight data;

[0040] Determine the positive flight data corresponding to the first distance between the current anchor data and the first target distance being greater than the first target distance as the current positive flight data; the first target distance is the sum of the minimum negative distance and the preset distance;

[0041] Determine the negative flight data corresponding to the second distance between the current anchor data and the second target distance being less than the second target distance as the current negative flight data; the second target distance is the difference between the maximum positive distance and the preset distance;

[0042] Take the current anchor data, the current positive flight data, and the current negative flight data as a triple;

[0043] After traversing each flight data feature in the flight sample data features, take the obtained multiple triples as a triple set.

[0044] In some possible implementation manners, the unknown anomaly detection processing of the flight multimodal feature data based on the unknown anomaly detection module to obtain the unknown anomaly detection information of the flight multimodal feature data includes:

[0045] Determine multiple preset known anomaly feature data and the flight multimodal feature data as mixed feature data;

[0046] Perform clustering processing on the mixed feature data based on a preset clustering algorithm to obtain a set of clustering feature data clusters;

[0047] Traverse each clustering feature data cluster in the set of clustering feature data clusters;

[0048] When traversing to any clustering feature data cluster, determine that there is no preset known anomaly feature data in the cluster, and the feature data in the current clustering feature data cluster where the number of feature data in the cluster is greater than or equal to the preset number is unknown anomaly data;

[0049] Determine the unknown anomaly detection information of the flight multimodal feature data based on the unknown anomaly data.

[0050] In some possible implementation manners, the unknown anomaly recognition processing of the unknown anomaly detection information based on the unknown anomaly detection information and a preset unknown anomaly recognition algorithm to obtain the unknown anomaly type information of the flight multimodal feature data includes:

[0051] Obtain a training set, a test set, and a support set; the support set includes a preset number of data with unknown anomaly types, and the preset number of data with unknown anomaly types is obtained by performing unknown anomaly type marking processing on a preset number of first unknown anomaly data, and the preset number of first unknown anomaly data is obtained from the unknown anomaly detection information; the test set includes a known anomaly type data set and an unknown anomaly data set, the unknown anomaly data set includes a plurality of second unknown anomaly data, and the preset number of first unknown anomaly data and the plurality of second unknown anomaly data constitute the unknown anomaly detection information; the training set includes known anomaly type data samples;

[0052] Determine the support set and the training set as a mixed data set;

[0053] Input the mixed data set into a multi-modal feature extraction network for feature extraction to obtain a mixed data feature set;

[0054] Input the test set into a multi-modal feature extraction network for feature extraction to obtain a test data feature set;

[0055] Traverse each test data feature in the test data feature set;

[0056] When traversing any test data feature, determine the vector distance between the current test data feature and each mixed data feature;

[0057] Determine the anomaly type label of the mixed data feature corresponding to the minimum distance among the N vector distances as the anomaly type label of the current test data feature;

[0058] Based on the anomaly type label of each test data feature, determine the unknown anomaly type information.

[0059] In some possible implementation manners, the performing known anomaly recognition processing on the flight multi-modal feature data based on the known anomaly recognition module to obtain the known anomaly type information of the flight multi-modal feature data includes:

[0060] Based on a preset known anomaly recognition algorithm, perform known anomaly recognition processing on the flight multi-modal feature data to obtain the known anomaly type information of the flight multi-modal feature data, where the preset known anomaly recognition algorithm includes the nearest neighbor algorithm.

[0061] In some possible implementation manners, before obtaining the drone flight data collected by the drone multi-sensor system based on the anomaly analysis task, it includes:

[0062] Based on the abnormal analysis task, obtain the original UAV flight data collected by the UAV multi-sensor system. The original UAV flight data includes multiple flight status data, and each flight status data includes multiple status acquisition data;

[0063] Perform quantity alignment processing on the status acquisition data of the multiple flight status data to obtain multiple aligned flight status data, and each aligned flight status data includes multiple aligned status acquisition data;

[0064] Perform data standardization processing on the multiple aligned status acquisition data of each aligned flight status data to obtain multiple standardized flight status data, and each standardized flight status data includes multiple standardized status acquisition data;

[0065] Based on the multiple standardized status acquisition data of each standardized flight status data, combine with a sliding window to perform data reconstruction processing to determine the UAV flight data.

[0066] On the other hand, the present invention provides a computer storage medium, in which at least one instruction and at least one program segment are stored. The at least one instruction and the at least one program segment are loaded and executed by a processor to implement the abnormal detection method for the UAV multi-sensor system as described above.

[0067] Compared with the prior art, the beneficial effects achieved by the present invention:

[0068] In the present invention, by receiving the abnormal analysis task sent by the user, the needs of the user for abnormal detection of the UAV multi-sensor system can be determined; then based on the abnormal analysis task, obtain the UAV flight data collected by the UAV multi-sensor system, input the UAV flight data into the UAV abnormal analysis model, and obtain the abnormal analysis information corresponding to the abnormal analysis task. The abnormalities existing in the UAV multi-sensor system and the corresponding abnormal types can be determined based on the user's needs, improving the accuracy and efficiency of abnormal detection of the UAV multi-sensor system, and reducing resource waste;

[0069] The UAV anomaly analysis model includes a multi-modal feature network, a known anomaly recognition module, and an unknown anomaly recognition module. The multi-modal feature extraction network is used for feature extraction, and the multi-modal feature extraction network is obtained by adjusting the parameters of the initial multi-modal feature extraction network based on the triplet margin total loss. It can effectively balance the relationship between different modal features, reduce the distance between the same type of features, and increase the distance between different types of features, thereby improving the accuracy and efficiency of known anomaly recognition and unknown anomaly recognition. The known anomaly recognition module is used to recognize known anomaly types, and the unknown anomaly detection module is used to detect unknown anomalies. When the unknown anomaly detection module detects an unknown anomaly, it combines a preset unknown anomaly recognition algorithm to identify the unknown anomaly type, which can realize the discovery of potential unknown anomalies in the UAV multi-sensor system and the identification of known and unknown anomaly types in the UAV multi-sensor system, and improve the accuracy and efficiency of the identification of known and unknown anomaly types. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0071] Figure 1 is a schematic flowchart of an anomaly detection method for a UAV multi-sensor system provided by an embodiment of the present invention;

[0072] Figure 2 is a schematic flowchart of the process before obtaining the UAV flight data collected by the UAV multi-sensor system based on an anomaly analysis task provided by an embodiment of the present invention;

[0073] Figure 3 is a schematic flowchart of the process of inputting the UAV flight data into the UAV anomaly analysis model to obtain the anomaly analysis information corresponding to the anomaly analysis task provided by an embodiment of the present invention;

[0074] Figure 4 is a schematic flowchart of the determination of flight multi-modal feature data provided by an embodiment of the present invention;

[0075] Figure 5 is a schematic diagram of a temporal feature extraction network and an attitude feature extraction network provided by an embodiment of the present invention;

[0076] Figure 6 is a schematic flowchart of the operation of a Transformer encoder provided by an embodiment of the present invention;

[0077] Figure 7 It is a schematic flow chart of constructing a multi-modal feature extraction network provided by an embodiment of the present invention;

[0078] Figure 8 It is a schematic flow chart of performing triple extraction processing on the features of flight sample data to obtain a triple set provided by an embodiment of the present invention;

[0079] Figure 9 It is a schematic diagram of a triple provided by an embodiment of the present invention;

[0080] Figure 10 It is a schematic flow chart of performing unknown anomaly detection processing on flight multi-modal feature data based on an unknown anomaly detection module to obtain unknown anomaly detection information of the flight multi-modal feature data provided by an embodiment of the present invention;

[0081] Figure 11 It is a schematic flow chart of performing unknown anomaly recognition processing on the unknown anomaly detection information based on the unknown anomaly detection information and a preset unknown anomaly recognition algorithm to obtain unknown anomaly type information of the flight multi-modal feature data provided by an embodiment of the present invention;

[0082] Figure 12 It is a schematic structural diagram of an anomaly detection device for a multi-sensor system of an unmanned aerial vehicle provided by an embodiment of the present invention. Detailed implementation manners

[0083] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0084] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or server including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0085] In the embodiments of the present invention, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of the module or unit.

[0086] Various exemplary embodiments, features, and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0087] The special term "exemplary" herein means "serving as an example, an embodiment, or illustrative". Any embodiment described as "exemplary" herein does not have to be construed as superior to or better than other embodiments.

[0088] The term "and / or" in this document is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" in this document means any one of multiple or any combination of at least two of multiple. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set composed of A, B, and C.

[0089] In addition, to better illustrate the present invention, numerous specific details are given in the following specific implementation manners. Those skilled in the art should understand that the present invention can also be implemented without some specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail to highlight the gist of the present invention.

[0090] Figure 1 is a schematic flowchart of an anomaly detection method for a drone multi-sensor system provided by an embodiment of the present invention. This specification provides method operation steps such as in the embodiment or the flowchart, but based on conventional or non-creative labor, there can be more or fewer operation steps. The order of steps listed in the embodiment is only one way among the execution orders of numerous steps and does not represent the only execution order. When the actual system or server product is executed, it can be executed in the order of the method shown in the embodiment or the drawings or executed in parallel (for example, in an environment of parallel processors or multi-threaded processing). Specifically, as Figure 1 shown, the above method may include:

[0091] S101: Receive the abnormal analysis task sent by the user;

[0092] In a specific embodiment, the abnormal analysis task may be a task for determining the abnormal situation of the multi-sensor system of the unmanned aerial vehicle. Optionally, the abnormal analysis task may include a known abnormal analysis task and an unknown abnormal analysis task. The known abnormal analysis task may be a task for determining whether there is a known abnormality in the multi-sensor system of the unmanned aerial vehicle and the type of the existing known abnormality; the unknown abnormal analysis task may be a task for determining whether there is an unknown abnormality in the multi-sensor system of the unmanned aerial vehicle and the type of the existing unknown abnormality. Optionally, the known abnormality may be a clearly defined abnormality; the unknown abnormality may be an undefined abnormality.

[0093] S102: Based on the abnormal analysis task, obtain the unmanned aerial vehicle flight data collected by the multi-sensor system of the unmanned aerial vehicle;

[0094] In a specific embodiment, the unmanned aerial vehicle flight data may be data characterizing the flight state of the unmanned aerial vehicle. Specifically, the unmanned aerial vehicle flight data may reflect the motion information and spatial attitude of the unmanned aerial vehicle. Optionally, the unmanned aerial vehicle flight data may include time-series data and attitude data. The time-series data may reflect the dynamic motion information of the unmanned aerial vehicle. Specifically, the time-series data may include the angular velocity, linear acceleration, etc. of the unmanned aerial vehicle; the attitude data may reflect the spatial attitude of the unmanned aerial vehicle. Specifically, the attitude data may include the pitch angle, roll angle, yaw angle, etc. of the unmanned aerial vehicle. In the case of clarifying the abnormal analysis task sent by the user, obtain the unmanned aerial vehicle flight data collected by the multi-sensor system of the unmanned aerial vehicle for corresponding abnormal analysis processing.

[0095] In an optional embodiment, Figure 2 is a schematic flow diagram before obtaining the unmanned aerial vehicle flight data collected by the multi-sensor system of the unmanned aerial vehicle based on the abnormal analysis task provided by the embodiment of the present invention; as Figure 2 shown, before obtaining the unmanned aerial vehicle flight data collected by the multi-sensor system of the unmanned aerial vehicle based on the abnormal analysis task, it may include:

[0096] S201: Obtain the original unmanned aerial vehicle flight data collected by the multi-sensor system of the unmanned aerial vehicle. The original unmanned aerial vehicle flight data includes multiple flight state data, and each flight state data includes multiple state acquisition data;

[0097] S202: Perform quantity alignment processing on the state acquisition data of the multiple flight state data to obtain multiple aligned flight state data, and each aligned flight state data includes multiple aligned state acquisition data;

[0098] S203: Perform data standardization processing on the multiple aligned state acquisition data of each aligned flight state data to obtain multiple standardized flight state data, where each standardized flight state data includes multiple standardized state acquisition data;

[0099] S204: Based on the multiple standardized state acquisition data of each standardized flight state data, perform data reconstruction processing in combination with a sliding window to determine the UAV flight data.

[0100] In a specific embodiment, the multiple flight state data may be data representing the kinematic state of the UAV. Specifically, the multiple flight state data may include angular velocity data rotating around the X-axis, linear acceleration data in the X-axis direction, pitch angle data, roll angle data, etc. Specifically, classifying the multiple flight state data can be divided into time series data and attitude data. The multiple state acquisition data may be obtained by sampling each flight state data according to the sampling frequency.

[0101] The multiple flight state data can be denoted as , indicating the number of flight state data; optionally, for any UAV flight state data there is a sampling frequency ; after sampling, will include multiple state acquisition data, that is, where represents the th state acquisition data in the th flight state data, represents the number of state acquisition data corresponding to the th flight state data; optionally, each state acquisition data has a corresponding timestamp, that is, the timestamp of the sampled state acquisition data. At this time, the timestamp set of can be where represents the timestamp corresponding to the th state acquisition data in the th flight state data; optionally, since the sampling frequencies corresponding to different flight state data may be different, that is, when there may be ; and the number of state acquisition data corresponding to different flight state data may be different, that is, when there may be represents the th flight state data.

[0102] In a specific embodiment, the above process of aligning the number of status acquisition data for multiple flight status data to obtain multiple aligned flight status data may include: using a linear interpolation algorithm to perform the process of aligning the number of status acquisition data for multiple flight status data to obtain multiple aligned flight status data. The specific steps are as follows:

[0103] Based on the number of status acquisition data for each flight status data, perform a descending order sorting on the multiple flight status data to obtain a descending order sorting result of the multiple flight status data;

[0104] Based on the descending order sorting result of the multiple flight status data, determine a reference status data; optionally, the reference status data may be the flight status data with the first position in the descending order sorting result. Specifically, the reference status data may be the flight status data with the largest number of status acquisition data;

[0105] Obtain the reference timestamp set of the reference status data;

[0106] Select any one of the multiple flight status data as the flight status data to be interpolated, and obtain the timestamp set of the flight status data to be interpolated;

[0107] Select a reference timestamp from the reference timestamp set as the current reference timestamp;

[0108] Determine a first timestamp and a second timestamp in the timestamp set; the first timestamp and the second timestamp are the two timestamps closest to the current reference timestamp, and the current reference timestamp is greater than the first timestamp and less than the second timestamp;

[0109] Obtain the first status acquisition data corresponding to the first timestamp and the second status acquisition data corresponding to the second timestamp from the flight status data to be interpolated;

[0110] Determine the interpolated status acquisition data corresponding to the flight status data to be interpolated at the current reference timestamp as shown in the following formula:

[0111] ;

[0112] Wherein, represents the th interpolated status acquisition data of the th flight status data to be interpolated; represents the current reference timestamp; represents the first status acquisition data; represents the second status acquisition data; represents the first timestamp; represents the second timestamp; represents the The nd linear interpolation processing of the flight state data to be interpolated;

[0113] Obtain the next reference timestamp in the reference timestamp set as the current reference timestamp, and jump to the above to determine the first timestamp and the second timestamp in the timestamp set, and obtain the interpolation state acquisition data of the flight state data to be interpolated at the next reference timestamp;

[0114] Repeat the above process of selecting a reference timestamp from the reference timestamp set as the current reference timestamp until obtaining the next reference timestamp in the reference timestamp set as the current reference timestamp, and jump to the above to determine the first timestamp and the second timestamp in the timestamp set, and obtain the interpolation state acquisition data of the flight state data to be interpolated at the next reference timestamp, until all timestamps in the reference timestamp set are traversed, and the interpolated flight state data can be expressed as follows:

[0115] ;

[0116] wherein, represents the th interpolated flight state data; represents the th interpolated flight state data corresponding to the th interpolation state acquisition data; represents the number of state acquisition data corresponding to the reference state data;

[0117] Select the next flight state data to be interpolated, that is, jump to select any flight state data from multiple flight state data as the flight state data to be interpolated, and obtain the timestamp set of the flight state data to be interpolated, until all flight state data have completed linear interpolation, and multiple aligned flight state data are obtained; each aligned flight state data includes multiple aligned state acquisition data (interpolation state acquisition data), and the aligned state acquisition data corresponding to multiple aligned flight state data are aligned in quantity.

[0118] In a specific embodiment, the above data normalization processing of multiple aligned state acquisition data of each aligned flight state data to obtain multiple normalized flight state data may include:

[0119] Select any aligned flight state data from multiple aligned flight state data as the flight state data to be normalized;

[0120] Determine the state acquisition mean data corresponding to the flight state data to be normalized, as shown in the following formula:

[0121] ;

[0122] Among them, represents the mean value of state acquisition corresponding to the th flight state data to be standardized (aligned flight state data);

[0123] Determine the standard deviation data of state acquisition corresponding to the flight state data to be standardized, as shown in the following formula:

[0124] ;

[0125] Among them, represents the standard deviation data of state acquisition corresponding to the th flight state data to be standardized;

[0126] According to the mean value data of state acquisition and the standard deviation data of state acquisition, perform data standardization processing on each aligned state acquisition data of the flight state data to be standardized, and obtain multiple standardized state acquisition data, as shown in the following formula:

[0127] ;

[0128] Among them, represents the th standardized state acquisition data in the th flight state data to be standardized; that is, represents the th aligned state acquisition data in the th flight state data to be standardized, and after performing data standardization processing, the th standardized state acquisition data obtained;

[0129] According to multiple standardized state acquisition data, obtain standardized flight state data;

[0130] Select the next aligned flight state data, that is, jump to select any aligned flight state data from multiple aligned flight state data as the flight state data to be standardized, until all aligned flight state data have completed data standardization processing, and obtain multiple standardized flight state data, and each standardized flight state data includes multiple standardized state acquisition data.

[0131] In a specific embodiment, the above-mentioned multiple standardized state acquisition data based on each standardized flight state data, combined with a sliding window for data reconstruction processing to determine the UAV flight data, may include:

[0132] Parallelly splice multiple standardized state acquisition data together according to the time stamp order;

[0133] Create a sliding window with a length equal to the preset window length and a sliding step equal to the preset sliding step; optionally, the preset window length and the preset sliding step can be set according to the actual application;

[0134] Start from the first timestamp of the sliding window and slide on the time axis;

[0135] Before each slide, extract the standardized flight state data corresponding to the timestamps of the preset window length from all the standardized flight state data within the sliding window, and combine them to obtain the reconstructed flight state data. Each reconstructed flight state data can include multiple reconstructed state acquisition data;

[0136] Based on multiple reconstructed flight state data, obtain the reconstructed UAV flight data, that is:

[0137] ;

[0138] where, represents the th reconstructed UAV flight data; represents the th reconstructed flight state data in the th reconstructed UAV flight data; represents the th reconstructed state acquisition data in the th reconstructed flight state data; represents the preset window length;

[0139] The sliding window will slide backward along the time axis by the number of timestamps of the preset sliding step and complete the construction of the next reconstructed UAV flight data. Repeat the above process that the sliding window will slide backward along the time axis by the number of timestamps of the preset sliding step and complete the construction of the next reconstructed UAV flight data until the sliding window traverses all timestamps, obtaining multiple reconstructed UAV flight data, and each reconstructed UAV flight data includes multiple reconstructed flight state data;

[0140] Based on multiple reconstructed UAV flight data, obtain the UAV flight data.

[0141] In the above embodiments, the data quality can be improved, the data diversity can be increased, and the associations between heterogeneous features at the same moment can be captured. At the same time, the temporal relationships existing between heterogeneous features can also be captured, thereby enhancing the model's ability and improving the accuracy of subsequent abnormal analysis information determination, and further improving the accuracy and effectiveness of UAV multi-sensor system anomaly detection.

[0142] S103: Input the UAV flight data into the UAV anomaly analysis model to obtain the anomaly analysis information corresponding to the anomaly analysis task.

[0143] In a specific embodiment, the UAV anomaly analysis model can be used to perform anomaly analysis on UAV flight data; the UAV anomaly analysis model can include a multi-modal feature extraction network, a known anomaly recognition module, and an unknown anomaly detection module; the multi-modal feature extraction network can be used for feature extraction, and the multi-modal feature extraction network can be obtained by adjusting the parameters of the initial multi-modal feature extraction network based on the triplet margin total loss, and the triplet margin total loss is determined based on the triplet set in combination with the triplet loss function; the known anomaly recognition module can be used to recognize known anomaly types; the unknown anomaly detection module can be used to detect unknown anomalies; the unknown anomaly detection module can, in the case of detecting an unknown anomaly, be combined with a preset unknown anomaly recognition algorithm to identify the unknown anomaly type.

[0144] The anomaly analysis information can characterize the anomaly situation of the UAV multi-sensor system detected by using the UAV anomaly analysis model; optionally, the anomaly analysis information can include known anomaly analysis information and unknown anomaly analysis information, the known anomaly analysis information can characterize whether there is a known anomaly and the type of the known anomaly, and the unknown anomaly analysis information can characterize whether there is an unknown anomaly and the type of the unknown anomaly.

[0145] In an alternative embodiment, Figure 3 is a schematic flow diagram of an embodiment of the present invention for inputting UAV flight data into the UAV anomaly analysis model to obtain anomaly analysis information corresponding to the anomaly analysis task; as Figure 3 shown, the above-mentioned inputting of UAV flight data into the UAV anomaly analysis model to obtain anomaly analysis information corresponding to the anomaly analysis task can include:

[0146] S301: In the case where the anomaly analysis task is a known anomaly analysis task and an unknown anomaly analysis task, input the UAV flight data into the multi-modal feature extraction network for feature extraction to obtain flight multi-modal feature data;

[0147] S302: Based on the known anomaly recognition module, perform known anomaly recognition processing on the flight multi-modal feature data to obtain known anomaly type information of the flight multi-modal feature data;

[0148] S303: Based on the unknown anomaly detection module, perform unknown anomaly detection processing on the flight multi-modal feature data to obtain unknown anomaly detection information of the flight multi-modal feature data;

[0149] S304: Based on the unknown anomaly detection information and the preset unknown anomaly recognition algorithm, perform unknown anomaly recognition processing on the unknown anomaly detection information to obtain unknown anomaly type information of the flight multi-modal feature data;

[0150] S305: Determine the anomaly analysis information for the anomaly analysis task based on the known anomaly type information and the unknown anomaly type information.

[0151] In a specific embodiment, the anomaly analysis tasks of known anomaly analysis task and unknown anomaly analysis task can characterize the user's anomaly detection requirements for the drone multi-sensor system, that is, the user wants to determine the known anomalies and unknown anomalies of the drone multi-sensor system.

[0152] In an alternative embodiment, the above-mentioned multi-modal feature extraction network includes a temporal feature extraction module, an attitude feature extraction module, and a feature fusion module;

[0153] Figure 4 is a schematic flowchart of determining flight multi-modal feature data provided by an embodiment of the present invention; as Figure 4 shown, the above-mentioned inputting the drone flight data into the multi-modal feature extraction network for feature extraction to obtain flight multi-modal feature data may include:

[0154] S401: Input the temporal data into the temporal feature extraction network for temporal feature extraction to obtain temporal features;

[0155] S402: Input the attitude data into the attitude feature extraction network for attitude feature extraction to obtain attitude features;

[0156] S403: Input the temporal features and the attitude features into the feature fusion network for feature fusion processing to obtain flight multi-modal feature data.

[0157] In a specific embodiment, the multi-modal feature extraction network can be a deep learning network for performing multi-modal feature extraction. The temporal data can be at least one temporal data, and the attitude data can be at least one attitude data. The temporal feature extraction network can be used to extract features from the temporal data of the drone flight data; optionally, the network structure of the temporal feature extraction network can be set according to the actual application requirements. Specifically, Figure 5 is a schematic diagram of a temporal feature extraction network and an attitude feature extraction network provided by an embodiment of the present invention; as Figure 5 shown, the temporal feature extraction network can include a fully connected layer. Optionally, the above-mentioned inputting the temporal data into the temporal feature extraction network for temporal feature extraction to obtain temporal features may include:

[0158] Input the temporal data into the temporal fully connected layer for temporal feature extraction to obtain temporal features, as shown in the following formula:

[0159] ;

[0160] where represents the The output features of the time series fully connected layer corresponding to the reconstructed UAV flight data, i.e., time series features, represents the preset window length, represents the time series feature extraction network; represents the th time series data corresponding to the reconstructed UAV flight data, represents the number of time series data; represents the weight matrix of the time series feature extraction network; represents the bias vector of the time series feature extraction network; represents the input dimension of the fully connected layer (of the time series feature extraction network); represents the output dimension of the fully connected layer (of the time series feature extraction network).

[0161] In a specific embodiment, the attitude feature extraction network can be used to extract features from the attitude data of the UAV flight data; optionally, the network structure of the attitude feature extraction network can be set according to the actual application requirements. Specifically, as Figure 5 shown, the attitude feature extraction network can include an attitude fully connected layer, multiple bottleneck layers, and a reshaping layer. Optionally, the above-mentioned inputting the attitude data into the attitude feature extraction network for attitude feature extraction to obtain attitude features can include:

[0162] Input the attitude data into the attitude fully connected layer to obtain the output features of the fully connected layer, as shown in the following formula:

[0163] ;

[0164] where, represents the output features of the attitude fully connected layer corresponding to the th reconstructed UAV flight data, represents the intermediate layer dimension of the attitude feature extraction network ATE; represents the th attitude data corresponding to the reconstructed UAV flight data, represents the number of attitude data; represents the weight matrix of the fully connected layer FC, represents the input dimension of the fully connected layer; represents the bias vector of the fully connected layer;

[0165] Input the output features of the fully connected layer into multiple bottleneck layers for feature extraction to obtain the output features of the bottleneck layers;

[0166] Input the output features of the bottleneck layers into the reshaping layer to obtain the attitude features.

[0167] Optionally, the bottleneck layer may include a dilation layer, a depth convolution layer, a projection layer, and a residual connection layer;

[0168] Inputting the output features of the fully connected layer into multiple bottleneck layers for feature extraction to obtain the output features of the total bottleneck layer may include:

[0169] Inputting the output features of the fully connected layer into the dilation layer for dimensional expansion to obtain the output features of the dilation layer; wherein, the dilation layer includes a pointwise convolution layer, a batch normalization layer, and an activation layer, the convolution kernel of the pointwise convolution layer is 1*1, the input dimension is , and the output dimension is , is the expansion multiple. Inputting the output features of the fully connected layer into the dilation layer for dimensional expansion to obtain the output features of the dilation layer can be shown as the following formula:

[0170] ;

[0171] wherein, represents the output features of the dilation layer in the th bottleneck layer; represents the convolution operation; represents the convolution kernel of the dilation layer in the th bottleneck layer; represents the bias vector of the dilation layer in the th bottleneck layer; represents batch normalization; represents the activation function;

[0172] Inputting the output features of the dilation layer into the depth convolution layer for feature extraction to obtain the output features of the depth convolution layer; wherein, the depth convolution layer includes depth convolution, a batch normalization layer, and an activation layer, the convolution kernel of the depth convolution is 3*3, and both the input dimension and the output dimension are . Inputting the output features of the dilation layer into the depth convolution layer for feature extraction to obtain the output features of the depth convolution layer can be shown as the following formula:

[0173] ;

[0174] wherein, represents the output features of the depth convolution layer in the th bottleneck layer; represents the convolution kernel of the depth convolution layer in the th bottleneck layer; represents the bias vector of the depth convolution layer in the th bottleneck layer;

[0175] The output features of the depth convolution layer are input into the projection layer for dimensionality compression to obtain the output features of the projection layer; among them, the projection layer includes a pointwise convolution layer and a batch normalization layer, and the convolution kernel of the pointwise convolution layer is 1*1, and the input dimension is and the output dimension is ,

[0176] The above-mentioned input of the output features of the depth convolution layer into the projection layer for dimensionality compression to obtain the output features of the projection layer can be shown as follows;

[0177] ;

[0178] Among them, represents the output features of the projection layer in the th bottleneck layer; represents the convolution kernel of the projection layer in the th bottleneck layer; represents the bias vector of the projection layer in the th bottleneck layer;

[0179] Using residual connection, add the input data of the bottleneck layer and the output features of the projection layer to obtain the output features of the bottleneck layer as shown in the following formula:

[0180] ;

[0181] Among them, represents the output features of the th bottleneck layer, represents the output of the previous bottleneck layer (i.e., the input of the current bottleneck layer). When , that is, when the current bottleneck layer is the first bottleneck layer in the pose feature extraction network, its output will be added to the output of the fully connected layer of the pose feature extraction network;

[0182] Send the output features of the th bottleneck layer into the th bottleneck layer, input the output features of the th bottleneck layer into the dilation layer for dimensionality expansion to obtain the output features of the dilation layer, repeat the above-mentioned input of the output features of the dilation layer into the depth convolution layer for feature extraction to obtain the output features of the depth convolution layer to using residual connection, add the input data of the bottleneck layer and the output features of the projection layer to obtain the output features of the bottleneck layer, and obtain the output features of the th bottleneck layer;

[0183] Repeat the above-mentioned step of inputting the output features of the bottleneck layer into the bottleneck layer for feature extraction until multiple bottleneck layers are traversed, and obtain the output features of the last bottleneck layer (total bottleneck layer output features) , Represents the number of multiple bottleneck layers.

[0184] Optionally, a reshaping layer can be used to adjust the dimension of the output features of the total bottleneck layer and obtain the output of the pose feature extraction network, that is, the pose feature , and the output dimensions of the temporal feature extraction network and the pose feature extraction network are set to be the same, that is .

[0185] In a specific embodiment, the feature fusion network can be used to fuse temporal data and pose data; optionally, the network structure of the feature fusion network can be set according to actual application requirements. Specifically, the feature fusion network can include a Transformer layer, and the Transformer layer can include an encoding layer and multiple Transformer encoders. Optionally, inputting the temporal feature and the pose feature into the feature fusion network for feature fusion processing to obtain flight multi-modal feature data may include:

[0186] Concatenate the temporal feature and the pose feature as follows to obtain the concatenated output feature:

[0187] ;

[0188] Among them, represents the tensor obtained after concatenation, that is, the concatenated output feature, represents the temporal feature The th element in , represents the pose feature The th element in , and finally use to represent The elements in ; represents the input dimension of the Transformer layer, where ;

[0189] Send the concatenated output feature into the encoding layer for information encoding, as shown in the following formula:

[0190] ;

[0191] ;

[0192] Among them, , can represent the dimension index of the position information after information encoding, Can be used to ensure that adjacent dimensions alternately use sine and cosine functions; Represents a feature vector; Represents being located at The feature vector of, and there is ; Represents the th position information located at ;

[0193] Add the feature vector located at and the position information and position information to obtain a new feature vector after position encoding, as shown in the following formula:

[0194] ;

[0195] Combine all the after position encoding to obtain the output of the encoding layer, and there is ;

[0196] Send the output of the encoding layer into the Transformer encoder; assume there are Transformer encoders;

[0197] Optionally, Figure 6 is a schematic diagram of the process of running a Transformer encoder provided by an embodiment of the present invention; as Figure 6 shown, use the output of the encoding layer as the input of the Transformer encoder; and the Transformer encoder may include a multi-head attention module, a batch normalization module, a feed-forward module, and a residual connection. The multi-head attention module has parallel self-attention heads. The running process of the Transformer encoder is as follows:

[0198] Split the input of the Transformer encoder into groups, denoted as , represents the input of the th Transformer encoder;

[0199] Send the split input into self-attention heads respectively. The running process of the above self-attention heads can be as follows:

[0200] Use the input of the self-attention head with the parameter matrix , and The linear transformation is , and , can represent the first parameter matrix used for the linear transformation of the input . can represent the second parameter matrix used for the linear transformation of the input . can represent the third parameter matrix used for the linear transformation of the input ; can represent the first eigenvector obtained after the linear transformation of the input using the first parameter matrix, can represent the second eigenvector obtained after the linear transformation of the input using the second parameter matrix, can represent the third eigenvector obtained after the linear transformation of the input using the third parameter matrix;

[0201] Determine the output of the self-attention head using the attention mechanism , as shown in the following formula:

[0202] ;

[0203] ;

[0204] where represents the dimension corresponding to the input , that is, the th dimension. Also, since the input of the self-attention head is split from the input of the multi-head attention mechanism, so there is ; represents the output of the th self-attention head; can respectively represent the query, key, and value in the self-attention mechanism, which can be used to implement the matching, screening, and aggregation of the self-attention mechanism, and can dynamically focus on the key information of the input sequence.

[0205] Repeat the steps to linearly transform the input of the self-attention head using the parameter matrices , and into , and until the output , until the outputs of all self-attention heads are computed;

[0206] Combine the outputs of all self-attention heads and perform a linear transformation using the parameter matrix to obtain the output of the multi-head attention mechanism, denoted here by as the input to the multi-head attention mechanism, as shown in the following equation:

[0207] ;

[0208] Perform batch normalization and residual connection operations on the output of the multi-head attention mechanism, as shown in the following equation:

[0209] ;

[0210] Feed the output of the upper layer into the feed-forward module, which consists of two fully-connected layers and is activated using the ReLU function. The calculation of the feed-forward module is shown in the following equation:

[0211] ;

[0212] where represents the output of the feed-forward module; and represent the weight matrix and bias vector of the first fully-connected layer respectively, while and represent the weight matrix and bias vector of the second fully-connected layer respectively, and there are and , represents the input dimension and output dimension of the feed-forward module;

[0213] Perform batch normalization and residual connection operations on the output of the feed-forward module and obtain the output of the Transformer encoder, as shown in the following equation:

[0214] ;

[0215] where represents the output of the th Transformer encoder; optionally, the input of the th Transformer encoder can be ; Feed into the next Transformer encoder and repeat the step of splitting the input of the Transformer encoder into groups, denoted as until batch normalization and residual connection operations are performed on the output of the feed-forward module, and the output of the Transformer encoder is obtained. Until all the Transformer encoders are traversed, the output of the last, i.e., the th Transformer encoder is , which is also the output of the entire Transformer layer; among them, in the figure represents the output of the 1st Transformer encoder, represents the output of the 2nd Transformer encoder, represents the number of Transformer encoders;

[0216] Keep the first eigenvector in, i.e., , and discard the remaining eigenvectors, which is the output of the Transformer layer and can be flight multi-modal feature data; specifically, a post-processing is added to make adjustments; since in the finally output , each eigenvector has received the extraction information from all the inputs, so only the first eigenvector in is retained in the post-processing, and the remaining eigenvectors are discarded.

[0217] In the above embodiments, the multi-modal feature extraction network can effectively handle multi-modal heterogeneous inputs, and can synchronously extract features of different modalities of the UAV flight data collected by the multi-sensor system; while fully extracting the temporal features and attitude features in the UAV flight data, the multi-modal feature extraction network also explores the relationships between multi-modal heterogeneous features, and can map the features in the high-dimensional space to the low-dimensional space, which is convenient for subsequent anomaly detection and anomaly type recognition. By extracting features from the UAV flight data through the multi-modal feature extraction network, the relationships between different modal features can be effectively balanced, and different modal features can be fused, thereby realizing the efficient extraction of multi-modal heterogeneous features, as well as improving the accuracy and effectiveness of multi-modal heterogeneous feature extraction, and further improving the accuracy of subsequent known anomaly type recognition and unknown anomaly type recognition.

[0218] In an alternative embodiment, Figure 7 is a schematic flow chart of constructing a multi-modal feature extraction network provided by an embodiment of the present invention; as Figure 7 shown, the above multi-modal feature extraction network can be constructed in the following manner:

[0219] S701: Train an initial multi-modal feature extraction network including an initial timing feature extraction network, an initial pose feature extraction network, and an initial feature fusion network to obtain an initial multi-modal feature extraction network;

[0220] S702: Obtain multi-modal flight sample data;

[0221] S703: Input the multi-modal flight sample data into the initial multi-modal feature extraction network for feature extraction processing to obtain flight sample data features;

[0222] S704: Perform triplet extraction processing on the flight sample data features to obtain a triplet set;

[0223] S705: Based on the triplet set, combine with a triplet loss function to determine the total triplet margin loss;

[0224] S706: Based on the total triplet margin loss, adjust the parameters of the initial multi-modal feature extraction network to obtain a multi-modal feature extraction network.

[0225] In a specific embodiment, the multi-modal flight sample data can be multi-modal flight data collected by a drone multi-sensor system; optionally, a preset abnormal type label corresponding to the multi-modal flight sample data is obtained while obtaining the multi-modal flight sample data. The flight sample data features can be features containing multiple flight data features obtained by performing feature extraction on the multi-modal flight sample data. The initial multi-modal feature extraction network can be a preliminarily trained feature extraction network, and the multi-modal feature extraction network can be a feature extraction network that has been adjusted for the parameters of the initial multi-modal feature extraction network and has been trained. Using the initial multi-modal feature extraction network to perform feature extraction on the flight sample data can map the flight sample data to a low-dimensional space to obtain flight sample data features.

[0226] In an alternative embodiment, Figure 8 is a schematic flowchart of a process for performing triplet extraction processing on flight sample data features to obtain a triplet set provided by an embodiment of the present invention; as Figure 8 shown, the above-mentioned triplet extraction processing on the flight sample data features to obtain a triplet set may include:

[0227] S801: Traverse each flight data feature in the flight sample data features;

[0228] When any flight data feature is traversed, determine the flight data feature as the current anchor data;

[0229] S802: Determine multiple positive flight data with the same abnormal type as the current anchor data in the flight sample data features;

[0230] S803: Determine the maximum positive distance based on the distances between the current anchor data and each positive flight data feature.

[0231] S804: Determine multiple negative flight data with different anomaly types from the current anchor data among the flight sample data features.

[0232] S805: Determine the minimum negative distance based on the distances between the current anchor data and each negative flight data.

[0233] S806: Determine the positive flight data for which the first distance from the current anchor data is greater than the first target distance as the current positive flight data; the first target distance is the sum of the minimum negative distance and a preset distance.

[0234] S807: Determine the negative flight data for which the second distance from the current anchor data is less than the second target distance as the current negative flight data; the second target distance is the difference between the maximum positive distance and a preset distance.

[0235] S808: Use the current anchor data, the current positive flight data, and the current negative flight data as a triple.

[0236] S809: After traversing each flight data feature in the flight sample data features, use the obtained multiple triples as a triple set.

[0237] In a specific embodiment, among the flight sample data features, select one flight data feature as the current anchor data. The positive flight data can be flight data features with the same anomaly type as the current anchor data; optionally, the current anchor data and the positive flight data can form a positive pair; optionally, based on the preset anomaly type labels of the multimodal flight sample data, determine multiple positive flight data with the same anomaly type as the current anchor data among the flight sample data features. The negative flight data can be flight data features with different anomaly types from the current anchor data; optionally, the current anchor data and the negative flight data can form a negative pair; optionally, based on the preset anomaly type labels of the multimodal flight sample data, determine multiple negative flight data with different anomaly types from the current anchor data among the flight sample data features.

[0238] Optionally, determining the maximum positive distance based on the distance between the current anchor point data and each positive flight data may include: determining the Euclidean distance between the current anchor point data and each positive flight data to obtain a plurality of first distances, where the first distance may represent the within-class distance; sorting the plurality of first distances in descending order to obtain a descending order result of positive distances; using the first distance ranked first in the descending order result of positive distances as the maximum positive distance. Determining the minimum negative distance based on the distance between the current anchor point data and each negative flight data may include: determining the Euclidean distance between the current anchor point data and each negative flight data to obtain a plurality of second distances, where the second distance may represent the between-class distance; sorting the plurality of second distances in ascending order to obtain an ascending order result of negative distances; using the second distance ranked first in the ascending order result of negative distances as the minimum negative distance.

[0239] Optionally, the preset distance may be the margin distance. Specifically, the preset distance may be set in combination with the actual application. The first distance may be the distance between the positive flight data and the current anchor point data, and the second distance may be the distance between the negative flight data and the current anchor point data. Optionally, determining the positive flight data with the first distance greater than the sum of the minimum negative distance and the margin distance as the current positive flight data, and determining the negative flight data with the second distance less than the difference between the maximum positive distance and the margin distance as the current negative flight data. At this time, the current anchor point data, the current positive flight data, and the current negative flight data may be used as a triple; while the positive flight data that does not satisfy the condition that the first distance is greater than the sum of the minimum negative distance and the margin distance and the negative flight data that does not satisfy the condition that the second distance is less than the difference between the maximum positive distance and the margin distance will not be selected as a triple. Optionally, after traversing each flight data feature in the flight sample data, all the obtained triples may be integrated into a triple set.

[0240] In a specific embodiment, Figure 9 is a schematic diagram of a triple provided by an embodiment of the present invention; as Figure 9 shown, may represent the anchor point data in the triple, may represent the positive flight data in the triple, may represent the negative flight data in the triple; may represent the within-class distance between the positive flight data and the anchor point data in the triple, may represent the between-class distance between the negative flight data and the anchor point data in the triple, and the within-class distance is much smaller than the between-class distance ; Optionally, in the process of further increasing the distance between the anchor point data and the negative flight data and reducing the distance between it and the positive flight data, a margin distance m may be added, and there is .

[0241] In a specific embodiment, determining the total triplet margin loss based on the triplet set in combination with the triplet loss function may include: selecting a triplet from the triplet set as the current triplet; determining the triplet margin loss corresponding to the current triplet based on the current triplet in combination with the triplet loss function; selecting the next triplet from the triplet set as the current triplet, and repeating the above step of determining the triplet margin loss corresponding to the current triplet based on the current triplet in combination with the triplet loss function until the triplet margin losses of all triplets in the triplet set are determined; adding up the triplet margin losses of all triplets to obtain the total triplet margin loss.

[0242] Optionally, the triplet loss function can be set in combination with the actual application. Specifically, the triplet loss function can be . Optionally, based on the total triplet margin loss, the parameters of the initial multi-modal feature extraction network can be optimized using the Adam (Adaptive Moment Estimation) optimizer to obtain the multi-modal feature extraction network.

[0243] In the above embodiment, optimizing the multi-modal feature extraction network through the total triplet margin loss can constrain the relative distances of different modal data in the feature space, ensure that flight data of the same abnormal type is highly aggregated to reduce classification ambiguity, and ensure that flight data of different abnormal types is clearly separated to reduce the misjudgment probability. Furthermore, it can improve the accuracy, robustness, and efficiency of subsequent anomaly detection and recognition.

[0244] In an alternative embodiment, the above-mentioned known anomaly recognition module performs known anomaly recognition processing on the flight multi-modal feature data to obtain the known anomaly type information of the flight multi-modal feature data, which may include:

[0245] Performing known anomaly recognition processing on the flight multi-modal feature data based on a preset known anomaly recognition algorithm to obtain the known anomaly type information of the flight multi-modal feature data, where the preset known anomaly recognition algorithm includes the nearest neighbor algorithm.

[0246] In a specific embodiment, the preset known anomaly recognition algorithm can be set in combination with the actual application. Specifically, the preset known anomaly recognition algorithm can be the K-nearest neighbor algorithm. Specifically, by using the K-nearest neighbor algorithm to classify the flight multi-modal feature data, the known anomaly type information corresponding to the UAV flight data can be obtained. Optionally, the flight multi-modal feature data can be the data obtained by mapping the complex and high-dimensional UAV flight data to a low-dimensional space by a multi-modal feature extraction network. The K-nearest neighbor algorithm is used to classify the flight multi-modal feature data in the low-dimensional space. Optionally, the known anomaly type information can represent the known anomaly data and the corresponding anomaly types of the UAV multi-sensor system, and the known anomaly type information can be the known anomaly data of the determined anomaly types existing in the UAV multi-sensor system. Optionally, while determining the known anomaly type information, normal information can also be determined; based on the preset known anomaly recognition algorithm, the flight multi-modal feature data classified into different known anomaly categories can obtain the known anomaly type information, and the flight multi-modal feature data classified into the normal category can obtain the normal information.

[0247] In a specific embodiment, the above-mentioned known anomaly recognition module can be constructed in the following manner:

[0248] Obtain the anomaly sample to be recognized and the preset sample recognition type of the anomaly sample to be recognized;

[0249] Input the anomaly sample to be recognized into the multi-modal feature extraction network for feature extraction to obtain the feature of the anomaly sample to be recognized;

[0250] Based on the preset known anomaly detection algorithm, perform recognition processing on the feature of the anomaly sample to be recognized to obtain the predicted sample recognition type of the anomaly sample to be recognized;

[0251] Based on the preset sample recognition type and the predicted sample recognition type, adjust the parameters of the known anomaly recognition module to be constructed;

[0252] Repeat the above steps until the first preset condition is reached to determine the known anomaly recognition module.

[0253] In a specific embodiment, before obtaining the anomaly sample to be recognized, it may include: obtaining the original anomaly sample to be recognized, where the original anomaly sample to be recognized includes multiple state features, and each state feature includes multiple state data; performing quantity alignment processing on the state data of the multiple state features to obtain multiple aligned state features, and each aligned state feature includes multiple aligned state data; performing data standardization processing on the multiple aligned state data of each aligned state feature to obtain multiple standardized state features, and each standardized state feature includes multiple standardized state data; combining the multiple standardized state data of each standardized state feature with a sliding window for data reconstruction processing to determine the anomaly sample to be recognized.

[0254] In a specific embodiment, the refinement before obtaining the abnormal samples to be recognized can refer to the relevant refinement before obtaining the UAV flight data collected by the UAV multi-sensor system based on the abnormal analysis task as described above, which will not be elaborated here. During the construction process of the abnormal samples to be recognized herein, starting from the first timestamp of the sliding window and sliding on the time axis; before each sliding, extract the standardized state features corresponding to the timestamps of the preset window length from all the standardized state features within the sliding window, and combine them to obtain the reconstructed state features. Each reconstructed state feature may include multiple reconstructed state data; based on the multiple reconstructed state features, obtain the reconstructed abnormal samples to be recognized; use the abnormal type corresponding to the last timestamp as the preset sample recognition type label of the reconstructed abnormal samples to be recognized; the sliding window will slide backward along the time axis by a preset sliding step of timestamps, and complete the construction of the next reconstructed abnormal samples to be recognized. The preset sample recognition type label of the next reconstructed abnormal samples to be recognized is also the abnormal type at the last timestamp within the sliding window. Repeat the above process that the sliding window will slide backward along the time axis by a preset sliding step of timestamps, and complete the construction of the next reconstructed abnormal samples to be recognized until the sliding window traverses all timestamps, obtaining multiple reconstructed abnormal samples to be recognized. Each reconstructed abnormal samples to be recognized includes multiple reconstructed state features, and each reconstructed abnormal samples to be recognized corresponds to a preset sample recognition type label; based on the multiple reconstructed abnormal samples to be recognized and the corresponding preset sample recognition type labels, form the abnormal samples to be recognized and the preset sample recognition type of the abnormal samples to be recognized. Specifically, the number of the reconstructed abnormal samples to be recognized in the abnormal samples to be recognized can be shown by the following formula: ; where represents the number of the reconstructed abnormal samples to be recognized; represents the preset window length; represents the preset sliding step; represents the number of timestamps, represents rounding up. The reconstructed abnormal samples to be recognized can expand the sample size so as to improve the accuracy of known abnormal recognition.

[0255] In a specific embodiment, the first preset condition can be set in combination with the actual application. Optionally, the first preset condition can be that the number of iterations reaches the preset number, or all the reconstructed abnormal samples to be recognized in the abnormal samples to be recognized are traversed. Optionally, during the construction process of the known abnormal recognition module, the features of the abnormal samples to be recognized for recognition processing will be pre-stored.

[0256] In the above embodiments, the method based on deep metric learning does not rely on a pre-designed model structure, and can distinguish different types of data based on the distance between data, improving the efficiency and accuracy of identifying known anomaly types; specifically, using the metric learning algorithm K-NN (K-nearest neighbor) algorithm can achieve accurate classification of known anomalies in the UAV multi-sensor system, thereby improving the accuracy and efficiency of identifying known anomaly types.

[0257] In a specific embodiment, the anomaly analysis task in the above embodiments is a known anomaly analysis task among the known anomaly analysis task and the unknown anomaly analysis task; and in the case where the anomaly analysis task is a known anomaly analysis task, in addition to using the above-mentioned preset known anomaly recognition algorithm to perform known anomaly recognition processing on the flight multi-modal feature data to obtain the known anomaly type information of the flight multi-modal feature data, a classification model based on deep learning in the known anomaly recognition module can also be used. Specifically, the flight multi-modal feature data is input into the classification model for classification processing to obtain the predicted known anomaly type information of the flight multi-modal feature data.

[0258] In a specific embodiment, the classification model can be trained in the following manner:

[0259] Obtain the anomaly sample to be recognized and the preset sample recognition type of the anomaly sample to be recognized;

[0260] Input the anomaly sample to be recognized into the multi-modal feature extraction network for feature extraction to obtain the feature of the anomaly sample to be recognized;

[0261] Input the feature of the anomaly sample to be recognized into the classification model to be trained for classification processing to obtain the predicted sample recognition type;

[0262] Based on the preset sample recognition type and the predicted sample recognition type, train the classification model to be trained to obtain the classification model.

[0263] In the above embodiments, different processing is performed based on different anomaly analysis tasks, which can improve the efficiency and accuracy of processing the anomaly analysis tasks.

[0264] In an alternative embodiment, Figure 10 is a schematic flowchart of a process for performing unknown anomaly detection processing on flight multi-modal feature data by an unknown anomaly detection module provided in an embodiment of the present invention to obtain unknown anomaly detection information of the flight multi-modal feature data; as Figure 10 shown, the above-mentioned performing unknown anomaly detection processing on flight multi-modal feature data by the unknown anomaly detection module to obtain unknown anomaly detection information of the flight multi-modal feature data may include:

[0265] S1001: Determine multiple preset known abnormal feature data and flight multi-modal feature data as mixed feature data;

[0266] S1002: Perform clustering processing on the mixed feature data based on a preset clustering algorithm to obtain a set of clustering feature data clusters;

[0267] S1003: Traverse each clustering feature data cluster in the set of clustering feature data clusters;

[0268] In the case of traversing any clustering feature data cluster, determine that there is no preset known abnormal feature data in the cluster, and the feature data in the current clustering feature data cluster where the number of feature data in the cluster is greater than or equal to a preset number is unknown abnormal data;

[0269] S1004: Determine unknown abnormal detection information of the flight multi-modal feature data based on the unknown abnormal data.

[0270] In a specific embodiment, the preset known abnormal feature data may be feature data obtained by performing feature extraction on preset known abnormal data with a determined abnormal type; specifically, preset known abnormal data can be obtained, and the preset known abnormal data can be input into a multi-modal feature extraction network for feature extraction to obtain the preset known abnormal feature data. Optionally, the preset known abnormal feature data may include multi-modal feature data marked with an abnormal type based on a known abnormal type. Optionally, the preset clustering algorithm can be set in combination with the actual application. Specifically, the preset clustering algorithm can be the K-means algorithm. Optionally, the set of clustering feature data clusters may include multiple clustering feature data clusters. Optionally, the preset number can be set in combination with the actual application, and the preset number can be used to exclude the probability of isolated data (only one data in a cluster) occurring. The unknown abnormal data may be data with an unclear abnormal type; the unknown abnormal detection information may characterize whether there is unknown abnormality in the flight multi-modal feature data and data without a clear abnormal type.

[0271] In a specific embodiment, the K-means algorithm can be used to cluster the mixed feature data to obtain multiple clusters of feature data, and the sum of squared errors within the clusters is minimized; select one cluster of feature data from the multiple clusters of feature data as the current cluster of feature data; determine whether there is preset known abnormal feature data in the current cluster of feature data, and determine whether the number of feature data in the current cluster of feature data is greater than or equal to a preset number; in the case that the current cluster of feature data satisfies the conditions of not having preset known abnormal feature data and the number of feature data being greater than or equal to the preset number, determine the current cluster of feature data as unknown abnormal data; select the next cluster of feature data from the multiple clusters of feature data, and repeat the above steps of determining whether there is preset known abnormal feature data in the current cluster of feature data and determining whether the number of feature data in the current cluster of feature data is greater than or equal to the preset number; in the case that the current cluster of feature data satisfies the conditions of not having preset known abnormal feature data and the number of feature data being greater than or equal to the preset number, determine the current cluster of feature data as unknown abnormal data until all the multiple clusters of feature data are traversed; based on the determined multiple pieces of unknown abnormal data, the unknown abnormal detection information of the flight multi-modal feature data can be determined.

[0272] In a specific embodiment, in the case that the abnormal analysis task is an unknown abnormal analysis task, after obtaining the flight multi-modal feature data, based on the unknown abnormal detection module, the unknown abnormal detection processing can be performed on the flight multi-modal feature data to obtain the unknown abnormal detection information. By performing different task processing procedures for different abnormal analysis tasks, the efficiency and accuracy of the abnormal analysis task processing can be improved, and resource waste can be reduced.

[0273] In a specific embodiment, the above-mentioned unknown abnormal detection module can be constructed in the following manner:

[0274] Obtain a mixed sample set, where the mixed sample set includes a known abnormal sample set and an unknown abnormal sample set;

[0275] Input the mixed sample set into a multi-modal feature extraction network for feature extraction processing to obtain mixed multi-modal feature data;

[0276] Based on a preset clustering algorithm, perform clustering processing on the mixed multi-modal feature data to obtain a set of mixed feature data clusters;

[0277] Traverse each mixed feature data cluster in the set of mixed feature data clusters;

[0278] In the case of traversing any mixed feature data cluster, determine that there are known abnormal samples in the cluster, and determine the mixed multi-modal feature data in the current mixed feature data cluster where the number of samples in the cluster is greater than or equal to the preset number as unknown abnormal data;

[0279] Based on the unknown abnormal data, determine the sample unknown abnormal detection information of the mixed sample set;

[0280] Based on the unknown abnormal sample set and the sample unknown abnormal detection information, adjust the parameters of the to-be-constructed unknown abnormal detection module;

[0281] Repeat the above steps until the second preset condition is met, and determine the unknown abnormal detection module.

[0282] In a specific embodiment, the refinement before obtaining the mixed sample set can refer to the relevant refinement before obtaining the abnormal samples to be identified as described above, which will not be elaborated here. Reconstructing the mixed sample set can increase the sample size so as to improve the accuracy of unknown abnormal detection. The second preset condition can be set in combination with the actual application. Optionally, the second preset condition can be that the number of iterations reaches a preset number, or all the mixed sample sets are traversed. Optionally, during the process of constructing the unknown abnormal detection module, the abnormal samples to be identified for detection processing can be pre-stored.

[0283] In the above embodiment, deep metric learning can also handle the open set problem, and unknown abnormal detection can be realized based on deep metric learning, thereby effectively discovering potential unknown abnormalities in the UAV flight data.

[0284] In an alternative embodiment, Figure 11 is a schematic flow chart of a method for performing unknown abnormal identification processing on unknown abnormal detection information based on unknown abnormal detection information and a preset unknown abnormal identification algorithm to obtain unknown abnormal type information of flight multi-modal feature data; as Figure 11 shown, the above method for performing unknown abnormal identification processing on unknown abnormal detection information based on unknown abnormal detection information and a preset unknown abnormal identification algorithm to obtain unknown abnormal type information of flight multi-modal feature data may include:

[0285] S1101: Obtain a training set, a test set, and a support set;

[0286] S1102: Determine the support set and the training set as a mixed data set;

[0287] S1103: Input the mixed data set into a multi-modal feature extraction network for feature extraction to obtain a mixed data feature set;

[0288] S1104: Input the test set into a multi-modal feature extraction network for feature extraction to obtain a test data feature set;

[0289] S1105: Traverse each test data feature in the test data feature set;

[0290] When traversing any test data feature, determine the vector distance between the current test data feature and each mixed data feature;

[0291] S1106: Determine the anomaly type label of the mixed data feature corresponding to the minimum distance among the multiple vector distances as the anomaly type label of the current test data feature;

[0292] S1107: Based on the anomaly type labels of each test data feature, determine the unknown anomaly type information.

[0293] In a specific embodiment, the support set may include a preset number of unknown anomaly type data, and the preset number of unknown anomaly type data is obtained by performing unknown anomaly type marking processing on a preset number of first unknown anomaly data, and the preset number of first unknown anomaly data is obtained from the unknown anomaly detection information; optionally, a small number of unknown anomaly data with unknown anomaly types may be marked to determine the anomaly types of the small number of unknown anomaly data, and then the support set is formed, that is, the preset number of unknown anomaly type data can represent the preset number of unknown anomaly data and the corresponding anomaly types. Optionally, the marking processing may include manual marking. The test set may include a known anomaly type data set and an unknown anomaly data set. The unknown anomaly data set may include multiple second unknown anomaly data. The preset number of first unknown anomaly data and the multiple second unknown anomaly data may form the unknown anomaly detection information; optionally, the test set does not include the data in the support set; optionally, the test set may include the known anomaly type information identified by the above-mentioned known anomaly recognition module and the unknown anomaly detection information detected by the above-mentioned unknown anomaly detection module, and the unknown anomaly detection information does not include the preset number of first unknown anomaly data that make up the support set. The training set may include known anomaly type data samples; optionally, the training set combined with the support set may be used to update the unknown anomaly detection module, and then unknown anomaly recognition is achieved to determine the unknown anomaly type. Optionally, the anomaly type label of the mixed data feature may be obtained based on the anomaly types determined from the known anomaly type data samples and the preset number of first unknown anomaly type data. Optionally, the unknown anomaly type information may represent the unknown anomaly data of the UAV multi-sensor system and the corresponding anomaly types, and the unknown anomaly type information may be the unknown anomaly data with determined anomaly types existing in the UAV multi-sensor system.

[0294] In a specific embodiment, during the process of determining the anomaly type label of the mixed data feature corresponding to the minimum distance among the multiple vector distances as the anomaly type label of the current test data feature, the anomaly type label of the known anomaly type data may also be determined. Therefore, the known anomaly type information can also be determined in this process, and fine-grained distinction between known anomalies and unknown anomalies can be achieved simultaneously.

[0295] In the above embodiments, fine-grained recognition of unknown anomalies can be achieved through online update based on online learning and few-shot learning, the information of unknown anomaly types can be determined, the transformation from unknown anomaly detection to unknown anomaly recognition can be completed, and the computing cost can be reduced and resource waste can be reduced.

[0296] In a specific embodiment, when the information of known anomaly types and unknown anomaly types of the UAV multi-sensor system is determined, the anomalies existing in the UAV multi-sensor system can be determined in a timely manner, and the UAV multi-sensor system can be repaired in a timely manner for this anomaly, so as to improve the safety and accuracy of the use of the UAV multi-sensor system.

[0297] As can be seen from the technical solutions provided in the embodiments of this specification above, this specification can determine the needs of users for anomaly detection of the UAV multi-sensor system by receiving the anomaly analysis tasks sent by users; then based on the anomaly analysis tasks, obtain the UAV flight data collected by the UAV multi-sensor system, input the UAV flight data into the UAV anomaly analysis model, and obtain the anomaly analysis information corresponding to the anomaly analysis tasks, and the anomalies existing in the UAV multi-sensor system and the corresponding anomaly types can be determined based on the user needs, improving the accuracy and efficiency of anomaly detection of the UAV multi-sensor system, and reducing resource waste;

[0298] The UAV anomaly analysis model includes a multi-modal feature extraction network, a known anomaly recognition module, and an unknown anomaly recognition module. The multi-modal feature extraction network is used for feature extraction, and the multi-modal feature extraction network is obtained by adjusting the parameters of the initial multi-modal feature extraction network based on the triplet margin total loss, which can effectively balance the relationship between different modal features, reduce the distance between the same type of features, increase the distance between different types of features, and thus improve the accuracy and efficiency of known anomaly recognition and unknown anomaly recognition; the known anomaly recognition module is used to recognize known anomaly types, the unknown anomaly detection module is used to detect unknown anomalies, and when the unknown anomaly detection module detects an unknown anomaly, it combines a preset unknown anomaly recognition algorithm to identify known anomaly types, which can realize the discovery of potential unknown anomalies in the UAV multi-sensor system, and realize the recognition of known anomaly types and unknown anomaly types in the UAV multi-sensor system, and improve the accuracy and efficiency of the recognition of known anomaly types and unknown anomaly types.

[0299] The embodiment of the present invention also provides an anomaly detection device for a UAV multi-sensor system. Correspondingly, Figure 12 is a schematic structural diagram of an anomaly detection device for a UAV multi-sensor system provided by the embodiment of the present invention; as Figure 12 shown, the above device includes:

[0300] A task receiving module 1210, configured to receive an anomaly analysis task sent by a user;

[0301] A data acquisition module 1220, configured to acquire drone flight data collected by a drone multi-sensor system based on the anomaly analysis task;

[0302] An anomaly analysis and processing module 1230, configured to input the drone flight data into a drone anomaly analysis model to obtain anomaly analysis information corresponding to the anomaly analysis task; wherein, the drone anomaly analysis model includes a multi-modal feature extraction network, a known anomaly recognition module, and an unknown anomaly detection module; the multi-modal feature extraction network is used for feature extraction, and the multi-modal feature extraction network is obtained by adjusting the parameters of an initial multi-modal feature extraction network based on a triplet margin total loss, and the triplet margin total loss is determined based on a triplet set in combination with a triplet loss function; the known anomaly recognition module is used to recognize known anomaly types; the unknown anomaly detection module is used to detect unknown anomalies; the unknown anomaly detection module, when detecting an unknown anomaly, is used to recognize the unknown anomaly type in combination with a preset unknown anomaly recognition algorithm.

[0303] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0304] An embodiment of the present invention further provides an electronic device, the electronic device includes: a processor and a memory, and at least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the anomaly detection method for a drone multi-sensor system as described in any one of the method embodiments.

[0305] An embodiment of the present invention further provides a computer storage medium, the computer storage medium can be set in a server to store at least one instruction, at least one program, a code set or an instruction set for implementing the method embodiment, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the anomaly detection method for a drone multi-sensor system as described in any one of the method embodiments.

[0306] Optionally, in an embodiment of the present invention, the above storage medium may be located in at least one of a plurality of network servers in a computer network. Optionally, in an embodiment of the present invention, the above storage medium may include, but is not limited to: various media that can store program codes such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), external hard drives, magnetic disks, or optical discs.

[0307] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0308] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one flow or multiple flowcharts and / or blocks Figure 1 one block or multiple blocks.

[0309] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one flow or multiple flowcharts and / or blocks Figure 1 one block or multiple blocks.

[0310] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in Figure 1 one flow or multiple flowcharts and / or blocks Figure 1 one block or multiple blocks.

[0311] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0312] Finally, it should be noted that the embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit of the present invention and the scope protected by the claims. All of these are within the protection scope of the present invention.

Claims

1. A method for detecting anomalies in a multi-sensor system of an unmanned aerial vehicle, characterized in that: The method comprises: Receive exception analysis tasks sent by users; Based on the anomaly analysis task, obtaining the UAV flight data collected by the UAV multi-sensor system; Inputting the UAV flight data into a UAV anomaly analysis model to obtain anomaly analysis information corresponding to the anomaly analysis task; Among them, the drone anomaly analysis model includes a multimodal feature extraction network, a known anomaly recognition module and an unknown anomaly detection module; the multimodal feature extraction network is used for feature extraction, and the multimodal feature extraction network is obtained by adjusting the parameters of the initial multimodal feature extraction network based on the triple marginal total loss, and the triple marginal total loss is based on the triple set and determined in combination with the triple loss function; the known anomaly recognition module is used to identify known anomaly types; the unknown anomaly detection module is used to detect unknown anomalies; when the unknown anomaly is detected, the unknown anomaly detection module is combined with a preset unknown anomaly recognition algorithm to identify the unknown anomaly type.

2. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 1, characterized in that: The anomaly analysis tasks include known anomaly analysis tasks and unknown anomaly analysis tasks; The step of inputting the UAV flight data into the UAV anomaly analysis model to obtain anomaly analysis information corresponding to the anomaly analysis task includes: In the case where the anomaly analysis task is a known anomaly analysis task and an unknown anomaly analysis task, the UAV flight data is input into the multimodal feature extraction network for feature extraction to obtain flight multimodal feature data; Based on the known anomaly recognition module, performing known anomaly recognition processing on the flight multimodal feature data to obtain known anomaly type information of the flight multimodal feature data; Based on the unknown anomaly detection module, performing unknown anomaly detection processing on the flight multimodal feature data to obtain unknown anomaly detection information of the flight multimodal feature data; Based on the unknown anomaly detection information and a preset unknown anomaly recognition algorithm, performing unknown anomaly recognition processing on the unknown anomaly detection information to obtain unknown anomaly type information of the flight multimodal feature data; Based on the known exception type information and the unknown exception type information, the exception analysis information of the exception analysis task is determined.

3. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 2, characterized in that: The multimodal feature extraction network includes a time series feature extraction network, a posture feature extraction network and a feature fusion network; the UAV flight data includes time series data and posture data; The step of inputting the UAV flight data into the multimodal feature extraction network for feature extraction to obtain flight multimodal feature data includes: Inputting the time series data into the time series feature extraction network to extract time series features to obtain time series features; Inputting the posture data into the posture feature extraction network to extract posture features and obtain posture features; The time series features and the posture features are input into a feature fusion network for feature fusion processing to obtain flight multimodal feature data.

4. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 1, characterized in that: The multimodal feature extraction network is constructed in the following way: Training an initial multimodal feature extraction network to be trained, including an initial temporal feature extraction network, an initial posture feature extraction network, and an initial feature fusion network, to obtain an initial multimodal feature extraction network; Obtain multi-modal flight sample data; Inputting the multimodal flight sample data into the initial multimodal feature extraction network for feature extraction processing to obtain flight sample data features; Performing triple extraction processing on the flight sample data features to obtain a triple set; Based on the triplet set, determining the triplet marginal total loss in combination with the triplet loss function; Based on the triplet marginal total loss, the parameters of the initial multimodal feature extraction network are adjusted to obtain a multimodal feature extraction network.

5. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 4, characterized in that: The triplet extraction process is performed on the flight sample data features to obtain a triplet set, including: Traversing each flight data feature in the flight sample data features; In case of traversing to any flight data feature, determining the flight data feature as current anchor point data; determining, in the flight sample data features, at least one positive flight data having the same anomaly type as the current anchor point data; determining a maximum positive distance based on the distance between the current anchor point data and each of the positive flight data; determining, in the flight sample data features, at least one negative flight data having a different anomaly type from the current anchor point data; Determining a minimum negative distance based on the distance between the current anchor point data and each of the negative flight data; Determine that the positive flight data corresponding to the first distance between the current anchor point data and the first target distance is the current positive flight data; the first target distance is the sum of the minimum negative distance and the preset distance; Determine that the negative flight data corresponding to the second distance between the current anchor point data and the second target distance is less than the current negative flight data; the second target distance is the difference between the maximum positive distance and the preset distance; taking the current anchor point data, the current positive flight data and the current negative flight data as a triplet; After traversing each flight data feature in the flight sample data features, at least one triplet is obtained as a triplet set.

6. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 2, characterized in that: The unknown anomaly detection module is used to perform unknown anomaly detection processing on the flight multimodal feature data to obtain unknown anomaly detection information of the flight multimodal feature data, including: Determining at least one preset known abnormal characteristic data and the flight multimodal characteristic data as mixed characteristic data; Performing clustering processing on the mixed feature data based on a preset clustering algorithm to obtain a cluster feature data cluster set; Traversing each clustering feature data cluster in the clustering feature data cluster set; In the case of traversing to any cluster feature data cluster, determining that there is no preset known abnormal feature data in the cluster, and that the feature data in the current cluster feature data cluster whose number of feature data in the cluster is greater than or equal to the preset number is unknown abnormal data; Based on the unknown abnormal data, unknown abnormality detection information of the flight multimodal feature data is determined.

7. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 2, characterized in that: The step of performing unknown abnormality recognition processing on the unknown abnormality detection information based on the unknown abnormality detection information and a preset unknown abnormality recognition algorithm to obtain unknown abnormality type information of the flight multimodal feature data includes: Acquire a training set, a test set and a support set; the support set includes a preset number of unknown abnormal type data, the preset number of unknown abnormal type data is obtained by performing unknown abnormal type labeling processing on a preset number of first unknown abnormal data, and the preset number of first unknown abnormal data is obtained from the unknown abnormal detection information; the test set includes a known abnormal type data set and an unknown abnormal data set, the unknown abnormal data set includes at least one second unknown abnormal data, and the preset number of first unknown abnormal data and the at least one second unknown abnormal data constitute the unknown abnormal detection information; the training set includes known abnormal type data samples; Determine the support set and the training set as a mixed data set; Inputting the mixed data set into a multimodal feature extraction network for feature extraction to obtain a mixed data feature set; Inputting the test set into a multimodal feature extraction network for feature extraction to obtain a test data feature set; Traversing each test data feature in the test data feature set; When any test data feature is traversed, the vector distance between the current test data feature and each mixed data feature is determined; Determine the abnormal type label of the mixed data feature corresponding to the minimum distance among the N vector distances as the abnormal type label of the current test data feature; Based on the abnormality type label of each test data feature, the unknown abnormality type information is determined.

8. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 2, characterized in that: The known anomaly identification module is used to perform known anomaly identification processing on the flight multimodal feature data to obtain known anomaly type information of the flight multimodal feature data, including: Based on a preset known anomaly recognition algorithm, known anomaly recognition processing is performed on the flight multimodal feature data to obtain known anomaly type information of the flight multimodal feature data, wherein the preset known anomaly recognition algorithm includes a nearest neighbor algorithm.

9. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 1, characterized in that: Before obtaining the UAV flight data collected by the UAV multi-sensor system based on the abnormality analysis task, the method includes: Based on the abnormal analysis task, the original UAV flight data collected by the UAV multi-sensor system is obtained, and the original UAV flight data includes flight status data, each flight status data includes at least one status collection data; Regarding the The flight status data is processed by the number alignment of the status collection data to obtain alignment flight status data, each alignment flight status data includes at least one alignment status acquisition data; Perform data standardization on at least one alignment state acquisition data of each alignment flight state data to obtain standardized flight status data, each standardized flight status data includes at least one standardized status collection data; Based on at least one standardized state acquisition data of each standardized flight state data, data reconstruction processing is performed in combination with a sliding window to determine the UAV flight data.

10. A computer storage medium, characterized in that: The computer storage medium stores at least one instruction and at least one program, and the at least one instruction and the at least one program are loaded and executed by the processor to implement the anomaly detection method for a drone multi-sensor system as described in any one of claims 1 to 9.

Images