Anomaly Detection Method and Storage Medium for UAV Multi-Sensor System

Abstract

The present invention discloses an abnormal detection method and a storage medium for a multi-sensor system of an unmanned aerial vehicle, belonging to the technical field of abnormal detection of unmanned aerial vehicles. The method includes: receiving an abnormal analysis task; inputting the flight data of the unmanned aerial vehicle into an unmanned aerial vehicle abnormal analysis model to obtain abnormal analysis information of the abnormal analysis task; the unmanned aerial vehicle abnormal analysis model includes a multi-modal feature extraction network, a known abnormal recognition module, and an unknown abnormal detection module; the multi-modal feature extraction network is used for feature extraction; the known abnormal recognition module is used for recognizing known abnormal types; the unknown abnormal detection module is used for detecting unknown abnormalities; when the unknown abnormal detection module detects an unknown abnormality, in combination with a preset unknown abnormal recognition algorithm, it is used for recognizing the unknown abnormal type. By using the technical solution provided by the present invention, fine-grained distinction between known and unknown abnormalities of the multi-sensor system of the unmanned aerial vehicle can be realized, and the accuracy of abnormal detection of the multi-sensor system of the unmanned aerial vehicle can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) anomaly detection, and particularly to an anomaly detection method and a storage medium for a UAV multi-sensor system. Background Art

[0002] With the continuous development of UAV technology, UAV technology can be integrated with many cutting-edge technologies such as electronic information and artificial intelligence, which not only greatly increases the functions of UAVs, but also significantly improves their working efficiency. Currently, UAVs are widely used in many civilian fields due to their convenience, flexibility and cost-effectiveness advantages, including target detection, wireless communication, agricultural remote sensing, disaster management, intelligent transportation systems, etc. Although UAVs have great application potential, they still face serious safety problems. A UAV is a complex physical system composed of many components such as sensors, engines, and wings. Once these mechanical parts fail, it will also lead to the failure or even crash of the UAV flight, causing serious consequences. In order to timely detect various anomalies occurring during the operation of UAVs and respond quickly, UAV anomaly detection technology has emerged and gradually become a research hotspot in the UAV field.

[0003] The purpose of anomaly detection is to identify abnormal behaviors or patterns in data, which is an important prerequisite for subsequent implementation of corrective measures. During the flight of a UAV, a large amount of flight data will be generated. These data are usually collected by various sensors and are important indicators reflecting the performance of the UAV such as stability, maneuverability, and flight efficiency. Therefore, anomaly detection for UAV flight data is an effective measure to ensure UAV flight safety. Currently, there are many studies on UAV anomaly detection, and common detection methods can be divided into three categories, including knowledge-based methods, model-based methods, and data-driven methods. Knowledge-based methods design anomaly detection rules and algorithms based on the professional skills and knowledge of UAV experts. This method has very good detection effects when the UAV system is relatively simple or the failure modes are clearly distinguishable. However, these methods rely heavily on prior knowledge and have very limited flexibility. Model-based methods first comprehensively model the physical characteristics of UAVs, and then use the established model to estimate the behavior of UAVs. By comparing the difference between the estimated value and the actual value, it is judged whether the UAV is abnormal. Although model-based methods are robust and can be adjusted according to specific fields and dataset types, it is very difficult to establish an accurate physical model of UAVs in practice.

[0004] Compared with knowledge-based and model-based methods, data-driven methods do not require modeling the complex physical characteristics of drones, nor do they rely on a large amount of prior knowledge, and can make full use of drone flight data. Therefore, data-driven methods have attracted the attention of researchers and have become the mainstream means of drone anomaly detection. Data-driven methods aim to analyze the statistical characteristics or distributions of drone flight data and identify abnormal data that differs from the normal data distribution. Among numerous data-driven methods, due to the rapid development of computer science and artificial intelligence, data-driven methods based on deep learning have become increasingly prominent in the field of drone anomaly detection. These methods can effectively capture the complexity and variability in drone flight data.

[0005] Although data-driven methods based on deep learning have achieved good detection results in drone anomaly detection, there are still many problems to be solved. Drone anomaly detection methods based on deep learning can generally be divided into two categories, including supervised learning methods and unsupervised learning methods.

[0006] Existing supervised learning methods can only detect known drone anomaly patterns. However, drones have complex structures and changing operating environments. During initial deployment, abnormal states that have never occurred may occur. Once unknown anomalies appear, existing models will be powerless. And existing unsupervised learning methods can only achieve binary classification of normal / abnormal and cannot distinguish more finely what kind of anomaly has occurred. This has hindered the subsequent targeted maintenance of drones. Therefore, more reliable solutions are needed. Summary of the Invention

[0007] The purpose of the present invention is to overcome the deficiencies in the prior art and provide an anomaly detection method and storage medium for a drone multi-sensor system, which can improve the accuracy and efficiency of anomaly detection in the drone multi-sensor system.

[0008] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0009] On the one hand, the present invention provides an anomaly detection method for a drone multi-sensor system, and the method includes:

[0010] Receiving an anomaly analysis task sent by a user;

[0011] Based on the anomaly analysis task, obtaining drone flight data collected by the drone multi-sensor system;

[0012] Inputting the drone flight data into a drone anomaly analysis model to obtain anomaly analysis information corresponding to the anomaly analysis task;

[0013] Among them, the UAV anomaly analysis model includes a multi-modal feature extraction network, a known anomaly recognition module, and an unknown anomaly detection module; the multi-modal feature extraction network is used for feature extraction, and the multi-modal feature extraction network is obtained by adjusting the parameters of the initial multi-modal feature extraction network based on the triplet margin total loss, and the triplet margin total loss is determined based on the triplet set in combination with the triplet loss function; the known anomaly recognition module is used to recognize known anomaly types; the unknown anomaly detection module is used to detect unknown anomalies; when the unknown anomaly detection module detects an unknown anomaly, in combination with a preset unknown anomaly recognition algorithm, it is used to recognize the unknown anomaly type.

[0014] In some possible implementation manners, the anomaly analysis task includes a known anomaly analysis task and an unknown anomaly analysis task;

[0015] Inputting the UAV flight data into the UAV anomaly analysis model to obtain the anomaly analysis information corresponding to the anomaly analysis task includes:

[0016] When the anomaly analysis task is a known anomaly analysis task and an unknown anomaly analysis task, inputting the UAV flight data into the multi-modal feature extraction network for feature extraction to obtain flight multi-modal feature data;

[0017] Based on the known anomaly recognition module, performing known anomaly recognition processing on the flight multi-modal feature data to obtain the known anomaly type information of the flight multi-modal feature data;

[0018] Based on the unknown anomaly detection module, performing unknown anomaly detection processing on the flight multi-modal feature data to obtain the unknown anomaly detection information of the flight multi-modal feature data;

[0019] Based on the unknown anomaly detection information and the preset unknown anomaly recognition algorithm, performing unknown anomaly recognition processing on the unknown anomaly detection information to obtain the unknown anomaly type information of the flight multi-modal feature data;

[0020] Based on the known anomaly type information and the unknown anomaly type information, determining the anomaly analysis information of the anomaly analysis task.

[0021] In some possible implementation manners, the multi-modal feature extraction network includes a time series feature extraction network, an attitude feature extraction network, and a feature fusion network; the UAV flight data includes time series data and attitude data;

[0022] Inputting the UAV flight data into the multi-modal feature extraction network for feature extraction to obtain flight multi-modal feature data includes:

[0023] Input the timing data into the timing feature extraction network for timing feature extraction to obtain timing features;

[0024] Input the pose data into the pose feature extraction network for pose feature extraction to obtain pose features;

[0025] Input the timing features and the pose features into the feature fusion network for feature fusion processing to obtain flight multi-modal feature data.

[0026] In some possible implementation manners, the multi-modal feature extraction network is constructed in the following manner:

[0027] Train an initial multi-modal feature extraction network including an initial timing feature extraction network, an initial pose feature extraction network, and an initial feature fusion network to obtain an initial multi-modal feature extraction network;

[0028] Obtain multi-modal flight sample data;

[0029] Input the multi-modal flight sample data into the initial multi-modal feature extraction network for feature extraction processing to obtain flight sample data features;

[0030] Perform triplet extraction processing on the flight sample data features to obtain a triplet set;

[0031] Based on the triplet set, combine a triplet loss function to determine the total triplet margin loss;

[0032] Based on the total triplet margin loss, adjust the parameters of the initial multi-modal feature extraction network to obtain a multi-modal feature extraction network.

[0033] In some possible implementation manners, the performing triplet extraction processing on the flight sample data features to obtain a triplet set includes:

[0034] Traverse each flight data feature in the flight sample data features;

[0035] When any flight data feature is traversed, determine the flight data feature as the current anchor data;

[0036] Determine multiple positive flight data with the same abnormal type as the current anchor data in the flight sample data features;

[0037] Based on the distance between the current anchor data and each positive flight data, determine the maximum positive distance;

[0038] Determine multiple negative flight data with different abnormal types from the current anchor data in the flight sample data features;

[0039] Determine the minimum negative distance based on the distance between the current anchor data and each piece of the negative flight data;

[0040] Determine the positive flight data corresponding to the first distance between the current anchor data and the first target distance being greater than the first target distance as the current positive flight data; the first target distance is the sum of the minimum negative distance and a preset distance;

[0041] Determine the negative flight data corresponding to the second distance between the current anchor data and the second target distance being less than the second target distance as the current negative flight data; the second target distance is the difference between the maximum positive distance and the preset distance;

[0042] Take the current anchor data, the current positive flight data, and the current negative flight data as a triple;

[0043] After traversing each flight data feature in the flight sample data features, take the obtained multiple triples as a triple set.

[0044] In some possible implementation manners, the unknown anomaly detection processing of the flight multi-modal feature data based on the unknown anomaly detection module to obtain the unknown anomaly detection information of the flight multi-modal feature data includes:

[0045] Determine multiple preset known anomaly feature data and the flight multi-modal feature data as mixed feature data;

[0046] Perform clustering processing on the mixed feature data based on a preset clustering algorithm to obtain a set of clustering feature data clusters;

[0047] Traverse each clustering feature data cluster in the set of clustering feature data clusters;

[0048] In the case of traversing any clustering feature data cluster, determine that there is no preset known anomaly feature data in the cluster, and the feature data in the current clustering feature data cluster where the number of feature data in the cluster is greater than or equal to a preset number are unknown anomaly data;

[0049] Determine the unknown anomaly detection information of the flight multi-modal feature data based on the unknown anomaly data.

[0050] In some possible implementation manners, the unknown anomaly recognition processing of the unknown anomaly detection information based on the unknown anomaly detection information and a preset unknown anomaly recognition algorithm to obtain the unknown anomaly type information of the flight multi-modal feature data includes:

[0051] Obtain a training set, a test set, and a support set; the support set includes a preset number of data with unknown anomaly types, and the preset number of data with unknown anomaly types is obtained by performing unknown anomaly type marking processing on a preset number of first unknown anomaly data, and the preset number of first unknown anomaly data is obtained from the unknown anomaly detection information; the test set includes a known anomaly type data set and an unknown anomaly data set, the unknown anomaly data set includes a plurality of second unknown anomaly data, and the preset number of first unknown anomaly data and the plurality of second unknown anomaly data constitute the unknown anomaly detection information; the training set includes known anomaly type data samples;

[0052] Determine the support set and the training set as a mixed data set;

[0053] Input the mixed data set into a multi-modal feature extraction network for feature extraction to obtain a mixed data feature set;

[0054] Input the test set into a multi-modal feature extraction network for feature extraction to obtain a test data feature set;

[0055] Traverse each test data feature in the test data feature set;

[0056] When traversing any test data feature, determine the vector distance between the current test data feature and each mixed data feature;

[0057] Determine the anomaly type label of the mixed data feature corresponding to the minimum distance among the N vector distances as the anomaly type label of the current test data feature;

[0058] Based on the anomaly type labels of each test data feature, determine the unknown anomaly type information.

[0059] In some possible implementation manners, the known anomaly type information of the flight multi-modal feature data is obtained by performing known anomaly recognition processing on the flight multi-modal feature data based on the known anomaly recognition module, including:

[0060] Based on a preset known anomaly recognition algorithm, perform known anomaly recognition processing on the flight multi-modal feature data to obtain the known anomaly type information of the flight multi-modal feature data, where the preset known anomaly recognition algorithm includes the nearest neighbor algorithm.

[0061] In some possible implementation manners, before obtaining the drone flight data collected by the drone multi-sensor system based on the anomaly analysis task, it includes:

[0062] Based on the abnormal analysis task, obtain the original UAV flight data collected by the UAV multi-sensor system. The original UAV flight data includes multiple flight status data, and each flight status data includes multiple status acquisition data;

[0063] Perform quantity alignment processing on the status acquisition data of the multiple flight status data to obtain multiple aligned flight status data. Each aligned flight status data includes multiple aligned status acquisition data;

[0064] Perform data standardization processing on the multiple aligned status acquisition data of each aligned flight status data to obtain multiple standardized flight status data. Each standardized flight status data includes multiple standardized status acquisition data;

[0065] Based on the multiple standardized status acquisition data of each standardized flight status data, combine with a sliding window to perform data reconstruction processing to determine the UAV flight data.

[0066] On the other hand, the present invention provides a computer storage medium. At least one instruction and at least one program segment are stored in the computer storage medium. The at least one instruction and the at least one program segment are loaded and executed by a processor to implement the above-mentioned abnormal detection method for a UAV multi-sensor system.

[0067] Compared with the prior art, the beneficial effects achieved by the present invention:

[0068] In the present invention, by receiving the abnormal analysis task sent by the user, the needs of the user for abnormal detection of the UAV multi-sensor system can be determined; then based on the abnormal analysis task, the UAV flight data collected by the UAV multi-sensor system is obtained, and the UAV flight data is input into the UAV abnormal analysis model to obtain the abnormal analysis information corresponding to the abnormal analysis task. The abnormalities existing in the UAV multi-sensor system and the corresponding abnormal types can be determined based on the user's needs, improving the accuracy and efficiency of abnormal detection of the UAV multi-sensor system, and reducing resource waste;

[0069] The UAV anomaly analysis model includes a multi-modal feature network, a known anomaly recognition module, and an unknown anomaly recognition module. The multi-modal feature extraction network is used for feature extraction, and it is obtained by adjusting the parameters of the initial multi-modal feature extraction network based on the triplet margin total loss. It can effectively balance the relationship between different modal features, reduce the distance between features of the same type, increase the distance between features of different types, and thus improve the accuracy and efficiency of known anomaly recognition and unknown anomaly recognition. The known anomaly recognition module is used to recognize known anomaly types, and the unknown anomaly detection module is used to detect unknown anomalies. When the unknown anomaly detection module detects an unknown anomaly, it combines a preset unknown anomaly recognition algorithm to identify the unknown anomaly type, which can achieve the discovery of potential unknown anomalies in the UAV multi-sensor system, as well as the recognition of known and unknown anomaly types in the UAV multi-sensor system, and improve the accuracy and efficiency of the recognition of known and unknown anomaly types. Brief Description of the Drawings

[0070] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0071] Figure 1 It is a flowchart showing the process of an anomaly detection method for a UAV multi-sensor system provided by an embodiment of the present invention;

[0072] Figure 2 It is a flowchart showing the process before obtaining the UAV flight data collected by the UAV multi-sensor system based on an anomaly analysis task provided by an embodiment of the present invention;

[0073] Figure 3 It is a flowchart showing the process of inputting the UAV flight data into the UAV anomaly analysis model to obtain the anomaly analysis information corresponding to the anomaly analysis task provided by an embodiment of the present invention;

[0074] Figure 4 It is a flowchart showing the process of determining flight multi-modal feature data provided by an embodiment of the present invention;

[0075] Figure 5 It is a schematic diagram of a time series feature extraction network and an attitude feature extraction network provided by an embodiment of the present invention;

[0076] Figure 6 It is a flowchart showing the process of running a Transformer encoder provided by an embodiment of the present invention;

[0077] Figure 7 It is a schematic flow chart of constructing a multi-modal feature extraction network provided by an embodiment of the present invention;

[0078] Figure 8 It is a schematic flow chart of performing triple extraction processing on flight sample data features to obtain a triple set provided by an embodiment of the present invention;

[0079] Figure 9 It is a schematic diagram of a triple provided by an embodiment of the present invention;

[0080] Figure 10 It is a schematic flow chart of performing unknown anomaly detection processing on flight multi-modal feature data based on an unknown anomaly detection module to obtain unknown anomaly detection information of the flight multi-modal feature data provided by an embodiment of the present invention;

[0081] Figure 11 It is a schematic flow chart of performing unknown anomaly recognition processing on unknown anomaly detection information based on unknown anomaly detection information and a preset unknown anomaly recognition algorithm to obtain unknown anomaly type information of the flight multi-modal feature data provided by an embodiment of the present invention;

[0082] Figure 12 It is a schematic structural diagram of an anomaly detection device for a drone multi-sensor system provided by an embodiment of the present invention. Detailed implementation manners

[0083] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0084] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or server including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0085] In an embodiment of the present invention, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of that module or unit.

[0086] Various exemplary embodiments, features, and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0087] The special term "exemplary" herein means "serving as an example, an embodiment, or illustrative". Any embodiment described as "exemplary" herein does not have to be construed as superior to or better than other embodiments.

[0088] The term "and / or" herein merely describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of multiple types or any combination of at least two of multiple types. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set composed of A, B, and C.

[0089] In addition, for a better description of the present invention, numerous specific details are given in the following specific embodiments. Those skilled in the art should understand that the present invention can also be implemented without some specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present invention.

[0090] Figure 1 is a schematic flowchart of an anomaly detection method for a multi-sensor system of an unmanned aerial vehicle provided by an embodiment of the present invention. This specification provides method operation steps such as in the embodiment or flowchart, but based on routine or non-creative labor, there can be more or fewer operation steps. The step order listed in the embodiment is only one way among the execution orders of numerous steps and does not represent the only execution order. When the actual system or server product executes, it can be executed in the order of the method shown in the embodiment or the drawings or executed in parallel (for example, in an environment of parallel processors or multi-threaded processing). Specifically, as Figure 1 shown, the above method may include:

[0091] S101: Receive an anomaly analysis task sent by a user;

[0092] In a specific embodiment, the anomaly analysis task may be a task for determining anomalies in a drone multi-sensor system. Optionally, the anomaly analysis task may include a known anomaly analysis task and an unknown anomaly analysis task. The known anomaly analysis task may be a task for determining whether there are known anomalies in the drone multi-sensor system and the types of known anomalies that exist; the unknown anomaly analysis task may be a task for determining whether there are unknown anomalies in the drone multi-sensor system and the types of unknown anomalies that exist. Optionally, a known anomaly may be an anomaly that has been clearly defined; an unknown anomaly may be an anomaly that has not been clearly defined.

[0093] S102: Based on the anomaly analysis task, obtain the drone flight data collected by the drone multi-sensor system;

[0094] In a specific embodiment, the drone flight data may be data characterizing the flight state of the drone. Specifically, the drone flight data may reflect the motion information and spatial attitude of the drone. Optionally, the drone flight data may include time-series data and attitude data. The time-series data may reflect the dynamic motion information of the drone. Specifically, the time-series data may include the angular velocity and linear acceleration of the drone, etc.; the attitude data may reflect the spatial attitude of the drone. Specifically, the attitude data may include the pitch angle, roll angle, yaw angle, etc. of the drone. In the case of clarifying the anomaly analysis task sent by the user, obtain the drone flight data collected by the drone multi-sensor system for corresponding anomaly analysis processing.

[0095] In an alternative embodiment, Figure 2 is a schematic flowchart before obtaining the drone flight data collected by the drone multi-sensor system based on an anomaly analysis task provided by an embodiment of the present invention; as Figure 2 shown, before obtaining the drone flight data collected by the drone multi-sensor system based on the anomaly analysis task, it may include:

[0096] S201: Obtain the original drone flight data collected by the drone multi-sensor system. The original drone flight data includes a plurality of flight state data, and each flight state data includes a plurality of state acquisition data;

[0097] S202: Perform a quantity alignment process on the state acquisition data of the plurality of flight state data to obtain a plurality of aligned flight state data, and each aligned flight state data includes a plurality of aligned state acquisition data;

[0098] S203: Perform data standardization processing on multiple aligned state acquisition data for each aligned flight state data to obtain multiple standardized flight state data, and each standardized flight state data includes multiple standardized state acquisition data;

[0099] S204: Based on the multiple standardized state acquisition data of each standardized flight state data, perform data reconstruction processing in combination with a sliding window to determine the UAV flight data.

[0100] In a specific embodiment, the multiple flight state data may be data characterizing the kinematic state of the UAV. Specifically, the multiple flight state data may include angular velocity data rotating around the X-axis, linear acceleration data in the X-axis direction, pitch angle data, roll angle data, etc. Specifically, classifying the multiple flight state data can be divided into time series data and attitude data. The multiple state acquisition data may be obtained by sampling each flight state data according to the sampling frequency.

[0101] The multiple flight state data can be denoted as , representing the number of flight state data; optionally, for any UAV flight state data there is a sampling frequency ; after sampling, will include multiple state acquisition data, that is , where represents the th state acquisition data in the th flight state data, represents the number of state acquisition data corresponding to the th flight state data; optionally, each state acquisition data has a corresponding timestamp, that is, the timestamp of the sampled state acquisition data. At this time, 's timestamp set can be , where represents the timestamp corresponding to the th state acquisition data in the th flight state data; optionally, since the sampling frequencies corresponding to different flight state data may be different, that is, when , there may be ; and the number of state acquisition data corresponding to different flight state data may be different, that is, when , there may be ; where represents the th flight state data.

[0102] In a specific embodiment, the above-mentioned alignment processing of the number of status acquisition data for multiple flight status data to obtain multiple aligned flight status data may include: using a linear interpolation algorithm to perform alignment processing on the number of status acquisition data for multiple flight status data to obtain multiple aligned flight status data. The specific steps are as follows:

[0103] Based on the number of status acquisition data for each flight status data, perform a descending order sorting on multiple flight status data to obtain a descending order sorting result of multiple flight status data;

[0104] Based on the descending order sorting result of multiple flight status data, determine the reference status data; optionally, the reference status data may be the flight status data with the first position in the descending order sorting result. Specifically, the reference status data may be the flight status data with the largest number of status acquisition data;

[0105] Obtain the reference timestamp set of the reference status data;

[0106] Select any flight status data from multiple flight status data as the flight status data to be interpolated, and obtain the timestamp set of the flight status data to be interpolated;

[0107] Select a reference timestamp from the reference timestamp set as the current reference timestamp;

[0108] Determine the first timestamp and the second timestamp in the timestamp set; the first timestamp and the second timestamp are the two timestamps closest to the current reference timestamp, and the current reference timestamp is greater than the first timestamp and less than the second timestamp;

[0109] Obtain the first status acquisition data corresponding to the first timestamp and the second status acquisition data corresponding to the second timestamp from the flight status data to be interpolated;

[0110] Determine the interpolated status acquisition data corresponding to the flight status data to be interpolated at the current reference timestamp, as shown in the following formula:

[0111] ;

[0112] Wherein, represents the th interpolated status acquisition data of the th flight status data to be interpolated; represents the current reference timestamp; represents the first status acquisition data; represents the second status acquisition data; represents the first timestamp; represents the second timestamp; represents the The -th linear interpolation process of the flight state data to be interpolated;

[0113] Obtain the next reference timestamp in the reference timestamp set as the current reference timestamp, and jump to the above to determine the first timestamp and the second timestamp in the timestamp set, and obtain the interpolation state acquisition data of the flight state data to be interpolated at the next reference timestamp;

[0114] Repeat the above process of selecting a reference timestamp from the reference timestamp set as the current reference timestamp until obtaining the next reference timestamp in the reference timestamp set as the current reference timestamp, and then jump to the above to determine the first timestamp and the second timestamp in the timestamp set, and obtain the interpolation state acquisition data of the flight state data to be interpolated at the next reference timestamp, until all timestamps in the reference timestamp set are traversed, and the interpolated flight state data can be expressed as follows:

[0115] ;

[0116] wherein, represents the -th interpolated flight state data; represents the -th interpolation state acquisition data corresponding to the -th interpolated flight state data; represents the number of state acquisition data corresponding to the reference state data;

[0117] Select the next flight state data to be interpolated, that is, jump to select any flight state data from multiple flight state data as the flight state data to be interpolated, and obtain the timestamp set of the flight state data to be interpolated, until all flight state data have completed linear interpolation, and multiple aligned flight state data are obtained; each aligned flight state data includes multiple aligned state acquisition data (interpolation state acquisition data), and the aligned state acquisition data corresponding to multiple aligned flight state data are aligned in quantity.

[0118] In a specific embodiment, the above data normalization process for multiple aligned state acquisition data of each aligned flight state data to obtain multiple normalized flight state data may include:

[0119] Select any aligned flight state data from multiple aligned flight state data as the flight state data to be normalized;

[0120] Determine the state acquisition mean data corresponding to the flight state data to be normalized, as shown in the following formula:

[0121] ;

[0122] Among them, represents the mean value of state acquisition corresponding to the th flight state data to be standardized (aligned flight state data);

[0123] Determine the standard deviation data of state acquisition corresponding to the flight state data to be standardized, as shown in the following formula:

[0124] ;

[0125] Among them, represents the standard deviation data of state acquisition corresponding to the th flight state data to be standardized;

[0126] According to the mean value of state acquisition data and the standard deviation data of state acquisition, perform data standardization processing on each aligned state acquisition data of the flight state data to be standardized, and obtain multiple standardized state acquisition data, as shown in the following formula:

[0127] ;

[0128] Among them, represents the th standardized state acquisition data in the th flight state data to be standardized; that is, represents the th standardized state acquisition data obtained by performing data standardization processing on the th aligned state acquisition data in the th flight state data to be standardized;

[0129] According to multiple standardized state acquisition data, obtain the standardized flight state data;

[0130] Select the next aligned flight state data, that is, jump to select any aligned flight state data from multiple aligned flight state data as the flight state data to be standardized, until all aligned flight state data have completed data standardization processing, and obtain multiple standardized flight state data, and each standardized flight state data includes multiple standardized state acquisition data.

[0131] In a specific embodiment, the above-mentioned multiple standardized state acquisition data based on each standardized flight state data, combined with a sliding window for data reconstruction processing to determine the UAV flight data, may include:

[0132] Parallelly splice multiple standardized state acquisition data together according to the time stamp order;

[0133] Create a sliding window with a length equal to the preset window length and a sliding step equal to the preset sliding step; optionally, the preset window length and the preset sliding step can be set according to the actual application;

[0134] Start from the first timestamp of the sliding window and slide on the time axis;

[0135] Before each slide, extract the normalized flight state data corresponding to the timestamps of the preset window length from all the normalized flight state data within the sliding window, and combine them to obtain the reconstructed flight state data. Each reconstructed flight state data can include multiple reconstructed state acquisition data;

[0136] Based on multiple reconstructed flight state data, obtain the reconstructed UAV flight data, that is:

[0137] ;

[0138] Among them, represents the th reconstructed UAV flight data; represents the th reconstructed flight state data in the th reconstructed UAV flight data; represents the th reconstructed state acquisition data in the th reconstructed flight state data; represents the preset window length;

[0139] The sliding window will slide backward along the time axis by the number of timestamps of the preset sliding step and complete the construction of the next reconstructed UAV flight data. Repeat the above process that the sliding window will slide backward along the time axis by the number of timestamps of the preset sliding step and complete the construction of the next reconstructed UAV flight data until the sliding window traverses all the timestamps, obtaining multiple reconstructed UAV flight data, and each reconstructed UAV flight data includes multiple reconstructed flight state data;

[0140] Based on multiple reconstructed UAV flight data, obtain the UAV flight data.

[0141] In the above embodiments, the data quality can be improved, the data diversity can be increased, and the associations between heterogeneous features at the same moment can be captured. At the same time, the temporal relationships existing between heterogeneous features can also be captured, thereby enhancing the model's ability and improving the accuracy of subsequent abnormal analysis information determination, and further improving the accuracy and effectiveness of UAV multi-sensor system anomaly detection.

[0142] S103: Input the UAV flight data into the UAV anomaly analysis model to obtain the anomaly analysis information corresponding to the anomaly analysis task.

[0143] In a specific embodiment, the UAV anomaly analysis model can be used to analyze the anomalies in UAV flight data; the UAV anomaly analysis model can include a multi-modal feature extraction network, a known anomaly recognition module, and an unknown anomaly detection module; the multi-modal feature extraction network can be used for feature extraction, and the multi-modal feature extraction network can be obtained by adjusting the parameters of the initial multi-modal feature extraction network based on the triplet margin total loss, and the triplet margin total loss is determined based on the triplet set in combination with the triplet loss function; the known anomaly recognition module can be used to recognize known anomaly types; the unknown anomaly detection module can be used to detect unknown anomalies; the unknown anomaly detection module can, in the case of detecting an unknown anomaly, be combined with a preset unknown anomaly recognition algorithm to identify the unknown anomaly type.

[0144] The anomaly analysis information can characterize the anomaly situation of the UAV multi-sensor system detected by using the UAV anomaly analysis model; optionally, the anomaly analysis information can include known anomaly analysis information and unknown anomaly analysis information, the known anomaly analysis information can characterize whether there is a known anomaly and the type of the known anomaly, and the unknown anomaly analysis information can characterize whether there is an unknown anomaly and the type of the unknown anomaly.

[0145] In an alternative embodiment, Figure 3 is a schematic flowchart of a process of inputting UAV flight data into the UAV anomaly analysis model to obtain anomaly analysis information corresponding to an anomaly analysis task provided by an embodiment of the present invention; as Figure 3 shown, the above process of inputting UAV flight data into the UAV anomaly analysis model to obtain anomaly analysis information corresponding to an anomaly analysis task can include:

[0146] S301: When the anomaly analysis task is a known anomaly analysis task and an unknown anomaly analysis task, input the UAV flight data into the multi-modal feature extraction network for feature extraction to obtain flight multi-modal feature data;

[0147] S302: Based on the known anomaly recognition module, perform known anomaly recognition processing on the flight multi-modal feature data to obtain known anomaly type information of the flight multi-modal feature data;

[0148] S303: Based on the unknown anomaly detection module, perform unknown anomaly detection processing on the flight multi-modal feature data to obtain unknown anomaly detection information of the flight multi-modal feature data;

[0149] S304: Based on the unknown anomaly detection information and the preset unknown anomaly recognition algorithm, perform unknown anomaly recognition processing on the unknown anomaly detection information to obtain unknown anomaly type information of the flight multi-modal feature data;

[0150] S305: Determine the anomaly analysis information of the anomaly analysis task based on the known anomaly type information and the unknown anomaly type information.

[0151] In a specific embodiment, the anomaly analysis task being a known anomaly analysis task and an unknown anomaly analysis task can characterize the user's anomaly detection requirements for the drone multi-sensor system, that is, the user wants to determine the known anomalies and unknown anomalies of the drone multi-sensor system.

[0152] In an alternative embodiment, the above-mentioned multi-modal feature extraction network includes a temporal feature extraction module, an attitude feature extraction module, and a feature fusion module;

[0153] Figure 4 is a schematic flow diagram for determining flight multi-modal feature data provided by an embodiment of the present invention; as Figure 4 shown, inputting the drone flight data into the multi-modal feature extraction network for feature extraction to obtain flight multi-modal feature data may include:

[0154] S401: Input the temporal data into the temporal feature extraction network for temporal feature extraction to obtain temporal features;

[0155] S402: Input the attitude data into the attitude feature extraction network for attitude feature extraction to obtain attitude features;

[0156] S403: Input the temporal features and the attitude features into the feature fusion network for feature fusion processing to obtain flight multi-modal feature data.

[0157] In a specific embodiment, the multi-modal feature extraction network can be a deep learning network for performing multi-modal feature extraction. The temporal data can be at least one piece of temporal data, and the attitude data can be at least one piece of attitude data. The temporal feature extraction network can be used to extract features from the temporal data of the drone flight data; optionally, the network structure of the temporal feature extraction network can be set according to the actual application requirements. Specifically, Figure 5 is a schematic diagram of a temporal feature extraction network and an attitude feature extraction network provided by an embodiment of the present invention; as Figure 5 shown, the temporal feature extraction network can include a fully connected layer. Optionally, the above-mentioned inputting the temporal data into the temporal feature extraction network for temporal feature extraction to obtain temporal features may include:

[0158] Input the temporal data into the temporal fully connected layer for temporal feature extraction to obtain temporal features, as shown in the following formula:

[0159] ;

[0160] where represents the The output features of the temporal fully-connected layer corresponding to the reconstructed UAV flight data, i.e., the temporal features denotes the preset window length denotes the temporal feature extraction network denotes the th temporal data corresponding to the reconstructed UAV flight data denotes the quantity of the temporal data denotes the weight matrix of the temporal feature extraction network denotes the bias vector of the temporal feature extraction network denotes the input dimension of the fully-connected layer (of the temporal feature extraction network) denotes the output dimension of the fully-connected layer (of the temporal feature extraction network)

[0161] In a specific embodiment, the attitude feature extraction network can be used to extract features from the attitude data of the UAV flight data; optionally, the network structure of the attitude feature extraction network can be set according to the actual application requirements. Specifically, as Figure 5 shown, the attitude feature extraction network can include an attitude fully-connected layer, multiple bottleneck layers, and a reshaping layer. Optionally, the above-mentioned inputting the attitude data into the attitude feature extraction network to extract attitude features may include:

[0162] Inputting the attitude data into the attitude fully-connected layer to obtain the output features of the fully-connected layer, as shown in the following formula:

[0163] ;

[0164] wherein denotes the output features of the attitude fully-connected layer corresponding to the th reconstructed UAV flight data denotes the intermediate layer dimension of the attitude feature extraction network ATE denotes the th attitude data corresponding to the reconstructed UAV flight data denotes the quantity of the attitude data denotes the weight matrix of the fully-connected layer FC denotes the input dimension of the fully-connected layer denotes the bias vector of the fully-connected layer

[0165] Inputting the output features of the fully-connected layer into multiple bottleneck layers for feature extraction to obtain the output features of the bottleneck layers

[0166] Inputting the output features of the bottleneck layers into the reshaping layer to obtain the attitude features

[0167] Optionally, the bottleneck layer may include a dilation layer, a depth convolution layer, a projection layer, and a residual connection layer;

[0168] Inputting the output features of the fully connected layer into multiple bottleneck layers for feature extraction to obtain the output features of the total bottleneck layer may include:

[0169] Inputting the output features of the fully connected layer into the dilation layer for dimensional expansion to obtain the output features of the dilation layer; wherein, the dilation layer includes a pointwise convolution layer, a batch normalization layer, and an activation layer, the convolution kernel of the pointwise convolution layer is 1*1, the input dimension is , and the output dimension is , is the expansion multiple. Inputting the output features of the fully connected layer into the dilation layer for dimensional expansion to obtain the output features of the dilation layer can be shown as the following formula:

[0170] ;

[0171] Wherein, represents the output features of the dilation layer in the th bottleneck layer; represents the convolution operation; represents the convolution kernel of the dilation layer in the th bottleneck layer; represents the bias vector of the dilation layer in the th bottleneck layer; represents batch normalization; represents the activation function;

[0172] Inputting the output features of the dilation layer into the depth convolution layer for feature extraction to obtain the output features of the depth convolution layer; wherein, the depth convolution layer includes depth convolution, a batch normalization layer, and an activation layer, the convolution kernel of the depth convolution is 3*3, and both the input dimension and the output dimension are . Inputting the output features of the dilation layer into the depth convolution layer for feature extraction to obtain the output features of the depth convolution layer can be shown as the following formula:

[0173] ;

[0174] Wherein, represents the output features of the depth convolution layer in the th bottleneck layer; represents the convolution kernel of the depth convolution layer in the th bottleneck layer; represents the bias vector of the depth convolution layer in the th bottleneck layer;

[0175] The output features of the depth convolution layer are input into the projection layer for dimensionality compression to obtain the output features of the projection layer; among them, the projection layer includes a pointwise convolution layer and a batch normalization layer, and the convolution kernel of the pointwise convolution layer is 1*1, and the input dimension is and the output dimension is ,

[0176] The above-mentioned input of the output features of the depth convolution layer into the projection layer for dimensionality compression to obtain the output features of the projection layer can be shown as the following formula;

[0177] ;

[0178] where, represents the output features of the projection layer in the th bottleneck layer; represents the convolution kernel of the projection layer in the th bottleneck layer; represents the bias vector of the projection layer in the th bottleneck layer;

[0179] Using residual connection, add the input data of the bottleneck layer and the output features of the projection layer to obtain the output features of the bottleneck layer as shown in the following formula:

[0180] ;

[0181] where, represents the output features of the th bottleneck layer, represents the output of the previous bottleneck layer (i.e., the input of the current bottleneck layer). When , that is, when the current bottleneck layer is the first bottleneck layer in the pose feature extraction network, its output will be added to the output of the fully connected layer of the pose feature extraction network;

[0182] Send the output features of the th bottleneck layer into the th bottleneck layer, input the output features of the th bottleneck layer into the dilation layer for dimensionality expansion to obtain the output features of the dilation layer, repeat the above-mentioned input of the output features of the dilation layer into the depth convolution layer for feature extraction to obtain the output features of the depth convolution layer to using residual connection, add the input data of the bottleneck layer and the output features of the projection layer to obtain the output features of the bottleneck layer, and obtain the output features of the th bottleneck layer;

[0183] Repeat the above-mentioned steps of inputting the output features of the bottleneck layer into the bottleneck layer for feature extraction until multiple bottleneck layers are traversed, and obtain the output features of the last bottleneck layer (total bottleneck layer output features) , Represents the number of multiple bottleneck layers.

[0184] Optionally, a reshaping layer can be used to adjust the dimension of the output features of the total bottleneck layer and obtain the output of the pose feature extraction network, i.e., the pose feature , and sometimes the output dimensions of the temporal feature extraction network and the pose feature extraction network are set to be the same, i.e., .

[0185] In a specific embodiment, the feature fusion network can be used to fuse temporal data and pose data; optionally, the network structure of the feature fusion network can be set according to actual application requirements. Specifically, the feature fusion network can include a Transformer layer, and the Transformer layer can include an encoding layer and multiple Transformer encoders. Optionally, the above-mentioned input of the temporal feature and the pose feature into the feature fusion network for feature fusion processing to obtain the flight multi-modal feature data can include:

[0186] Concatenate the temporal feature and the pose feature as follows to obtain the concatenated output feature:

[0187] ;

[0188] Among them, represents the tensor obtained after concatenation, i.e., the concatenated output feature, represents the temporal feature in the th element, , represents the pose feature in the th element, , and finally use to represent in the element, ; represents the input dimension of the Transformer layer, where ;

[0189] Send the concatenated output feature into the encoding layer for information encoding, as shown in the following formula:

[0190] ;

[0191] ;

[0192] Among them, , can represent the dimension index of the position information after information encoding, Can be used to ensure that adjacent dimensions alternately use sine and cosine functions; Represents a feature vector; Indicates being located at The feature vector of, and there is ; Represents the th position information located at ;

[0193] Add the feature vector located at and the position information and position information to obtain a new feature vector after position encoding, as shown in the following formula:

[0194] ;

[0195] Combine all the after position encoding to obtain the output of the encoding layer, and there is ;

[0196] Feed the output of the encoding layer into the Transformer encoder; assume there are Transformer encoders;

[0197] Optionally, Figure 6 is a schematic diagram of the process of running a Transformer encoder provided by an embodiment of the present invention; as Figure 6 shown, use the output of the encoding layer as the input of the Transformer encoder; and the Transformer encoder may include a multi-head attention module, a batch normalization module, a feed-forward module, and a residual connection. The multi-head attention module has parallel self-attention heads. The running process of the Transformer encoder is as follows:

[0198] Split the input of the Transformer encoder into groups, denoted as , represents the input of the th Transformer encoder;

[0199] Feed the split inputs into self-attention heads respectively. The running process of the above self-attention heads can be as follows:

[0200] Use the input of the self-attention head with the parameter matrix , and The linear transformation is , and , can represent the first parameter matrix used for the linear transformation of the input . can represent the second parameter matrix used for the linear transformation of the input . can represent the third parameter matrix used for the linear transformation of the input ; can represent the first eigenvector obtained by linearly transforming the input using the first parameter matrix, can represent the second eigenvector obtained by linearly transforming the input using the second parameter matrix, can represent the third eigenvector obtained by linearly transforming the input using the third parameter matrix;

[0201] Determine the output of the self-attention head using the attention mechanism , as shown in the following formula:

[0202] ;

[0203] ;

[0204] where represents the dimension corresponding to the input , that is, the th dimension. Also, since the input of the self-attention head is split from the input of the multi-head attention mechanism, there is ; ; represents the output of the th self-attention head; can respectively represent the query, key, and value in the self-attention mechanism, which can be used to implement the matching, screening, and aggregation of the self-attention mechanism and can dynamically focus on the key information of the input sequence.

[0205] Repeat the steps to linearly transform the input of the self-attention head using the parameter matrices , and into , and until the output , until the outputs of all self-attention heads are calculated;

[0206] Combine the outputs of all self-attention heads and use the parameter matrix to perform a linear transformation, then the output of the multi-head attention mechanism can be obtained. Here, denotes the input of the multi-head attention mechanism, as shown in the following formula:

[0207] ;

[0208] Perform batch normalization and residual connection operations on the output of the multi-head attention mechanism, as shown in the following formula:

[0209] ;

[0210] Feed the upper layer output into the feed-forward module. The feed-forward module consists of two fully connected layers and uses the ReLU function for activation. The calculation method of the feed-forward module is shown in the following formula:

[0211] ;

[0212] Among them, denotes the output of the feed-forward module; and respectively denote the weight matrix and bias vector of the first fully connected layer, while and respectively denote the weight matrix and bias vector of the second fully connected layer, and there are and , denotes the input dimension and output dimension of the feed-forward module;

[0213] Perform batch normalization and residual connection operations on the output of the feed-forward module and obtain the output of the Transformer encoder, as shown in the following formula:

[0214] ;

[0215] Among them, denotes the output of the th Transformer encoder; Optionally, the input of the th Transformer encoder can be ; Feed into the next Transformer encoder and repeat the step of splitting the input of the Transformer encoder into groups, denoted as until batch normalization and residual connection operations are performed on the output of the feedforward module, and the output of the Transformer encoder is obtained. Until all the Transformer encoders are traversed, the output of the last, i.e., the th Transformer encoder is , which is also the output of the entire Transformer layer; where, in the figure represents the output of the 1st Transformer encoder, represents the output of the 2nd Transformer encoder, represents the number of Transformer encoders;

[0216] Keep the first eigenvector in, i.e., , and discard the remaining eigenvectors, which is the output of the Transformer layer and can be flight multi-modal feature data; specifically, a post-processing is added to make adjustments; since in the finally output , each eigenvector has received the extraction information from all the inputs, so only the first eigenvector in is kept in the post-processing, and the remaining eigenvectors are discarded.

[0217] In the above embodiment, the multi-modal feature extraction network can effectively handle multi-modal heterogeneous inputs, and can synchronously extract features of different modalities of the UAV flight data collected by the multi-sensor system; while fully extracting the temporal features and attitude features in the UAV flight data, the multi-modal feature extraction network also explores the relationships between multi-modal heterogeneous features, and can map the features in the high-dimensional space to the low-dimensional space, which is convenient for subsequent anomaly detection and anomaly type recognition. By extracting features from the UAV flight data through the multi-modal feature extraction network, the relationships between different modal features can be effectively balanced, and different modal features can be fused, thereby realizing the efficient extraction of multi-modal heterogeneous features, as well as improving the accuracy and effectiveness of multi-modal heterogeneous feature extraction, and further improving the accuracy of subsequent known anomaly type recognition and unknown anomaly type recognition.

[0218] In an alternative embodiment, Figure 7 is a schematic flowchart of the construction of a multi-modal feature extraction network provided by an embodiment of the present invention; as Figure 7 shown, the above multi-modal feature extraction network can be constructed in the following manner:

[0219] S701: Train an initial multi-modal feature extraction network including an initial timing feature extraction network, an initial attitude feature extraction network, and an initial feature fusion network to obtain an initial multi-modal feature extraction network;

[0220] S702: Obtain multi-modal flight sample data;

[0221] S703: Input the multi-modal flight sample data into the initial multi-modal feature extraction network for feature extraction processing to obtain flight sample data features;

[0222] S704: Perform triplet extraction processing on the flight sample data features to obtain a triplet set;

[0223] S705: Based on the triplet set, combine with a triplet loss function to determine the total triplet margin loss;

[0224] S706: Based on the total triplet margin loss, adjust the parameters of the initial multi-modal feature extraction network to obtain a multi-modal feature extraction network.

[0225] In a specific embodiment, the multi-modal flight sample data can be multi-modal flight data collected by a drone multi-sensor system; optionally, a preset abnormal type label corresponding to the multi-modal flight sample data is obtained while obtaining the multi-modal flight sample data. The flight sample data features can be features containing multiple flight data features obtained by performing feature extraction on the multi-modal flight sample data. The initial multi-modal feature extraction network can be a preliminarily trained feature extraction network, and the multi-modal feature extraction network can be a feature extraction network that has been trained after adjusting the parameters of the initial multi-modal feature extraction network. Using the initial multi-modal feature extraction network to perform feature extraction on the flight sample data can map the flight sample data to a low-dimensional space to obtain flight sample data features.

[0226] In an alternative embodiment, Figure 8 is a schematic flowchart of a process for performing triplet extraction processing on flight sample data features to obtain a triplet set provided by an embodiment of the present invention; as Figure 8 shown, the above-mentioned performing triplet extraction processing on the flight sample data features to obtain a triplet set may include:

[0227] S801: Traverse each flight data feature in the flight sample data features;

[0228] When any flight data feature is traversed, determine the flight data feature as the current anchor data;

[0229] S802: Determine multiple positive flight data with the same abnormal type as the current anchor data in the flight sample data features;

[0230] S803: Determine the maximum positive distance based on the distances between the current anchor data and each positive flight data feature.

[0231] S804: Determine a plurality of negative flight data with different anomaly types from the current anchor data among the flight sample data features.

[0232] S805: Determine the minimum negative distance based on the distances between the current anchor data and each negative flight data.

[0233] S806: Determine the positive flight data whose first distance from the current anchor data is greater than the first target distance as the current positive flight data; the first target distance is the sum of the minimum negative distance and a preset distance.

[0234] S807: Determine the negative flight data whose second distance from the current anchor data is less than the second target distance as the current negative flight data; the second target distance is the difference between the maximum positive distance and a preset distance.

[0235] S808: Take the current anchor data, the current positive flight data, and the current negative flight data as a triple.

[0236] S809: After traversing each flight data feature in the flight sample data features, take the obtained multiple triples as a triple set.

[0237] In a specific embodiment, among the flight sample data features, select a flight data feature as the current anchor data. The positive flight data can be the flight data features with the same anomaly type as the current anchor data; optionally, the current anchor data and the positive flight data can form a positive pair; optionally, based on the preset anomaly type labels of the multimodal flight sample data, determine a plurality of positive flight data with the same anomaly type as the current anchor data among the flight sample data features. The negative flight data can be the flight data features with different anomaly types from the current anchor data; optionally, the current anchor data and the negative flight data can form a negative pair; optionally, based on the preset anomaly type labels of the multimodal flight sample data, determine a plurality of negative flight data with different anomaly types from the current anchor data among the flight sample data features.

[0238] Optionally, determining the maximum positive distance based on the distances between the current anchor data and each positive flight data may include: determining the Euclidean distances between the current anchor data and each positive flight data to obtain a plurality of first distances, where the first distances may represent within-class distances; sorting the plurality of first distances in descending order to obtain a descending order result of positive distances; using the first distance ranked first in the descending order result of positive distances as the maximum positive distance. Determining the minimum negative distance based on the distances between the current anchor data and each negative flight data may include: determining the Euclidean distances between the current anchor data and each negative flight data to obtain a plurality of second distances, where the second distances may represent between-class distances; sorting the plurality of second distances in ascending order to obtain an ascending order result of negative distances; using the second distance ranked first in the ascending order result of negative distances as the minimum negative distance.

[0239] Optionally, the preset distance may be a margin distance. Specifically, the preset distance may be set in combination with actual applications. The first distance may be the distance between the positive flight data and the current anchor data, and the second distance may be the distance between the negative flight data and the current anchor data. Optionally, determining the positive flight data with the first distance greater than the sum of the minimum negative distance and the margin distance as the current positive flight data, and determining the negative flight data with the second distance less than the difference between the maximum positive distance and the margin distance as the current negative flight data. At this time, the current anchor data, the current positive flight data, and the current negative flight data may be used as a triple; while the positive flight data that does not satisfy the condition that the first distance is greater than the sum of the minimum negative distance and the margin distance and the negative flight data that does not satisfy the condition that the second distance is less than the difference between the maximum positive distance and the margin distance will not be selected as triples. Optionally, after traversing each flight data feature in the flight sample data, all the obtained triples may be integrated into a triple set.

[0240] In a specific embodiment, Figure 9 is a schematic diagram of a triple provided by an embodiment of the present invention; as Figure 9 shown, may represent the anchor data in the triple, may represent the positive flight data in the triple, may represent the negative flight data in the triple; may represent the within-class distance between the positive flight data and the anchor data in the triple, may represent the between-class distance between the negative flight data and the anchor data in the triple, and the within-class distance is much smaller than the between-class distance ; Optionally, in the process of further increasing the distance between the anchor data and the negative flight data and reducing the distance between it and the positive flight data, a margin distance m may be added, and there is .

[0241] In a specific embodiment, determining the total triplet margin loss based on the triplet set and in combination with the triplet loss function may include: selecting a triplet from the triplet set as the current triplet; determining the triplet margin loss corresponding to the current triplet according to the current triplet and in combination with the triplet loss function; selecting the next triplet from the triplet set as the current triplet, and repeating the above step of determining the triplet margin loss corresponding to the current triplet according to the current triplet and in combination with the triplet loss function until the triplet margin losses of all triplets in the triplet set are determined; adding up the triplet margin losses of all triplets to obtain the total triplet margin loss.

[0242] Optionally, the triplet loss function can be set in combination with the actual application. Specifically, the triplet loss function can be . Optionally, based on the total triplet margin loss, the parameters of the initial multi-modal feature extraction network can be optimized using the Adam (Adaptive Moment Estimation) optimizer to obtain the multi-modal feature extraction network.

[0243] In the above embodiment, optimizing the multi-modal feature extraction network through the total triplet margin loss can constrain the relative distances of different modal data in the feature space, ensure that flight data of the same abnormal type is highly aggregated to reduce classification ambiguity, and ensure that flight data of different abnormal types is clearly separated to reduce the misjudgment probability. Furthermore, it can improve the accuracy, robustness, and efficiency of subsequent anomaly detection and recognition.

[0244] In an alternative embodiment, the above-mentioned known anomaly recognition module performs known anomaly recognition processing on the flight multi-modal feature data to obtain the known anomaly type information of the flight multi-modal feature data, which may include:

[0245] Performing known anomaly recognition processing on the flight multi-modal feature data based on a preset known anomaly recognition algorithm to obtain the known anomaly type information of the flight multi-modal feature data, where the preset known anomaly recognition algorithm includes the nearest neighbor algorithm.

[0246] In a specific embodiment, the preset known anomaly recognition algorithm can be set in combination with the actual application. Specifically, the preset known anomaly recognition algorithm can be the K-nearest neighbor algorithm. Specifically, by using the K-nearest neighbor algorithm to classify the flight multimodal feature data, the known anomaly type information corresponding to the UAV flight data can be obtained. Optionally, the flight multimodal feature data can be the data obtained by mapping the complex and high-dimensional UAV flight data to a low-dimensional space by a multimodal feature extraction network. The K-nearest neighbor algorithm is used to classify the flight multimodal feature data in the low-dimensional space. Optionally, the known anomaly type information can characterize the known anomaly data of the UAV multi-sensor system and the corresponding anomaly types, and the known anomaly type information can be the known anomaly data of the determined anomaly types existing in the UAV multi-sensor system. Optionally, while determining the known anomaly type information, normal information can also be determined; based on the preset known anomaly recognition algorithm, the flight multimodal feature data classified into different known anomaly categories can obtain the known anomaly type information, and the flight multimodal feature data classified into the normal category can obtain the normal information.

[0247] In a specific embodiment, the above-mentioned known anomaly recognition module can be constructed in the following manner:

[0248] Obtain the anomaly sample to be recognized and the preset sample recognition type of the anomaly sample to be recognized;

[0249] Input the anomaly sample to be recognized into a multimodal feature extraction network for feature extraction to obtain the feature of the anomaly sample to be recognized;

[0250] Based on the preset known anomaly detection algorithm, perform recognition processing on the feature of the anomaly sample to be recognized to obtain the predicted sample recognition type of the anomaly sample to be recognized;

[0251] Based on the preset sample recognition type and the predicted sample recognition type, adjust the parameters of the known anomaly recognition module to be constructed;

[0252] Repeat the above steps until the first preset condition is met to determine the known anomaly recognition module.

[0253] In a specific embodiment, before obtaining the anomaly sample to be recognized, it may include: obtaining the original anomaly sample to be recognized, where the original anomaly sample to be recognized includes multiple state features, and each state feature includes multiple state data; performing quantity alignment processing on the state data of the multiple state features to obtain multiple aligned state features, and each aligned state feature includes multiple aligned state data; performing data standardization processing on the multiple aligned state data of each aligned state feature to obtain multiple standardized state features, and each standardized state feature includes multiple standardized state data; combining the multiple standardized state data of each standardized state feature with a sliding window for data reconstruction processing to determine the anomaly sample to be recognized.

[0254] In a specific embodiment, the refinement before obtaining the abnormal samples to be recognized can refer to the relevant refinement before obtaining the UAV flight data collected by the UAV multi-sensor system based on the abnormal analysis task described above, which will not be elaborated here. During the construction process of the abnormal samples to be recognized here, start from the first timestamp of the sliding window and slide on the time axis; before each slide, extract the standardized state features corresponding to the timestamps of the preset window length from all the standardized state features within the sliding window and combine them to obtain the reconstructed state features. Each reconstructed state feature can include multiple reconstructed state data; according to multiple reconstructed state features, obtain the reconstructed abnormal samples to be recognized; use the abnormal type corresponding to the last timestamp as the preset sample recognition type label of the reconstructed abnormal samples to be recognized; the sliding window will slide backward along the time axis by a preset sliding step of timestamps and complete the construction of the next reconstructed abnormal samples to be recognized. The preset sample recognition type label of the next reconstructed abnormal samples to be recognized is also the abnormal type at the last timestamp within the sliding window. Repeat the above process that the sliding window will slide backward along the time axis by a preset sliding step of timestamps and complete the construction of the next reconstructed abnormal samples to be recognized until the sliding window traverses all timestamps, obtaining multiple reconstructed abnormal samples to be recognized. Each reconstructed abnormal samples to be recognized includes multiple reconstructed state features, and each reconstructed abnormal samples to be recognized corresponds to a preset sample recognition type label; according to multiple reconstructed abnormal samples to be recognized and the corresponding preset sample recognition type labels, form the abnormal samples to be recognized and the preset sample recognition type of the abnormal samples to be recognized. Specifically, the number of reconstructed abnormal samples to be recognized in the abnormal samples to be recognized can be shown by the following formula: ; where represents the number of reconstructed abnormal samples to be recognized; represents the preset window length; represents the preset sliding step; represents the number of timestamps, represents rounding up. The reconstructed abnormal samples to be recognized can expand the sample size so as to improve the accuracy of known abnormal recognition.

[0255] In a specific embodiment, the first preset condition can be set in combination with the actual application. Optionally, the first preset condition can be that the number of iterations reaches the preset number, or all the reconstructed abnormal samples to be recognized in the abnormal samples to be recognized are traversed. Optionally, during the construction process of the known abnormal recognition module, the features of the abnormal samples to be recognized for recognition processing will be pre-stored.

[0256] In the above embodiments, the method based on deep metric learning does not rely on a pre-designed model structure, and can distinguish different types of data based on the distance between data, improving the efficiency and accuracy of identifying known anomaly types; specifically, using the metric learning algorithm K-NN (K-nearest neighbor) algorithm can achieve precise classification of known anomalies in the UAV multi-sensor system, thereby improving the precision and efficiency of identifying known anomaly types.

[0257] In a specific embodiment, the anomaly analysis task in the above embodiments is the known anomaly analysis task among the known anomaly analysis task and the unknown anomaly analysis task; and in the case where the anomaly analysis task is the known anomaly analysis task, in addition to using the above-mentioned preset known anomaly recognition algorithm to perform known anomaly recognition processing on the flight multi-modal feature data to obtain the known anomaly type information of the flight multi-modal feature data, a classification model based on deep learning in the known anomaly recognition module can also be used. Specifically, the flight multi-modal feature data is input into the classification model for classification processing to obtain the predicted known anomaly type information of the flight multi-modal feature data.

[0258] In a specific embodiment, the classification model can be trained in the following manner:

[0259] Obtain the anomaly sample to be recognized and the preset sample recognition type of the anomaly sample to be recognized;

[0260] Input the anomaly sample to be recognized into the multi-modal feature extraction network for feature extraction to obtain the features of the anomaly sample to be recognized;

[0261] Input the features of the anomaly sample to be recognized into the classification model to be trained for classification processing to obtain the predicted sample recognition type;

[0262] Based on the preset sample recognition type and the predicted sample recognition type, train the classification model to be trained to obtain the classification model.

[0263] In the above embodiments, different processing is performed based on different anomaly analysis tasks, which can improve the efficiency and accuracy of processing the anomaly analysis tasks.

[0264] In an alternative embodiment, Figure 10 is a schematic flow diagram of a method provided by an embodiment of the present invention for performing unknown anomaly detection processing on flight multi-modal feature data based on an unknown anomaly detection module to obtain unknown anomaly detection information of the flight multi-modal feature data; as Figure 10 shown, the above-mentioned performing unknown anomaly detection processing on flight multi-modal feature data based on an unknown anomaly detection module to obtain unknown anomaly detection information of the flight multi-modal feature data may include:

[0265] S1001: Determine multiple preset known abnormal feature data and flight multimodal feature data as mixed feature data;

[0266] S1002: performing clustering processing on the mixed feature data based on a preset clustering algorithm to obtain a cluster feature data cluster set;

[0267] S1003: traverse each clustering feature data cluster in the clustering feature data cluster set;

[0268] In the case of traversing to any cluster feature data cluster, determining that there is no preset known abnormal feature data in the cluster, and that the feature data in the current cluster feature data cluster whose number of feature data in the cluster is greater than or equal to the preset number is unknown abnormal data;

[0269] S1004: Determine unknown anomaly detection information of the flight multimodal feature data based on the unknown anomaly data.

[0270] In a specific embodiment, the preset known abnormal feature data may be feature data obtained by extracting features from preset known abnormal data of a determined abnormal type; specifically, the preset known abnormal feature data may be obtained by obtaining the preset known abnormal data and inputting the preset known abnormal data into a multimodal feature extraction network for feature extraction. Optionally, the preset known abnormal feature data may include multimodal feature data labeled with abnormal types based on known abnormal types. Optionally, the preset clustering algorithm may be set in combination with actual applications, and specifically, the preset clustering algorithm may be a K-means algorithm. Optionally, the cluster feature data cluster set may include multiple cluster feature data clusters. Optionally, the preset number may be set in combination with actual applications, and the preset number may be used to exclude the probability of isolated data (a cluster containing only one data) appearing. Unknown abnormal data may be data of unclear abnormal type; unknown abnormality detection information may characterize whether there are unknown abnormalities in the flight multimodal feature data, and data of unclear abnormal type.

[0271] In a specific embodiment, the K-means algorithm can be used to cluster the mixed feature data to obtain multiple clusters of feature data, and the sum of squared errors within the clusters is minimized. Select one cluster of feature data from the multiple clusters of feature data as the current cluster of feature data. Determine whether there is preset known abnormal feature data in the current cluster of feature data, and determine whether the number of feature data in the current cluster of feature data is greater than or equal to a preset number. In the case where the current cluster of feature data meets the conditions of not having preset known abnormal feature data and the number of feature data being greater than or equal to the preset number, determine the current cluster of feature data as unknown abnormal data. Select the next cluster of feature data from the multiple clusters of feature data, and repeat the above steps of determining whether there is preset known abnormal feature data in the current cluster of feature data and determining whether the number of feature data in the current cluster of feature data is greater than or equal to the preset number. In the case where the current cluster of feature data meets the conditions of not having preset known abnormal feature data and the number of feature data being greater than or equal to the preset number, determine the current cluster of feature data as unknown abnormal data until all the multiple clusters of feature data are traversed. Based on the determined multiple pieces of unknown abnormal data, the unknown abnormal detection information of the flight multi-modal feature data can be determined.

[0272] In a specific embodiment, when the abnormal analysis task is an unknown abnormal analysis task, after obtaining the flight multi-modal feature data, based on the unknown abnormal detection module, the unknown abnormal detection processing can be performed on the flight multi-modal feature data to obtain the unknown abnormal detection information. By performing different task processing procedures for different abnormal analysis tasks, the efficiency and accuracy of the abnormal analysis task processing can be improved, and resource waste can be reduced.

[0273] In a specific embodiment, the above unknown abnormal detection module can be constructed in the following manner:

[0274] Obtain a mixed sample set, where the mixed sample set includes a known abnormal sample set and an unknown abnormal sample set;

[0275] Input the mixed sample set into a multi-modal feature extraction network for feature extraction processing to obtain mixed multi-modal feature data;

[0276] Based on a preset clustering algorithm, perform clustering processing on the mixed multi-modal feature data to obtain a set of mixed feature data clusters;

[0277] Traverse each mixed feature data cluster in the set of mixed feature data clusters;

[0278] When any mixed feature data cluster is traversed, determine that there are known abnormal samples in the cluster, and determine the mixed multi-modal feature data in the current mixed feature data cluster where the number of samples in the cluster is greater than or equal to the preset number as unknown abnormal data;

[0279] Determine the sample unknown anomaly detection information of the mixed sample set based on the unknown anomaly data;

[0280] Adjust the parameters of the to-be-built unknown anomaly detection module based on the unknown anomaly sample set and the sample unknown anomaly detection information;

[0281] Repeat the above steps until the second preset condition is reached, and determine the unknown anomaly detection module.

[0282] In a specific embodiment, the refinement before obtaining the mixed sample set can refer to the relevant refinement before obtaining the anomaly samples to be recognized above, which will not be elaborated here. Reconstructing the mixed sample set can increase the sample size so as to improve the accuracy of unknown anomaly detection. The second preset condition can be set in combination with the actual application. Optionally, the second preset condition can be that the number of iterations reaches a preset number, or all the mixed sample sets are traversed. Optionally, during the process of building the unknown anomaly detection module, the anomaly samples to be recognized for detection processing can be pre-stored.

[0283] In the above embodiment, deep metric learning can also handle the open set problem, and unknown anomaly detection can be realized based on deep metric learning, thereby effectively discovering potential unknown anomalies in the UAV flight data.

[0284] In an alternative embodiment, Figure 11 is a schematic flow chart of a method for performing unknown anomaly recognition processing on unknown anomaly detection information based on unknown anomaly detection information and a preset unknown anomaly recognition algorithm to obtain unknown anomaly type information of flight multi-modal feature data; as Figure 11 shown, the above method for performing unknown anomaly recognition processing on unknown anomaly detection information based on unknown anomaly detection information and a preset unknown anomaly recognition algorithm to obtain unknown anomaly type information of flight multi-modal feature data may include:

[0285] S1101: Obtain a training set, a test set, and a support set;

[0286] S1102: Determine the support set and the training set as a mixed data set;

[0287] S1103: Input the mixed data set into a multi-modal feature extraction network for feature extraction to obtain a mixed data feature set;

[0288] S1104: Input the test set into a multi-modal feature extraction network for feature extraction to obtain a test data feature set;

[0289] S1105: Traverse each test data feature in the test data feature set;

[0290] In the case of traversing any test data feature, determine the vector distance between the current test data feature and each mixed data feature;

[0291] S1106: Determine the abnormal type label of the mixed data feature corresponding to the minimum distance among the multiple vector distances as the abnormal type label of the current test data feature;

[0292] S1107: Based on the abnormal type label of each test data feature, determine the unknown abnormal type information.

[0293] In a specific embodiment, the support set may include a preset number of unknown abnormal type data, and the preset number of unknown abnormal type data is obtained by performing unknown abnormal type marking processing on a preset number of first unknown abnormal data, and the preset number of first unknown abnormal data is obtained from the unknown abnormal detection information; optionally, a small amount of unknown abnormal data with unknown abnormal types can be marked to determine the abnormal types of the small amount of unknown abnormal data, thereby constituting the support set, that is, the preset number of unknown abnormal type data can represent the preset number of unknown abnormal data and the corresponding abnormal types. Optionally, the marking processing may include manual marking. The test set may include a known abnormal type data set and an unknown abnormal data set, and the unknown abnormal data set may include multiple second unknown abnormal data. The preset number of first unknown abnormal data and the multiple second unknown abnormal data can form the unknown abnormal detection information; optionally, the test set does not include the data in the support set; optionally, the test set may include the known abnormal type information identified by the above-known abnormal identification module and the unknown abnormal detection information detected by the above-unknown abnormal detection module, and the unknown abnormal detection information does not include the preset number of first unknown abnormal data that make up the support set. The training set may include known abnormal type data samples; optionally, the training set combined with the support set can be used to update the unknown abnormal detection module, thereby realizing unknown abnormal identification and determining the unknown abnormal type. Optionally, the abnormal type label of the mixed data feature can be obtained based on the abnormal types determined from the known abnormal type data samples and the preset number of first unknown abnormal type data. Optionally, the unknown abnormal type information can represent the unknown abnormal data of the UAV multi-sensor system and the corresponding abnormal types, and the unknown abnormal type information can be the unknown abnormal data with determined abnormal types existing in the UAV multi-sensor system.

[0294] In a specific embodiment, in the process of determining the abnormal type label of the mixed data feature corresponding to the minimum distance among the multiple vector distances as the abnormal type label of the current test data feature, the abnormal type label of the known abnormal type data can also be determined. Therefore, this process can also determine the known abnormal type information and can simultaneously achieve fine-grained distinction between known and unknown abnormalities.

[0295] In the above embodiments, fine-grained recognition of unknown anomalies can be achieved through online learning and online updates of few-shot learning, the information of unknown anomaly types can be determined, the transformation from unknown anomaly detection to unknown anomaly recognition can be completed, and the computing cost can be reduced and resource waste can be reduced.

[0296] In a specific embodiment, when the known anomaly type information and unknown anomaly type information of the UAV multi-sensor system are determined, the anomalies existing in the UAV multi-sensor system can be determined in a timely manner, and the UAV multi-sensor system can be repaired in a timely manner for this anomaly, so as to improve the safety and accuracy of the use of the UAV multi-sensor system.

[0297] As can be seen from the technical solutions provided in the embodiments of this specification above, this specification can determine the needs of users for anomaly detection of the UAV multi-sensor system by receiving the anomaly analysis tasks sent by users; then, based on the anomaly analysis tasks, the UAV flight data collected by the UAV multi-sensor system is obtained, and the UAV flight data is input into the UAV anomaly analysis model to obtain the anomaly analysis information corresponding to the anomaly analysis task. The anomalies existing in the UAV multi-sensor system and the corresponding anomaly types can be determined based on the user needs, improving the accuracy and efficiency of anomaly detection of the UAV multi-sensor system, and reducing resource waste.

[0298] The UAV anomaly analysis model includes a multi-modal feature extraction network, a known anomaly recognition module, and an unknown anomaly recognition module. The multi-modal feature extraction network is used for feature extraction, and the multi-modal feature extraction network is obtained by adjusting the parameters of the initial multi-modal feature extraction network based on the triplet margin total loss, which can effectively balance the relationship between different modal features, reduce the distance between the same type of features, increase the distance between different types of features, and thus improve the accuracy and efficiency of known anomaly recognition and unknown anomaly recognition; the known anomaly recognition module is used to recognize known anomaly types, the unknown anomaly detection module is used to detect unknown anomalies, and when the unknown anomaly detection module detects an unknown anomaly, it combines a preset unknown anomaly recognition algorithm to recognize known anomaly types, which can realize the discovery of potential unknown anomalies in the UAV multi-sensor system, and realize the recognition of known anomaly types and unknown anomaly types in the UAV multi-sensor system, and improve the accuracy and efficiency of the recognition of known anomaly types and unknown anomaly types.

[0299] The embodiment of the present invention also provides an anomaly detection device for a UAV multi-sensor system. Correspondingly, Figure 12 is a schematic structural diagram of an anomaly detection device for a UAV multi-sensor system provided by an embodiment of the present invention; as Figure 12 shown, the above device includes:

[0300] A task receiving module 1210, configured to receive an anomaly analysis task sent by a user;

[0301] A data acquisition module 1220, configured to acquire drone flight data collected by a drone multi-sensor system based on the anomaly analysis task;

[0302] An anomaly analysis and processing module 1230, configured to input the drone flight data into a drone anomaly analysis model to obtain anomaly analysis information corresponding to the anomaly analysis task; wherein, the drone anomaly analysis model includes a multi-modal feature extraction network, a known anomaly recognition module, and an unknown anomaly detection module; the multi-modal feature extraction network is used for feature extraction, and the multi-modal feature extraction network is obtained by adjusting the parameters of an initial multi-modal feature extraction network based on a triplet margin total loss, and the triplet margin total loss is determined based on a triplet set in combination with a triplet loss function; the known anomaly recognition module is used to recognize known anomaly types; the unknown anomaly detection module is used to detect unknown anomalies; when the unknown anomaly detection module detects an unknown anomaly, it is used to recognize the unknown anomaly type in combination with a preset unknown anomaly recognition algorithm.

[0303] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0304] An embodiment of the present invention further provides an electronic device, the electronic device includes: a processor and a memory, and at least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the anomaly detection method for a drone multi-sensor system as described in any one of the method embodiments.

[0305] An embodiment of the present invention further provides a computer storage medium, the computer storage medium can be set in a server to store at least one instruction, at least one program, a code set or an instruction set for implementing the method embodiment, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the anomaly detection method for a drone multi-sensor system as described in any one of the method embodiments.

[0306] Optionally, in an embodiment of the present invention, the above storage medium may be located in at least one network server among a plurality of network servers of a computer network. Optionally, in an embodiment of the present invention, the above storage medium may include, but is not limited to: various media capable of storing program codes such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs.

[0307] Those skilled in the art should understand that the embodiments of the present invention may be provided as a method, a system, or a computer program product. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0308] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one flow or multiple flowcharts and / or blocks Figure 1 one block or multiple blocks.

[0309] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one flow or multiple flowcharts and / or blocks Figure 1 one block or multiple blocks.

[0310] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, so that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for realizing the functions specified in Figure 1 one flow or multiple flowcharts and / or blocks Figure 1 one block or multiple blocks.

[0311] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0312] Finally, it should be noted that the embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit and scope of the present invention as defined by the claims. All of these are within the scope of the present invention.

Claims

1. An anomaly detection method for a multi-sensor system of an unmanned aerial vehicle, characterized in that, The method includes: Receiving an anomaly analysis task sent by a user; Based on the anomaly analysis task, obtaining unmanned aerial vehicle (UAV) flight data collected by a UAV multi-sensor system; Inputting the UAV flight data into a UAV anomaly analysis model to obtain anomaly analysis information corresponding to the anomaly analysis task; Wherein, the UAV anomaly analysis model includes a multi-modal feature extraction network, a known anomaly recognition module, and an unknown anomaly detection module; the multi-modal feature extraction network is used for feature extraction, and the multi-modal feature extraction network is obtained by adjusting the parameters of an initial multi-modal feature extraction network based on a triplet margin total loss, and the triplet margin total loss is determined based on a triplet set in combination with a triplet loss function; the triplet set is obtained according to at least one triplet obtained after traversing each flight data feature in flight sample data features; a triplet is determined according to a current anchor data, a current positive flight data, and a current negative flight data; the current anchor data is determined based on any flight data feature traversed in the flight sample data features; the current positive flight data is determined according to positive flight data whose first distance from the current anchor data is greater than the sum of a minimum negative distance and a preset distance; the positive flight data is determined according to data having the same anomaly type as the current anchor data in the flight sample data features; the current negative flight data is determined according to negative flight data whose second distance from the current anchor data is less than the difference between a maximum positive distance and a preset distance; the negative flight data is determined according to data having a different anomaly type from the current anchor data in the flight sample data features; the maximum positive distance is determined based on the distances between the current anchor data and each positive flight data; the minimum negative distance is determined based on the distances between the current anchor data and each negative flight data; the flight sample data features are obtained by inputting multi-modal flight sample data into the initial multi-modal feature extraction network for feature extraction processing; The known anomaly recognition module is used for recognizing known anomaly types; the unknown anomaly detection module is used for detecting unknown anomalies; the unknown anomaly detection module, in the case of detecting an unknown anomaly, is used for recognizing the unknown anomaly type in combination with a preset unknown anomaly recognition algorithm.

2. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 1, wherein The anomaly analysis task includes a known anomaly analysis task and an unknown anomaly analysis task; The inputting the UAV flight data into a UAV anomaly analysis model to obtain anomaly analysis information corresponding to the anomaly analysis task includes: In the case where the anomaly analysis task is a known anomaly analysis task and an unknown anomaly analysis task, inputting the UAV flight data into the multi-modal feature extraction network for feature extraction to obtain flight multi-modal feature data; Based on the known anomaly recognition module, performing known anomaly recognition processing on the flight multi-modal feature data to obtain known anomaly type information of the flight multi-modal feature data; Based on the unknown anomaly detection module, perform unknown anomaly detection processing on the flight multi-modal feature data to obtain the unknown anomaly detection information of the flight multi-modal feature data; Based on the unknown anomaly detection information and a preset unknown anomaly recognition algorithm, perform unknown anomaly recognition processing on the unknown anomaly detection information to obtain the unknown anomaly type information of the flight multi-modal feature data; Based on the known anomaly type information and the unknown anomaly type information, determine the anomaly analysis information of the anomaly analysis task.

3. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 2, wherein The multi-modal feature extraction network includes a timing feature extraction network, an attitude feature extraction network, and a feature fusion network; the UAV flight data includes timing data and attitude data; The step of inputting the UAV flight data into the multi-modal feature extraction network for feature extraction to obtain flight multi-modal feature data includes: Input the timing data into the timing feature extraction network for timing feature extraction to obtain timing features; Input the attitude data into the attitude feature extraction network for attitude feature extraction to obtain attitude features; Input the timing features and the attitude features into the feature fusion network for feature fusion processing to obtain flight multi-modal feature data.

4. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 1, characterized in that, The multi-modal feature extraction network is constructed in the following manner: Train an initial multi-modal feature extraction network including an initial timing feature extraction network, an initial attitude feature extraction network, and an initial feature fusion network to obtain an initial multi-modal feature extraction network; Obtain multi-modal flight sample data; Input the multi-modal flight sample data into the initial multi-modal feature extraction network for feature extraction processing to obtain flight sample data features; Perform triplet extraction processing on the flight sample data features to obtain a triplet set; Based on the triplet set, combine a triplet loss function to determine the total triplet margin loss; Based on the total triplet margin loss, adjust the parameters of the initial multi-modal feature extraction network to obtain a multi-modal feature extraction network.

5. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 4, wherein The step of performing triplet extraction processing on the flight sample data features to obtain a triplet set includes: Traverse each flight data feature in the flight sample data features; When any flight data feature is traversed, determine the flight data feature as the current anchor data; Determine at least one positive flight data with the same anomaly type as the current anchor data in the flight sample data features; Based on the distance between the current anchor data and each positive flight data, determine the maximum positive distance; Determine at least one negative flight data with a different anomaly type from the current anchor data in the flight sample data features; Based on the distance between the current anchor data and each negative flight data, determine the minimum negative distance; Determine the positive flight data whose first distance from the current anchor data is greater than the first target distance as the current positive flight data; the first target distance is the sum of the minimum negative distance and a preset distance; Determine the negative flight data corresponding to the second distance between the current anchor data and the second target distance is less than the second target distance as the current negative flight data; the second target distance is the difference between the maximum positive distance and the preset distance; Take the current anchor data, the current positive flight data, and the current negative flight data as a triple; After traversing each flight data feature in the flight sample data features, take the at least one triple obtained as a triple set.

6. The anomaly detection method for the multi-sensor system of an unmanned aerial vehicle according to claim 2, wherein Based on the unknown anomaly detection module, perform unknown anomaly detection processing on the flight multi-modal feature data to obtain the unknown anomaly detection information of the flight multi-modal feature data, including: Determine at least one preset known anomaly feature data and the flight multi-modal feature data as mixed feature data; Perform clustering processing on the mixed feature data based on a preset clustering algorithm to obtain a set of clustering feature data clusters; Traverse each clustering feature data cluster in the set of clustering feature data clusters; In the case of traversing any clustering feature data cluster, determine that there is no preset known anomaly feature data in the cluster, and the feature data in the current clustering feature data cluster where the number of feature data in the cluster is greater than or equal to the preset number is unknown anomaly data; Based on the unknown anomaly data, determine the unknown anomaly detection information of the flight multi-modal feature data.

7. The anomaly detection method for a drone multi-sensor system according to claim 2, characterized in that, Based on the unknown anomaly detection information and a preset unknown anomaly recognition algorithm, perform unknown anomaly recognition processing on the unknown anomaly detection information to obtain the unknown anomaly type information of the flight multi-modal feature data, including: Obtain a training set, a test set, and a support set; the support set includes a preset number of unknown anomaly type data, the preset number of unknown anomaly type data is obtained by performing unknown anomaly type marking processing on a preset number of first unknown anomaly data, the preset number of first unknown anomaly data is obtained from the unknown anomaly detection information; the test set includes a known anomaly type data set and an unknown anomaly data set, the unknown anomaly data set includes at least one second unknown anomaly data, the preset number of first unknown anomaly data and the at least one second unknown anomaly data constitute the unknown anomaly detection information; the training set includes known anomaly type data samples; Determine the support set and the training set as a mixed data set; Input the mixed data set into a multi-modal feature extraction network for feature extraction to obtain a mixed data feature set; Input the test set into a multi-modal feature extraction network for feature extraction to obtain a test data feature set; Traverse each test data feature in the test data feature set; In the case of traversing any test data feature, determine the vector distance between the current test data feature and each mixed data feature; Determine the anomaly type label of the mixed data feature corresponding to the minimum distance among the N vector distances as the anomaly type label of the current test data feature; Based on the anomaly type labels of each test data feature, determine the unknown anomaly type information.

8. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 2, wherein Based on the known anomaly recognition module, perform known anomaly recognition processing on the flight multi-modal feature data to obtain the known anomaly type information of the flight multi-modal feature data, including: Based on a preset known anomaly recognition algorithm, perform known anomaly recognition processing on the flight multi-modal feature data to obtain the known anomaly type information of the flight multi-modal feature data, wherein the preset known anomaly recognition algorithm includes the nearest neighbor algorithm.

9. The anomaly detection method for a multi-sensor system of an unmanned aerial vehicle according to claim 1, characterized in that, Before obtaining the UAV flight data collected by the UAV multi-sensor system based on the anomaly analysis task, it includes: Based on the abnormal analysis task, obtain the original UAV flight data collected by the UAV multi-sensor system, where the original UAV flight data includes flight status data, and each flight status data includes at least one status acquisition data; Perform quantity alignment processing on the state acquisition data of the flight state data to obtain aligned flight state data, and each aligned flight state data includes at least one aligned state acquisition data; Perform data normalization processing on at least one aligned state acquisition data of each of the aligned flight state data to obtain normalized flight state data, and each normalized flight state data includes at least one normalized state acquisition data; Based on at least one normalized state acquisition data of each normalized flight state data, combine with a sliding window to perform data reconstruction processing to determine the UAV flight data.

10. A computer storage medium, characterized in that: At least one instruction and at least one program are stored in the computer storage medium, and the at least one instruction and the at least one program are loaded and executed by a processor to implement the anomaly detection method for a UAV multi-sensor system as described in any one of claims 1 to 9.

Images